Automatic scenario unlocking

[Tyler-lc]

A few days ago, for some reason, I found a bunch of my scenarios on ixmp-dev were locked (most likely due to a script that crashed).
After searching around in Slack for a solution, I came across a small script that @glatterf42 had shared a while ago to unlock a scenario, basically at no risk, by only knowing

• model
• scenario name
• version

So I decided to write a small script that is heavily based upon Fridolin's script. I took all of my locked scenario from ixmp-dev and unlocked them sequentially one-by-one. I think this could be a good start to implement a function in message_ix that unlocks scenarios easily and without too much hassle.

from jpype.types import JInt, JString

from ixmp.backend.jdbc import JDBCBackend
from ixmp.core.platform import Platform
import ixmp

mp = ixmp.Platform("ixmp-dev")
MODEL_NAME = "some_model_name_here"
df = mp.scenario_list(model="MODEL_NAME", default=False)
locked_scens = df[df.is_locked == True]


platform_name = "ixmp-dev"
ixmp_dev = Platform(platform_name)
assert isinstance(ixmp_dev._backend, JDBCBackend)  # for type checkers

for _, row in locked_scens.iterrows():
    model = row["model"]
    scenario = row["scenario"]
    version = row["version"]
    print(f"Unlocking {model} - {scenario} - {version}")

    # NOTE Unlocking an unlocked scenario is fine, so we can always run this :)
    try:
        run_id = ixmp_dev._backend.jobj.getDb().getRunId(
            JString(model), JString(scenario), JInt(version)
        )
        ixmp_dev._backend.jobj.unlockRunid(run_id)
    except Exception as e:
        print(f"Failed to unlock {model} - {scenario} - {version}: {e}")

# now let's re-import the scenarios list to check if they are unlocked
df_after = mp.scenario_list(model="MODEL_NAME", default=False)
if df_after[df_after.is_locked == True].empty:
    print("All scenarios are now unlocked.")
else:
    print("Some scenarios are still locked:")
    print(df_after[df_after.is_locked == True])

What do we think would be the appropriate place to implement this?
I even think this would be a pretty good safety protocol before implementing long-running pipelines.
If for example we have at some point the need to open/clone/modify a given scenario, it would be possible to first check if it is locked -> unlock -> proceed with pipeline, instead of breaking a whole sequence that might waste hours.

[Wegatriespython]

Thanks for the suggestion Luca! Yes this would be nice convenience tool, I am thinking whether it would make sense to have it implemented as a scenario method in ixmp and inherit it to message_ix. So that we have a scenario.unlock and potentially a batch wrapper in message_ix

[khaeru]

After searching around in Slack for a solution, I came across a small script that @glatterf42 had shared a while ago to unlock a scenario

It's important to preserve the link to this context, prior iterations of the script, and experiences of users. The exact message (for those who are on Slack) is here.

[Tyler-lc]

As discussed briefly with @khaeru , there might be a couple of instances where lowering the barrier to unlocking a scenario might become counter-productive if not actively damaging:

• the scenario is locked because someone is actively making changes in that moment
  • it can happen that one person is actually manually checking and modifying parameters (e.g. using a jupyter notebook)
  • it can happen in the course of a long running script that also modifies parameters and temporarily checks out the scenario (which makes is_locked = True)
• The scenario may be under active solving. Unlock in mid-solving may have unforseen consequences (i'm not sure what would be the actual version on ixmp with which data - pre or post unlocked?). But we can definitely agree this would be a bad case.

I'm not sure if someone has more ideas of when this can still be problematic.
But the concept that came out from a brief discussion with Paul is that we want to make this process frictionless enough so that it is easy to implement when scenarios are genuinely locked due to an improperly closed database (Whatever the reason may be).
At the same time, we do not want to make it so frictionless that anyone feels free to first run unlocks (attempting to unlock an already unlocked scenario does not raise an error, so it could be theoretically ran before any scen.solve() or scen.check_out().

I think that possibly a couple of checks should be made before unlocking the scenario:

1. is the unlocking user also the lock_user/upd_user/cre_user.

• this would make it so that the unlocking user doesn't unlock a scenario that they haven't created and they haven't locked.
• I think the lock_user may be the most important. Because you don't want to necessarily unlock a scenario that someone else has locked without asking before

2. lock_date

• The idea here is prevent unlocking scenarios that have been locked within 15-20 minutes maybe. Especially if the user does not match. If it is you who has locked the scenario in this locking only happened recently, most likely you should not be unlocking this scenario without talking to someone

So put these checks in place, and raise an error (value error/assertion/Unlock error?) when they do not match with user or lock_date too recent.
The error should point out the reason of the failed unlock (current user does not match lock_user, lock_date xx minutes ago), with a suggestion to talk to upd_user before unlocking.
I would also suggest to introduce an argument force = False (by default). Because often my upd/cre_user changes (windows machine is casamassima, linux WSL has something different -can't remember it, macos, is my full name). So the user can recognize their own name and decide to use force = True in the argument. But also it may be the case that a colleague asks another to unlock a scenario for them for any number of reasons.
So eventually we would implement two Force:

• force_time = False
• force_user = False

Only when time of unlock older than 45 minutes and current user is lock user the unlock happens automatically. Any other time the method should require an explicit force flag and raise an error with name of lock_user and lock_date.
Additionally, I would propose to add a alias_users = ["casamassima", "macos_casamassima", "linux_casamassima"] as an optional input to the method, so that they would be equivalent to the current user and no need to run necessarily force_user = True.

I'm not entirely sure what would be the correct implementation here, if it should be something like

import getpass
from datetime import datetime

MIN_LOCK_AGE_MINUTES = 45
USER_ALIASES = ["casamassima", "another_casamassima", "that_other_casamassima"]

df = mp.scenario_list(model="model", default = False)
df_scens = df[(df.scenario==scenario)&(df.is_locked=True)]

## assuming we iterate through a df row by row. Otherwise we can change this.
## The important part is the logic for the automatic unlock vs UnlockError.
## Potentially we can create a class of UnlockError(Exception)
def check_unlock(row, *, user_aliases = None, min_lock_age = MIN_LOCK_AGE_MINUTES, force_user = False, force_time = False):
"""
Check whether unlocking a scenario is first before actually doing so!
Only proceeds with unlock without interruption if:
- lock_user matches current user or aliases AND
- lock is older than min_lock_age_minutes
All other cases raise UnlockError unless the appropriate flag is set. Force flags downgrade erros to print warnings - never silent unlocks

"""
    current_user = getpass.getuser()
    known_users = {current_user, *(user_aliases or [])}

    lock_user = row.get("lock_user")
    lock_date_str = row.get("lock_date")
    label = f"{row['model']}/{row['scenario']} (v{row['version']})"

    # -- Determine conditions --
    user_known = lock_user in known_users if lock_user else True

    lock_stale = True  # default if no lock_date
    if lock_date_str:
        lock_time = datetime.strptime(lock_date_str, "%Y-%m-%d %H:%M:%S.%f")
        age_minutes = (datetime.now() - lock_time).total_seconds() / 60
        lock_stale = age_minutes >= min_lock_age

    # -- Apply matrix --
    # Only user_known + lock_stale passes without force.
    # Everything else raises unless the specific force is set.

    errors = []

    if not user_known:
        msg = (f"[{label}] lock_user '{lock_user}' is not in known users "
               f"{sorted(known_users)}. Contact {lock_user} before unlocking.")
        if force_user:
            print(f"  WARNING (force_user): {msg}")
        else:
            errors.append(msg)

    if not lock_stale:
        msg = (f"[{label}] locked {age_minutes:.0f} min ago by '{lock_user}' "
               f"(threshold: {min_lock_age} min). May be in active use.")
        if force_time:
            print(f"  WARNING (force_time): {msg}")
        else:
            errors.append(msg)

    if errors:
        raise UnlockError(
            "Unlock blocked:\n"
            + "\n".join(f"  - {e}" for e in errors)
            + "\n\nOverride with force_user=True / force_time=True as needed."
        )

    if user_known and lock_stale:
        print(f"  OK: [{label}] lock by '{lock_user}', stale. Proceeding.")

The 45 minutes is quite arbitrary from my side. I just thought it is a safe number to use. Because message-runs usually don't take much longer than 15 - 20 minutes. 45 minutes is just an extra precaution. But we can discuss whether this is enough or too much I suppose.

I think this should be the main logic though. And that this covers quite a number of scenarios and also acts in a safe-first manner.