FIX: learned costs survive updates, reset only by a declared cost epoch

.gitignore

@@ -43,6 +43,8 @@ local-server-*.log
 
 # The operator page's run history (server/.history, HISTORY_DIR)
 .history/
+# The learned cost table (server/.costs, COST_TABLE_PATH in docker-compose.yml)
+.costs/
 # The update agent and the server leave each other notes here (UPDATE_DIR)
 .update/
 # The update agent's log (scripts/server_ctl.py agent)

ADDING_A_TOOL.md

@@ -143,6 +143,29 @@ Raise `ValueError` or `FileNotFoundError` for something the caller can fix -
 the message reaches them as a `422`. Anything else is a `500` with a fixed
 message, and the traceback stays in the server log.
 
+## When a release changes what the tool costs
+
+The server learns each tool's RAM and VRAM from its own runs and keeps them
+across restarts and updates (`server/execution/costs.py`). An ordinary release
+keeps them too, which is what you want: a tool nobody has measured runs alone.
+
+When a release changes what the tool **costs** -- a new or larger model, a
+different handling of the input's size, a new engine -- the old figures would
+mislead admission. Say so in `pyproject.toml`:
+
+```toml
+[tool.sadt]
+tool = true
+name = "MyTool"
+cost_epoch = 1      # absent is 0; bump by one for such a release
+```
+
+At its next start the server drops that tool's learned figures, and no other
+tool's, logs `Learned costs  RESET MyTool: cost_epoch 0 -> 1`, and relearns
+from the next run. Do not bump it for a bug fix, a new argument or a UI change.
+A value that is not an integer is logged and ignored: the tool still loads and
+its figures are kept.
+
 ## The exceptions
 
 Only if a convention is wrong for you, add a section to

docker-compose.yml

@@ -146,6 +146,15 @@ services:
       # The tools mount is read-only, so regenerated schemas need somewhere of
       # their own to land.
       - SCHEMA_CACHE_DIR=/tmp/schema-cache
+      # The learned cost table, which is NOT a cache and must not share the
+      # schema cache's /tmp: that goes with the container, and the update agent
+      # recreates the container on every update touching this file, the API's
+      # requirements or the image -- each of which silently put every tool back
+      # to "unmeasured", reserving the whole budget and running alone. In the
+      # bind-mounted checkout instead, beside server/.history and
+      # server/.update, which outlive the container the same way; gitignored,
+      # so the clone stays clean. See COST_TABLE_PATH in server/config.py.
+      - COST_TABLE_PATH=/workspace/server/.costs/tool_costs.json
       # Which peers uvicorn believes about `X-Forwarded-For`. Its default is
       # 127.0.0.1, and behind the TLS terminator the README documents the peer
       # is never that: nginx reaches 127.0.0.1:8000, Docker's proxy forwards it
@@ -201,6 +210,8 @@ services:
       # The tools mount is read-only, so regenerated schemas need somewhere of
       # their own to land.
       - SCHEMA_CACHE_DIR=/tmp/schema-cache
+      # Outside the container's /tmp; see the `inference` service.
+      - COST_TABLE_PATH=/workspace/server/.costs/tool_costs.json
       # Trusted proxies for `X-Forwarded-For`; see the `inference` service.
       - FORWARDED_ALLOW_IPS=${FORWARDED_ALLOW_IPS:-172.16.0.0/12}
       # How clients reach it (see `ports` and server/serve.py). The

docs/INTERNALS.md

@@ -28,10 +28,10 @@ server/
 
 ## Discovery
 
-`_tool_folders` walks `TOOLS_DIR` to depth 2 and returns `(api name, path)` for
-every directory whose `pyproject.toml` declares `[tool.sadt] tool = true`. It
-does not descend into a folder that is already a tool, and a leading underscore
-or dot excludes one.
+`_tool_folders` walks `TOOLS_DIR` to depth 2 and returns
+`(api name, path, cost_epoch)` for every directory whose `pyproject.toml`
+declares `[tool.sadt] tool = true`. It does not descend into a folder that is
+already a tool, and a leading underscore or dot excludes one.
 
 Depth 2 exists so a grouping folder can hold several tools: `tools/ALI/` is not
 a tool, `tools/ALI/ALI_CBCT/` is. A shared package like `tools/ALI/common/` has
@@ -42,6 +42,12 @@ directory must still be NAMED after the tool, because the interpreter is looked
 up by name; what declaring it buys is that the tool's DEPTH may change without
 the name a client sends moving with it.
 
+`[tool.sadt] cost_epoch` (an integer, absent = 0) is carried on the tool and
+handed to `costs.reconcile` at startup: an entry of the learned cost table
+recorded under another epoch is dropped, logged with both numbers, and
+relearned. A value that is not an integer is logged and ignored -- the tool
+loads, and its learned entry is left as it is.
+
 ### Failures that are fatal, and why
 
 Most discovery failures cost one tool: it is skipped, recorded in

server/.env.example

@@ -81,6 +81,12 @@ SADT_DISPATCH_MODE=inprocess
 # them, so a regenerated one cannot live next to its tool.
 # SCHEMA_CACHE_DIR=/var/cache/sadt
 
+# Where the learned cost table is kept: what every run measured itself to
+# need. NOT a cache -- losing it makes every tool run alone until measured
+# again -- so it must outlive the container. Empty keeps it beside
+# SCHEMA_CACHE_DIR; docker-compose.yml sets it for the `inference` services.
+# COST_TABLE_PATH=/workspace/server/.costs/tool_costs.json
+
 # Per-tool server-side configuration: which of a tool's arguments may be
 # filled from DATA_DIR, and its upload limit. See deployment.toml.example.
 # A missing file is the normal case, not an error. Defaults to the copy beside

server/README.md

@@ -705,8 +705,10 @@ re-enters the server at all.
 
 What a run reserves is what it was **measured** to need. `runner.py` has written
 a peak per run since the subprocess path landed; `execution/costs.py` keeps it
-as a high-water mark per tool under `SCHEMA_CACHE_DIR`, and nothing is declared
-by hand anywhere. Three rules keep that safe:
+as a high-water mark per tool in `COST_TABLE_PATH` (beside `SCHEMA_CACHE_DIR`
+when unset; the compose `inference` services keep it in the checkout's
+`server/.costs/`, which outlives the container), and nothing is declared by
+hand anywhere. Three rules keep that safe:
 
 - a tool nothing has measured reserves the whole budget and **runs alone**, so
   an empty table behaves exactly like the counter it replaced, and one run of
@@ -716,6 +718,12 @@ by hand anywhere. Three rules keep that safe:
 - the card's **actually free** memory is checked too, because history cannot
   know what another process on the host is holding.
 
+The table survives restarts, recreated containers and updates of the server or
+the tools. Only a release resets it, and only by saying so: a tool's
+`[tool.sadt] cost_epoch` moving drops that tool's entry (`ADDING_A_TOOL.md`),
+and `costs.TABLE_EPOCH` moving drops the whole table. Either is done once at
+startup and logged as `Learned costs  RESET ...` with both epochs.
+
 There is no job COUNTER above the budget any more. `MAX_CONCURRENT_GPU_JOBS`
 was the last one and it is gone: despite its name it capped every running job,
 so pinning it to 1 would have serialised a tabular prediction behind a

server/base.py

@@ -450,6 +450,11 @@ class Tool(ABC):
     # A tool whose inputs are paired publishes this instead: see
     # registry/conventions.paired_batch_plan.
     paired_batch: Optional[dict] = None
+    # `[tool.sadt] cost_epoch`, set by the registry: moving it drops what this
+    # deployment learned about the tool's memory (execution/costs.py). None
+    # when the pyproject declares one that is not an integer, which leaves the
+    # learned figures alone. An in-process tool declares nothing, and is 0.
+    cost_epoch: Optional[int] = 0
 
     def check_schema(self) -> None:
         """Reject an invalid `arguments` declaration. Called by registry.py at

server/config.py

@@ -92,6 +92,24 @@ def tool_roots(self) -> tuple:
     # A .schema.json is a cache, and the tool folders are read-only to the
     # process serving them, so a regenerated one cannot live beside its tool.
     SCHEMA_CACHE_DIR: str = os.path.join(_SERVER_DIR, ".schema-cache")
+    # The learned cost table (execution/costs.py): what every run this
+    # deployment ever did measured itself to need. Empty keeps it beside the
+    # schema cache, where every existing deployment already has it -- the
+    # venvs image's cache volume, a checkout's .schema-cache.
+    #
+    # **A setting of its own because the two are not the same kind of thing.**
+    # A schema is a cache: losing it costs minutes of regeneration. The table is
+    # not regenerable at all: losing it puts every tool back to "unmeasured",
+    # which reserves the whole budget and runs it ALONE until it is measured
+    # again -- silently, since nothing fails. The compose `inference` services
+    # keep their schema cache in the container's /tmp, and the table went with
+    # it on every update that recreated the container; they point this at the
+    # bind-mounted checkout instead (server/.costs, gitignored), beside
+    # HISTORY_DIR and UPDATE_DIR, which outlive the container the same way.
+    #
+    # What does reset it is said explicitly by a release, and nothing else
+    # does: see `costs.TABLE_EPOCH` and `[tool.sadt] cost_epoch`.
+    COST_TABLE_PATH: str = ""
     # Where finished runs are kept for the operator page, so its history and
     # per-tool graphs survive an update. Beside the code by default, which in
     # the `inference` service is the bind-mounted checkout and therefore

server/execution/costs.py

@@ -44,6 +44,16 @@
 run is an affine function of its width with an intercept, and the intercept is
 most of it. `_fit` is where that is measured rather than declared, and the two
 measured tables that say so are in its docstring.
+
+**The table outlives the server, and only a release can end it.** It is the
+one piece of state nothing can regenerate, so it lives at COST_TABLE_PATH (see
+config.py) rather than wherever a cache happens to be, and it survives a
+restart, a recreated container, a server update and a tools update. Exactly two
+things drop learned figures, and both are written down by whoever released the
+change: a tool's `[tool.sadt] cost_epoch` moving drops that tool's entry, and
+TABLE_EPOCH below moving drops the whole table. `reconcile` applies both, once,
+at startup. A new schema, a new image, a rebuilt venv are not statements about
+what a tool costs, and reset nothing.
 """
 
 from __future__ import annotations
@@ -105,7 +115,37 @@
 OUTLIER_PEERS = 3
 QUARANTINE_KEY = "quarantine"
 
+# The server's epoch for the whole table. **Bump it in a server release that
+# changes what the recorded figures MEAN** -- what a peak includes, how runner.py
+# measures it, the unit a window entry is stored in -- so that every entry
+# learned under the old meaning would mislead admission. Never for an ordinary
+# release, and never for one tool: that is `[tool.sadt] cost_epoch`, in the
+# tool's own pyproject. A table whose epoch differs is dropped whole, once, at
+# the next startup (`reconcile`), and the drop is logged with both numbers.
+#
+# 0 because a table written before the epoch existed carries no marker and is
+# read as 0: the release that introduced it kept every deployment's table.
+TABLE_EPOCH = 0
+# Where the table's epoch is stored: in the table itself, so it moves with the
+# file and is replaced by the same atomic write. A key no discovered tool can
+# have (discovery skips a folder starting with "_"), and an INTEGER rather than
+# a dict, so every reader that looks a tool up by name -- runner.py's
+# `_measured_cost`, and a rolled-back server's `cost_of` -- reads it as "no
+# measurement" rather than as a tool costing nothing. Here it is skipped by
+# `_load`, so `known()`, `banner()` and every page built on them never list it.
+TABLE_EPOCH_KEY = "__table_epoch__"
+# Per entry: the `[tool.sadt] cost_epoch` the tool declared when the entry was
+# learned. Absent -- every entry written before the key existed -- reads as 0,
+# which is also what a tool declaring nothing declares, so both are kept.
+COST_EPOCH_KEY = "cost_epoch"
+
 _lock = threading.Lock()
+# `{tool: declared cost_epoch}` for the tools this server serves, handed over by
+# `reconcile` at startup. A tool missing from it is never judged: one that
+# failed to load this time, or whose epoch could not be read, keeps its entry.
+_declared: dict = {}
+# A table that cannot be written is said ONCE: it fails every run's record.
+_unwritable_said = False
 
 
 @dataclass(frozen=True)
@@ -286,10 +326,18 @@ def input_dependent(self) -> bool:
 
 
 def table_path() -> str:
-    return os.path.join(settings.SCHEMA_CACHE_DIR, "tool_costs.json")
+    """COST_TABLE_PATH, or beside the schema cache when it is not set."""
+    return settings.COST_TABLE_PATH or os.path.join(settings.SCHEMA_CACHE_DIR, "tool_costs.json")
 
 
-def _load() -> dict:
+def _epoch_of(value) -> int:
+    """An epoch as stored. Absent, or anything but an integer, is 0: the epoch
+    every table and entry written before epochs existed belongs to."""
+    return value if isinstance(value, int) and not isinstance(value, bool) else 0
+
+
+def _read() -> dict:
+    """The file as written, the epoch marker included."""
     try:
         with open(table_path(), encoding="utf-8") as handle:
             loaded = json.load(handle)
@@ -298,22 +346,105 @@ def _load() -> dict:
     return loaded if isinstance(loaded, dict) else {}
 
 
+def _current(name: str, entry: dict) -> bool:
+    """Was this entry learned under the cost_epoch its tool declares now?"""
+    if name not in _declared:
+        return True
+    return _epoch_of(entry.get(COST_EPOCH_KEY)) == _declared[name]
+
+
+def _load() -> dict:
+    """Every tool's entry admission may believe, and nothing that is not a tool.
+
+    The epochs are checked here as well as in `reconcile`, so that nothing an
+    epoch has retired is admitted on even before -- or without -- the startup
+    pass that drops it from the file.
+    """
+    raw = _read()
+    if _epoch_of(raw.get(TABLE_EPOCH_KEY)) != TABLE_EPOCH:
+        return {}
+    return {name: entry for name, entry in raw.items()
+            if name != TABLE_EPOCH_KEY and isinstance(entry, dict) and _current(name, entry)}
+
+
 def _store(table: dict) -> None:
-    """Replace the table atomically, or give up quietly.
+    """Replace the table atomically, stamped with TABLE_EPOCH, or give up.
 
     A cost table that cannot be written costs one thing: every tool looks
     unknown and runs alone, which is exactly today's behaviour. It must never
-    cost a run.
+    cost a run -- but it is said once, at WARNING: a table that silently stops
+    persisting is the very defect COST_TABLE_PATH exists to end, and its only
+    symptom is a server that has gone serial for no visible reason.
     """
-    directory = settings.SCHEMA_CACHE_DIR
+    global _unwritable_said
+    path = table_path()
+    directory = os.path.dirname(os.path.abspath(path))
     try:
         os.makedirs(directory, exist_ok=True)
         handle, temporary = tempfile.mkstemp(dir=directory, suffix=".tmp")
         with os.fdopen(handle, "w", encoding="utf-8") as stream:
-            json.dump(table, stream, indent=2, sort_keys=True)
-        os.replace(temporary, table_path())
+            json.dump(dict(table, **{TABLE_EPOCH_KEY: TABLE_EPOCH}), stream,
+                      indent=2, sort_keys=True)
+        os.replace(temporary, path)
     except OSError as exc:
-        logger.debug("could not write the tool cost table: %s", exc)
+        if _unwritable_said:
+            logger.debug("could not write the tool cost table: %s", exc)
+            return
+        _unwritable_said = True
+        logger.warning(
+            "Learned costs  cannot write %s (%s): nothing measured from now on is "
+            "learned, so an unmeasured tool keeps running alone. COST_TABLE_PATH "
+            "must name a writable place that outlives the container.", path, exc)
+
+
+def reconcile(declared: dict) -> None:
+    """Drop what a release SAID is out of date, once, at startup -- nothing else.
+
+    `declared` is `{tool: [tool.sadt] cost_epoch}` for the tools this server
+    serves. Two checks, both explicit decisions of whoever released the change:
+
+    - the table's epoch against TABLE_EPOCH -- a server release that changed
+      what the figures mean drops all of them;
+    - each entry's epoch against its tool's -- a tool release that changed
+      what the tool costs (a new model, different input handling) drops that
+      tool's entry and no other.
+
+    A tool missing from `declared` keeps its entry: one that failed to load
+    this time, or whose epoch could not be read, has not asked for anything.
+
+    Written back at once rather than left to the next `record`, because
+    runner.py reads the FILE: a nested level would otherwise go on sizing
+    itself from figures admission has just stopped believing. Each drop is
+    logged at WARNING with both epochs, which is the line an operator finds
+    when a tool that ran in parallel yesterday runs alone today.
+    """
+    global _declared
+    with _lock:
+        _declared = {name: epoch for name, epoch in (declared or {}).items()
+                     if isinstance(epoch, int) and not isinstance(epoch, bool)}
+        raw = _read()
+        stored = _epoch_of(raw.get(TABLE_EPOCH_KEY))
+        tools = {name: entry for name, entry in raw.items()
+                 if name != TABLE_EPOCH_KEY and isinstance(entry, dict)}
+        changed = False
+        if stored != TABLE_EPOCH:
+            if tools:
+                logger.warning(
+                    "Learned costs  RESET the whole table, %d tool(s): TABLE_EPOCH "
+                    "%d -> %d set by this server's release; every tool runs alone "
+                    "until it is measured again", len(tools), stored, TABLE_EPOCH)
+            tools, changed = {}, bool(raw)
+        for name in sorted(tools):
+            if _current(name, tools[name]):
+                continue
+            logger.warning(
+                "Learned costs  RESET %s: cost_epoch %d -> %d declared by the "
+                "tool's release; it runs alone until it is measured again",
+                name, _epoch_of(tools[name].get(COST_EPOCH_KEY)), _declared[name])
+            del tools[name]
+            changed = True
+        if changed:
+            _store(tools)
 
 
 def _window(entry: dict) -> list:
@@ -674,6 +805,10 @@ def record(tool_name: str, vram_bytes: Optional[int], ram_bytes: Optional[int],
             window = []
         if own_only:
             entry[OWN_ONLY_KEY] = True
+        # The epoch this is learned under, so a later release of the tool can
+        # say it no longer holds. Left as it was for a tool nothing declared.
+        if tool_name in _declared:
+            entry[COST_EPOCH_KEY] = _declared[tool_name]
         # Stored PER CHANNEL, beside the width it was measured at, which is
         # what makes the two recoverable from each other: `_fit` multiplies
         # them straight back into the run's total, and the intercept can only
@@ -783,16 +918,19 @@ def banner() -> str:
     ones costing the parallelism -- and which tools' memory moves with the
     request, because for those the single reserved figure is the worst seen
     rather than a property of the tool.
+
+    And WHERE the table is: the count read after an update, next to the path it
+    was read from, is how an operator sees that the update kept it.
     """
     table = {name: cost_of(name) for name in sorted(_load())}
     if not table:
-        return ("Learned costs  none yet; every tool runs alone until it has "
-                "been measured once")
+        return ("Learned costs  none yet in {}; every tool runs alone until it "
+                "has been measured once".format(table_path()))
     varying = [(name, cost) for name, cost in table.items()
                if cost and cost.input_dependent]
     lines = [
-        "Learned costs  {} tool(s) measured; heaviest {}".format(
-            len(table),
+        "Learned costs  {} tool(s) measured in {}; heaviest {}".format(
+            len(table), table_path(),
             max(table.items(), key=lambda item: item[1].vram_bytes if item[1] else 0)[0],
         )
     ]