fix: stop the fake-prediction scripts downloading a detector they discard

[MateoLostanlen]

Two independent failures that together destroyed twenty GPU-hours on 2026-09-07, plus the reason nothing survived them.

The crash

optimize_sequential.py, copy_failures.py and scan_raw_fp.py each construct a pyroengine Engine, then replace engine.model with _PassthroughModel on the very next line. They replay predictions predict_sequential.py already computed and never run a detector — but Engine.__init__ downloads one from HuggingFace anyway.

That download was dead weight from the start. It became fatal when the model repository stopped answering anonymously: pyronear/yolo11s_mighty-mongoose_v5.1.0, which uv.lock resolves, now returns HTTP 401. Stubbing Classifier before construction keeps the grid search hermetic, so the next model-repo rotation cannot break it again. Reuses the _Stub each file already defines for its API clients.

Why a three-second failure cost ten hours, twice

dvc push ran only after a full dvc repro. The runner is ephemeral, so train_yolo_best succeeding — ten hours of GPU — meant nothing once optimize_sequential_val failed three seconds later. The protection was inverted: the most expensive stage sat behind the most fragile ones, which cost minutes and depend on the network, external repositories and third-party packages.

The pipeline is now split, with a push between the two halves. Uploading the blob is not sufficient on its own — DVC storage is content-addressed, so the dvc.lock naming its hash is published to the result branch at the same point; otherwise the weights sit in S3 unreachable and the next run retrains them regardless.

Before training, the workflow looks for the corresponding result branch. It restores that branch's dvc.lock and pulls train_yolo_best with its dependencies only when the repository contents match, excluding dvc.lock and results/. This lets a fresh runner reuse the saved weights after a downstream failure. Different code or configuration starts a new training run; a failed checkpoint download stops the job before training.

Two smaller consequences, both from review: each result-branch commit is guarded against an empty index (train_yolo_best cached means dvc.lock is unchanged, and git commit treats that as an error), and a concurrency group serialises runs per train_* branch — two overlapping runs force-push the same result branch, so a slower older one could otherwise reset the pointer to its own model. It also releases the GPU instance a superseded run would hold for hours.

Validation

All three scripts build their engine under HF_HUB_OFFLINE=1 — so a warm cache could not mask a constructor download — keeping _PassthroughModel and carrying nb_consecutive_frames and conf_thresh through unchanged. pytest passes. ruff check reports the same 5 pre-existing errors before and after; the workflow YAML parses and the empty-commit guard was exercised on a scratch repository.

A local Git/DVC integration test exercises the actual workflow commands: first run without a checkpoint, retry after downstream failure with an empty cache, rejection after a code change outside DVC's declared dependencies, and failure when checkpoint storage is unavailable. Both pytest tests pass with DVC 3.67.0; Ruff check and format pass for the new test. No GPU or AWS run was launched.

Not fixed here

Upgrading to pyro-engine main — which moved to the standalone pyro_predictor package and a public model — is blocked upstream by huggingface_hub==0.23.1 against this repo's >=1.8.0. pyronear/pyro-engine#412 unpins it.