Traefik dynamic config is written non-atomically, briefly dropping all app routes

[ClaydeCode]

What happens

write_traefik_dyn_config() rewrites the single dynamic-config file that carries the routers for every installed app, and the write is not atomic:

https://github.com/FreeshardBase/freeshard/blob/main/shard_core/service/app_installation/util.py#L157-L162

async def _write_to_yaml(spec: pydantic.BaseModel, output_path: Path):
    output_path.parent.mkdir(exist_ok=True, parents=True)
    async with aiofiles.open(output_path, "w") as f:
        await f.write("# == DO NOT MODIFY ==\n# this file is auto-generated\n\n")
        await f.write(yaml.dump(spec.model_dump(exclude_none=True)))

Mode "w" truncates the file immediately, and the content is then written in two separate awaited writes. Traefik's file provider watches this path and re-reads on change. A reload landing inside that window sees either an empty file or a config with the header and no http: section, i.e. no routers at all — and Traefik answers every request for an unknown router with its own 404.

The file is rewritten on every install and uninstall (worker.py:150) and at startup (app_factory.py:87), so any app-install activity can knock out routing for all apps on the shard for the duration of the window.

Why it matters

The user-visible symptom is a 404 on an app that was working a moment ago, typically right after the app-starting splash screen. That 404 also kills the splash's own JS reload, so the page does not recover on its own and the user sees a dead tab rather than an app that starts a few seconds later. It is intermittent by nature, which makes it hard to attribute.

It also confuses any automated check: a 404 on an app URL currently has two very different meanings — "this app has no route because it is queued or in ERROR" (NOT_ROUTABLE_STATUS, util.py:27) and "routing is briefly gone for everyone".

Suggested fix

Write to a temporary file in the same directory and os.replace() it into place. The rename is atomic on POSIX, so Traefik either reads the previous complete config or the next one, never a partial file:

async def _write_to_yaml(spec: pydantic.BaseModel, output_path: Path):
    output_path.parent.mkdir(exist_ok=True, parents=True)
    tmp_path = output_path.with_suffix(output_path.suffix + ".tmp")
    async with aiofiles.open(tmp_path, "w") as f:
        await f.write("# == DO NOT MODIFY ==\n# this file is auto-generated\n\n")
        await f.write(yaml.dump(spec.model_dump(exclude_none=True)))
    os.replace(tmp_path, output_path)

Same filesystem, same directory, so the rename cannot fail with EXDEV.

How it was found

Reading the routing path while designing an unattended smoke test for app-update bundles in the app-repository, after the intermittent 404 was reported from manual testing. Not reproduced deliberately — the race is timing-dependent — so treat the mechanism as strongly suspected rather than confirmed. The non-atomic write is not in doubt either way, and the fix is worth making on its own.

[ClaydeCode]

Observed on a live shard while building an unattended smoke test for app-update bundles, which makes this more than a suspected race.

The script paired with a fresh trial shard, listed the installed apps successfully, uninstalled the three pre-installed ones (immich, paperless-ngx, filebrowser), and then the very next GET /core/protected/apps returned 404 — three seconds after the identical call had returned 200.

The blast radius is wider than app routes. compile_config emits the core's own routers into the same file it rewrites:

https://github.com/FreeshardBase/freeshard/blob/main/shard_core/service/traefik_dynamic_config.py#L41-L70

shard_core_public, shard_core_private, shard_core_management and web-terminal all live there. So while the file is truncated and being rewritten, the shard has no route to its own API or to the web terminal either, and Traefik answers 404 for all of it. Three uninstalls in a row means three rewrites in a few seconds, and a call landing in one of those windows fails.

That also explains the shape of the original report: a 404 arriving right after the app-starting splash, which the splash's own JS reload cannot recover from because the reload lands on a route that does not exist for that moment.

The client-side mitigation (retrying core calls on 404/5xx) is in the app-repository smoke test, but it is a workaround for callers. The atomic write via os.replace() remains the actual fix, and it also protects the web terminal, which no retry loop covers.