_demote_unverified_owner_email in shard_core/database/db_snapshot.py re-implements the data rule of migration 0005 inside the restore path, and identifies which era a snapshot came from by sniffing its shape — "recognisable by its identity rows still carrying an email column".
That function should be deleted once this issue is done. It is the first of its kind, not the last: as written, every future migration that carries a data rule needs another sniffer beside it, each guessing at the snapshot's age from whatever column happens to be diagnostic. It was the right call to unblock #221 and it is the wrong thing to keep.
Root cause
Two things, and the second is the structural one.
The snapshot records no schema version. db_snapshot.py deliberately excludes yoyo's bookkeeping tables (_YOYO_TABLE_MARKER), on the reasoning that they are "recreated by the migrations that run on the fresh database". So a snapshot is a bag of rows with no statement of which schema it belongs to, and a restore has nothing to check.
Migrations run before the snapshot is loaded. init_database() calls migrate() and only then restore_db_snapshot(). The schema is already current, on an empty database, when the rows arrive — so no migration can ever act on snapshot data. That is why the rule has to be re-applied by hand afterwards.
Worth stating plainly, because it is easy to miss: there is no user-facing restore path. restore_db_snapshot() has exactly one caller, app startup. Restoring means putting the file-level backup in place and letting the shard boot. So this is startup reconciliation, not an operation someone invokes.
What everyone else does
None of the comparable projects make the restore path schema-aware. They split two ways, and both keep migration logic in exactly one place:
- Refuse the cross-version restore. Sentry: "We strongly recommended that you restore your backup on the same version of Sentry on a fresh install (empty database but migrations are run) … Otherwise, you are very likely to hit errors and may corrupt your database." Paperless-ngx is blunter: "You cannot import the export generated with one version of paperless in a different version of paperless. The export contains an exact image of the database, and migrations may change the database layout."
- Load first, then migrate. AFFiNE, Umami and Matomo restore a dump that carries the schema and the migration-tracking table, then start the app so the normal chain runs.
Freeshard does neither: row-level JSON into an already-migrated schema.
Two stages
Stage 1 — make the version known and the mismatch visible. Record the applied migration version in the snapshot, and on restore compare it with the running schema. This is where Sentry's rule becomes available to us, and Freeshard is unusually well placed to use it: the controller keeps every historical core version (data/core-versions/v4 … v29), so "restore on the version it came from, then upgrade normally" is operationally available here in a way it is not for most projects.
The design question this stage has to answer, and it is not small: what does a mismatch do at boot? Failing to start strands a shard; skipping the snapshot silently boots it empty, which is worse. Neither is obviously right, and the answer probably differs between "older than current" and "newer than current".
Stage 2 — invert the order. Migrate the schema to the snapshot's recorded version, load the rows, then run the remaining migrations. Migration logic then lives once, in the migration chain, and old snapshots stay usable across versions. This is the real fix; stage 1 is what makes it safe to reach.
Done when
_demote_unverified_owner_email is gone, and a snapshot from a pre-0005 shard restores correctly without any code in db_snapshot.py knowing what 0005 did.
Context
Found while reviewing #221.
_demote_unverified_owner_emailinshard_core/database/db_snapshot.pyre-implements the data rule of migration 0005 inside the restore path, and identifies which era a snapshot came from by sniffing its shape — "recognisable by its identity rows still carrying an email column".That function should be deleted once this issue is done. It is the first of its kind, not the last: as written, every future migration that carries a data rule needs another sniffer beside it, each guessing at the snapshot's age from whatever column happens to be diagnostic. It was the right call to unblock #221 and it is the wrong thing to keep.
Root cause
Two things, and the second is the structural one.
The snapshot records no schema version.
db_snapshot.pydeliberately excludes yoyo's bookkeeping tables (_YOYO_TABLE_MARKER), on the reasoning that they are "recreated by the migrations that run on the fresh database". So a snapshot is a bag of rows with no statement of which schema it belongs to, and a restore has nothing to check.Migrations run before the snapshot is loaded.
init_database()callsmigrate()and only thenrestore_db_snapshot(). The schema is already current, on an empty database, when the rows arrive — so no migration can ever act on snapshot data. That is why the rule has to be re-applied by hand afterwards.Worth stating plainly, because it is easy to miss: there is no user-facing restore path.
restore_db_snapshot()has exactly one caller, app startup. Restoring means putting the file-level backup in place and letting the shard boot. So this is startup reconciliation, not an operation someone invokes.What everyone else does
None of the comparable projects make the restore path schema-aware. They split two ways, and both keep migration logic in exactly one place:
Freeshard does neither: row-level JSON into an already-migrated schema.
Two stages
Stage 1 — make the version known and the mismatch visible. Record the applied migration version in the snapshot, and on restore compare it with the running schema. This is where Sentry's rule becomes available to us, and Freeshard is unusually well placed to use it: the controller keeps every historical core version (
data/core-versions/v4…v29), so "restore on the version it came from, then upgrade normally" is operationally available here in a way it is not for most projects.The design question this stage has to answer, and it is not small: what does a mismatch do at boot? Failing to start strands a shard; skipping the snapshot silently boots it empty, which is worse. Neither is obviously right, and the answer probably differs between "older than current" and "newer than current".
Stage 2 — invert the order. Migrate the schema to the snapshot's recorded version, load the rows, then run the remaining migrations. Migration logic then lives once, in the migration chain, and old snapshots stay usable across versions. This is the real fix; stage 1 is what makes it safe to reach.
Done when
_demote_unverified_owner_emailis gone, and a snapshot from a pre-0005 shard restores correctly without any code indb_snapshot.pyknowing what 0005 did.Context
Found while reviewing #221.