Matching: signature store interface, optional Postgres store, owner decisions

.github/workflows/tests.yml

@@ -71,9 +71,22 @@ jobs:
           --health-interval 10s
           --health-timeout 5s
           --health-retries 30
+      # signature store of the experimental matching pipeline (tests/test_matching_store.py)
+      postgres:
+        image: postgres:17-alpine
+        env:
+          POSTGRES_PASSWORD: postgres
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U postgres"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 20
     env:
       PY_IMAGE_DEDUP_ELASTICSEARCH_HOST: 127.0.0.1
       PY_IMAGE_DEDUP_ELASTICSEARCH_PORT: 9200
+      PY_IMAGE_DEDUP_TEST_POSTGRES_DSN: postgresql://postgres:postgres@127.0.0.1:5432/postgres
     steps:
       - uses: actions/checkout@v7
       - name: Install poetry
@@ -83,7 +96,7 @@ jobs:
           python-version: "3.14"
           cache: poetry
       - name: Install dependencies
-        run: poetry install --no-interaction
+        run: poetry install --no-interaction --extras postgres
       # ElasticSearchStoreBackend retries forever until Elasticsearch answers, so make sure it is up first
       - name: Wait for Elasticsearch
         run: |

AGENTS.md

@@ -23,6 +23,7 @@ that watches the source directories, exposes Prometheus metrics and serves a web
 
 ```shell
 poetry install                     # install incl. dev deps (pytest)
+poetry install --extras postgres   # + optional Postgres signature store (psycopg) for the matching prototype
 poetry run pytest                  # run unit tests (no Elasticsearch needed, < 1s)
 poetry run py-image-dedup --help   # CLI entrypoint (py_image_dedup.cli:cli)
 docker build . -t py-image-dedup:test   # same build CI runs on every PR
@@ -73,13 +74,16 @@ py_image_dedup/
   matching/                    # EXPERIMENTAL prototype (not wired in): SQLite signatures (SHA-256, pixel hash,
                                # pHash/dHash), staged grouping + confidence, burst veto, union-find clustering
                                # (see docs/matching-prototype.md), benchmark.py: `python -m py_image_dedup.matching.benchmark`
+    store.py                   # SignatureStore interface, SqliteSignatureStore (default, own DB file, separate from
+                               # the state store), open_signature_store(url): sqlite:///... | postgresql://...
+    store_postgres.py          # PostgresSignatureStore, optional `postgres` extra (psycopg 3, imported lazily)
   util/                        # file + image helpers, echo()
 web/                           # Vue 3 + Vite + TypeScript single page app (no component library)
   src/api.ts, src/types.ts     # typed client for the JSON API (keep in sync with py_image_dedup/web/app.py)
   src/views/                   # Dashboard, Groups, GroupDetail, Run, Actions, Config
   src/polling.ts               # polling composable (status: 1s while busy, 3s idle; paused in hidden tabs)
 tests/                         # pytest/unittest tests, test config + sample images
-docs/                          # design notes (matching-prototype.md: benchmark results, open questions)
+docs/                          # design notes (matching-prototype.md: benchmark results, owner decisions, store backends)
 docker/entrypoint.sh           # drops privileges to PUID/PGID
 py_image_dedup_reference.yaml  # documented reference configuration
 ```
@@ -139,6 +143,12 @@ py_image_dedup_reference.yaml  # documented reference configuration
   docker run -d -p 9200:9200 -e discovery.type=single-node -e xpack.security.enabled=false elasticsearch:7.17.28
   poetry run pytest -m integration   # host/port via PY_IMAGE_DEDUP_ELASTICSEARCH_HOST / _PORT
   ```
+  The Postgres signature store tests (`tests/test_matching_store.py`) are integration tests too and are skipped
+  unless `PY_IMAGE_DEDUP_TEST_POSTGRES_DSN` is set (needs the `postgres` extra), CI runs them with a Postgres service:
+  ```shell
+  docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres:17-alpine
+  PY_IMAGE_DEDUP_TEST_POSTGRES_DSN=postgresql://postgres:postgres@127.0.0.1:5432/postgres poetry run pytest -m integration
+  ```
   Tested Elasticsearch versions: 7.9.2 and 7.17.28 (both run in CI). Only Elasticsearch 7.x is supported
   (the client is pinned to `elasticsearch = "^7"`, 8.x is untested).
 - Prioritization/selection logic (`_select_images_to_delete`, `_sort_by_quality_descending`, `_quality_criteria`)
@@ -160,7 +170,7 @@ GitHub Actions (`.github/workflows/`):
 - `dockerimage-latest.yml` / `dockerimage-release.yml`: build (buildx + QEMU, `linux/amd64` + `linux/arm64`) and push
   `markusressel/py-image-dedup` on master / tags. Every binary dependency needs a cp314 `manylinux` aarch64 wheel,
   otherwise the emulated arm64 build compiles it from source.
-- `tests.yml`: unit tests, web UI lint + type-check + build, and integration tests (Elasticsearch service container)
+- `tests.yml`: unit tests, web UI lint + type-check + build, and integration tests (Elasticsearch + Postgres service containers)
   on PRs and pushes to master.
 - `codespell.yml`: spelling (config in `pyproject.toml`, ignores in `.codespellignore`).
 - `codeql-analysis.yml`: CodeQL.

TODO.md

@@ -23,14 +23,26 @@ Replaces `image-match` + Elasticsearch. Goal: group all copies of a photo with h
 configurable prioritization rules pick the best one.
 
 - [x] Prototype + benchmark: `py_image_dedup/matching/`, findings in `docs/matching-prototype.md` (#534)
-- [ ] Answer the open questions in `docs/matching-prototype.md` (variants, burst window, rotations, crops, HEIC/RAW, migration)
+- [x] Answer the open questions in `docs/matching-prototype.md` (variants, burst window, rotations, crops, HEIC/RAW,
+      migration, store): see "Decisions" there
 - [ ] Run the benchmark read-only on a real library sample, calibrate thresholds and the auto-act confidence
-- [ ] Introduce a `SignatureBackend` / store interface so backends can be swapped and compared
+- [x] Signature store interface (`SignatureStore`, `open_signature_store(url)`), SQLite in its own file (default)
+- [x] Optional Postgres signature store (`matching/store_postgres.py`, `postgres` extra, integration tests in CI)
+- [ ] Wire the signature store URL into `config.py` (e.g. `matching.store: sqlite:////data/signatures.db`)
+- [ ] Introduce a `SignatureBackend` interface so image-match and the hash pipeline can be swapped and compared
+- [ ] Side-by-side comparison: `matching.backend: image-match|hashes|both`, compare the groups on the owner's
+      dataset in the web UI (`matcher` column)
+- [ ] Rotation/mirror matching: hash the 4 rotations (or 8 dihedral transforms) for copies without orientation tag
+- [ ] Variants (grayscale, edited) and crops: keep as matches, but always route to review instead of auto-acting
+- [ ] Burst window as a decision factor (confidence/review) instead of a hard veto
+- [ ] Crop detection with embeddings (SSCD via onnxruntime, opt-in); crops usually, but not always, win (review)
 - [ ] SQLite store for hashes, metadata, groups, decisions and the action log (single file on a PVC)
   - [x] Groups, candidates, decisions (deciding rule), user overrides, runs and the action log:
         `py_image_dedup/state` (web UI backend, backend independent `matcher` column)
-  - [ ] Hashes and image metadata (prototype: `matching/store.py`)
-- [ ] Decode & normalise: apply EXIF orientation before hashing, support HEIC (`pillow-heif`) and RAW previews (`rawpy`)
+  - [x] Hashes and image metadata: `matching/store.py` (own SQLite file, separate from the state store),
+        optional Postgres (`matching/store_postgres.py`)
+- [ ] Decode & normalise: apply EXIF orientation before hashing (prototype done)
+- [ ] HEIC (`pillow-heif`) and RAW previews (`rawpy`) as optional extras (not needed by the owner, later)
 - [ ] Extract matching metadata: `DateTimeOriginal` (+ sub-seconds), camera model, `ImageUniqueID` /
       Apple content identifier, `Software` (edited) (prototype done, Apple content identifier missing)
 - [ ] Staged grouping, each match tagged with confidence + reason:

docs/matching-prototype.md

@@ -2,7 +2,7 @@
 
 Status: **experimental prototype**, not used by the deduplicator, the CLI or the daemon. Production still uses
 `image-match` + Elasticsearch. This document summarises the design, the benchmark results on `tests/images/`
-and the decisions that are still open. See the "Matching & ranking pipeline" section of `TODO.md` for the plan.
+and the owner's decisions on the open questions. See the "Matching & ranking pipeline" section of `TODO.md` for the plan.
 
 ## What the prototype does
 
@@ -13,7 +13,8 @@ Package `py_image_dedup/matching/`:
 | `decoding.py`   | Pillow decode, JPEG draft mode (DCT scaling to >= 512 px), EXIF orientation applied (`ImageOps.exif_transpose`), conversion to RGB               |
 | `metadata.py`   | `DateTimeOriginal` + `SubSecTimeOriginal`, camera make/model, `ImageUniqueID`, `Software`                                                        |
 | `signatures.py` | SHA-256 of the file, SHA-256 of the decoded preview pixels, pHash + dHash (`imagehash`, 64 bit), grayscale flag, lazy full resolution pixel hash |
-| `store.py`      | `SignatureStore`: SQLite (stdlib `sqlite3`), one row per file, incremental (size + mtime + `SIGNATURE_VERSION`)                                  |
+| `store.py`      | `SignatureStore` interface + `SqliteSignatureStore` (stdlib `sqlite3`, default), one row per file, incremental (size + mtime + `SIGNATURE_VERSION`), `open_signature_store(url)` factory |
+| `store_postgres.py` | `PostgresSignatureStore` (psycopg 3, optional `postgres` extra), same record semantics                                                     |
 | `search.py`     | All-pairs Hamming distance search, numpy (`bitwise_count` over `uint64`), block-wise to bound memory                                             |
 | `grouping.py`   | Staged matching with reason + confidence, burst veto, constrained union-find clustering                                                          |
 | `pipeline.py`   | `MatchingPipeline`: analyse (thread pool) into the store, match, cluster                                                                         |
@@ -158,25 +159,54 @@ The SQLite store needs roughly 0.5 KB per file including indexes (~20 MB for 40k
    can show "why".
 6. Keep the full resolution pixel hash lazy: it costs 79 ms/file and is only needed for a handful of pairs.
 
-## Open questions for the owner
-
-1. **Are grayscale and edited versions duplicates you want removed**, or "variants" to keep next to the
-   original? The test data puts them in the same group; the prototype matches them (with a note).
-2. **Burst window**: 10 s is a guess. Should *any* different capture time of the same camera veto
-   (strictest; risk: tools that shift capture times, e.g. time zone fixes, would then never match their source),
-   or only a short window?
-3. **Rotated/mirrored copies without orientation tag**: worth the 4-8x hash cost, or ignore?
-4. **Crops**: is opt-in embedding search (SSCD + onnxruntime, ~100 MB model) wanted, or out of scope?
-5. **HEIC/RAW**: are there HEIC (iPhone) or RAW files in the library? Then `pillow-heif` / `rawpy` previews
-   are needed before switching (the prototype only handles what Pillow decodes).
-6. **Migration**: switch hard (full re-analysis, remove Elasticsearch) or run both backends side by side for a
-   while and compare the groups in the web UI (`matcher` column)?
-7. Should the SQLite signature store live in the same database file as the state store (one PVC file) or in a
-   separate one (can be deleted/rebuilt independently)?
+## Signature store backends
+
+The pipeline only talks to the `SignatureStore` interface (`store.py`): upsert/get records keyed by path
+(size, mtime, hashes, metadata), `get_all()`/`iter_all()` for the search, `get_file_states()` + `needs_analysis()`
+for incremental runs, `remove()`/`remove_missing()` for deleted files, `set_full_pixel_hash()` for the lazy
+pixel hash and `schema_version`. Stores are persistence only: the Hamming distance search always loads all
+signatures and runs in numpy (`search.py`), so no database specific similarity search is needed.
+
+`open_signature_store(url)` creates a store from a URL like setting (to be wired into `config.py` later):
+
+| URL                                            | Store                                                             |
+|------------------------------------------------|-------------------------------------------------------------------|
+| `sqlite:///relative/signatures.db`             | SQLite, path relative to the working directory (SQLAlchemy style) |
+| `sqlite:////data/signatures.db`                | SQLite, absolute path                                             |
+| `/data/signatures.db` (no scheme)              | SQLite file                                                       |
+| `sqlite://`, `sqlite:///:memory:`, `:memory:`  | SQLite in memory (tests)                                          |
+| `postgresql://user:password@host:5432/db`      | Postgres (`postgres://` works too, any libpq URI incl. `?sslmode=...`) |
+
+- **SQLite** (default) lives in its **own database file**, separate from the web UI state store
+  (`py_image_dedup/state`), so it can be deleted and rebuilt independently (everything in it can be recomputed).
+- **Postgres** needs the optional extra: `pip install py-image-dedup[postgres]` / `poetry install --extras postgres`
+  (`psycopg[binary]` 3.x, imported lazily, the package works without it). Tables are prefixed with `matching_`
+  (`matching_files`, `matching_meta`), so a database shared with other applications works; schema creation is
+  serialised with an advisory lock. All signatures are streamed with a server side cursor.
+- **Schema version**: `SCHEMA_VERSION` in `store.py` (SQLite `PRAGMA user_version`, Postgres `matching_meta`).
+  The store is a cache, so a store with another schema version is dropped and rebuilt (full re-analysis).
+  `SIGNATURE_VERSION` (`models.py`) triggers re-analysis of single records without touching the schema.
+- Tests: `tests/test_matching_store.py` runs the same contract tests against SQLite (always) and Postgres
+  (`integration` marker, `PY_IMAGE_DEDUP_TEST_POSTGRES_DSN`, Postgres service container in CI).
+
+## Decisions (owner, 2026-10-31)
+
+1. **Grayscale/edited versions** are variants to keep, unless the original does not look nice on its own; then
+   the original can be deleted. That is a human decision: such groups go to review, never automatic action.
+2. **Burst window**: keep a burst window as a decision factor; not a hard veto on any capture time difference.
+3. **Rotated/mirrored copies without orientation tag**: yes, match them, even if it is expensive
+   (4-8x hash storage and search time).
+4. **Crops**: yes to AI embeddings for crops. The crop should win in most cases, but not all (review).
+5. **HEIC/RAW**: possible, but not relevant to the owner (JPEG/PNG matter). Others may care, so support them
+   later as optional extras.
+6. **Migration**: run both backends side by side if easily possible and compare the results on the owner's dataset.
+7. **Signature store**: a separate SQLite file for the hashes (not the state store database); if feasible also
+   support Postgres (the owner runs a cluster instance). Done, see "Signature store backends".
 
 ## Not done (intentionally)
 
 - No wiring into `ImageMatchDeduplicator`, the daemon, CLI, config or Docker image.
-- No `SignatureBackend` interface yet: the prototype API (`MatchingPipeline.analyse/match`) should inform it.
+- No `SignatureBackend` interface (image-match vs. hashes) yet: the prototype API (`MatchingPipeline.analyse/match`)
+  should inform it. The store itself is abstracted (`SignatureStore`).
 - Quality signals for prioritization (format tier, JPEG quality estimate, sharpness) are not part of this
   prototype.