Skip to content

Feature: cache the checksums of files that Igir writes - #2427

Open
sdornan wants to merge 1 commit into
emmercm:mainfrom
sdornan:claude/rom-moving-rehash-rm3s2y
Open

Feature: cache the checksums of files that Igir writes#2427
sdornan wants to merge 1 commit into
emmercm:mainfrom
sdornan:claude/rom-moving-rehash-rm3s2y

Conversation

@sdornan

@sdornan sdornan commented Aug 14, 2026

Copy link
Copy Markdown

Problem

The file cache is keyed by absolute file path (FileCache#getCacheKey), so a ROM that Igir copies or moves to a new location has no cached result at its new path and has to be read again in full on the next run. Nothing in the write path ever touched the cache.

This is avoidable: Igir already knows the checksums of every file it writes, and currently throws that knowledge away.

ROM headers, file signatures, and paddings already survive a move, since they're keyed by checksum rather than path. Only plain-file checksums and archive entry checksums are path-keyed — and those are the expensive ones.

Changes

After a successful write, cache what we already know at the output path:

  • Raw copies and moves — the output file's bytes are the input file's bytes, so the input file's checksums are also the output file's checksums.
  • Zip writing — Igir knows the checksums of every entry it wrote, so they're cached as the output zip's archive entries. Without this, copy --zip / move --zip would still pay a full decompress on the next run.

New FileCache#setFileChecksums and FileCache#setArchiveChecksums, exposed through FileFactory, called from CandidateWriter after writeRawSingle and writeZip succeed.

Safety

  • Nothing is cached for writes that change the file's contents — header removal, patching, or restoring padding to trimmed ROMs — because the input's checksums no longer describe the output. Guarded by CandidateWriter.isTransformedWhenWritten.
  • Checksums aren't cached when the file on disk isn't the size we expected to write.
  • Caching failures are caught and trace-logged; they can't fail a write.
  • Cached results for written files are only as trustworthy as the write itself. Previously a silently-corrupt write would be caught by the next run's rehash; now it wouldn't be. This is the same trust Igir already places in an unverified copy when it deletes the source file in move mode, and test still verifies before anything is trusted. This is called out in the docs rather than left implicit.

Verification

Unit tests for the new cache methods, plus writer-level tests covering copy and move for both raw and zip output, and the header-removal negative case.

End-to-end against the repo's fixtures with a real cache file, using dir2dat to force a full scan-and-hash of the output directory:

copy move
cache keys for output paths after write 11 11
cache misses on a full rescan of the output 0 0
cache file changed by the rescan no no

The cache file is byte-identical before and after the rescan, so nothing was recomputed. Same result for the --zip path.

Control run with the source changes stashed: the copy produced 0 output-path cache keys and the following rescan had to compute all 11, confirming the difference is real.

Not included

Re-keying an existing archive's cached entry list when a whole .zip is moved as-is (an ArchiveFile input). That's a different mechanism from the seed-on-write approach here, and a narrower case — happy to follow up separately.

Pull request checklist

  • Validate your change works as expected locally by running Igir (not just the unit tests)
  • Unit tests have been added to cover your change
  • npm test has been run locally for your change (91 files, 3269 tests passing, lint clean)
  • Appropriate docs have been added or changed for your change (docs/advanced/file-cache.md)
  • Your branch has no conflicts with the main (destination) branch — branched from c5d485e, please confirm this is current with upstream main

The file cache is keyed by absolute file path, so a ROM that Igir copies
or moves to a new location has no cached result at its new path, and has
to be read again in full on the next run. This is the expensive part of
scanning, and it's entirely avoidable: Igir already knows the checksums
of every file it writes.

Cache what we already know at the output path after a successful write:

- Raw copies and moves write the input file's bytes unchanged, so the
  input file's checksums are also the output file's checksums.
- Zip writing knows the checksums of every entry it wrote, so cache them
  as the output zip's archive entries.

Results aren't cached for writes that change the file's contents, since
the input file's checksums no longer describe the output file. That
covers header removal, patching, and restoring padding to trimmed ROMs.
Checksums also aren't cached when the file on disk isn't the size we
expected to write.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FjyVaGcxiAUaAfjxGH2cqg
@github-actions github-actions Bot changed the title Feat: cache the checksums of files that Igir writes Feature: cache the checksums of files that Igir writes Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants