Repository navigation
Add TIBX archive parser - #30
falklindner wants to merge 40 commits into
Conversation
Add a parser for the Acronis CyberBackup/CyberProtect TIBX
("archive3") backup format as a dissect.archive.tibx subpackage.
The parser handles the 4 KiB page store, CRC-32C verification,
ARCH commit-root superblocks, TLV directory, LSM B-tree index
(data_map / segment_map with mem-tree and on-disk ctree layers),
zstd and linked-LZ4 decompression, AES-256-CBC decryption, and
full incremental/differential chain reconstruction with
recovery-point selection.
Volume reconstruction supports NTFS, FAT32, exFAT, FAT12/16 and
ext boot-sector detection for accurate volume sizing, split
archives, version sets (TLV[18] file table with compacted bases),
multi-disk archives, and tombstone masking for backup-version
cleanup.
Validated byte-for-byte against real Acronis True Image 2026
(Build 42980) output across single-partition, multi-partition,
whole-disk (MBR and GPT), multi-disk, split, incremental,
differential, and encrypted archives.
Closes fox-it#20
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a parser for the Acronis CyberBackup/CyberProtect TIBX
("archive3") backup format as a dissect.archive.tibx subpackage.
The parser handles the 4 KiB page store, CRC-32C verification,
ARCH commit-root superblocks, TLV directory, LSM B-tree index
(data_map / segment_map with mem-tree and on-disk ctree layers),
zstd and linked-LZ4 decompression, AES-256-CBC decryption, and
full incremental/differential chain reconstruction with
recovery-point selection.
Volume reconstruction supports NTFS, FAT32, exFAT, FAT12/16 and
ext boot-sector detection for accurate volume sizing, split
archives, version sets (TLV[18] file table with compacted bases),
multi-disk archives, and tombstone masking for backup-version
cleanup.
Validated byte-for-byte against real Acronis True Image 2026
(Build 42980) output across single-partition, multi-partition,
whole-disk (MBR and GPT), multi-disk, split, incremental,
differential, and encrypted archives.
Closes fox-it#20
Co-Authored-By: Matthias Niedermaier <matthias.niedermaier@hs-augsburg.de>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…archive into feature/tibx-parser
|
Very nice work, and thanks for sharing it! I haven't started reviewing yet, but I pushed a commit filling in a few of the unknown fields based on some of our own reversing efforts. |
Reduce the random-access parsing gap versus a raw disk image with caching and batched I/O on the segment read path. Output is byte-identical: every real-sample volume hash and file check is unchanged. - Segment cache: replace the fixed 32-entry limit with a 256 MiB byte-budget LRU, so random-access workloads (MFT, registry hives) that touch thousands of segments stop thrashing. - Multi-page segments: read the contiguous continuation pages in a single I/O instead of one read per page, and retain the segment header page so it is not read twice. - Add a small LRU page cache to the page store. - Use the C lz4 library for the linked-LZ4 decoder when it is importable (byte-identical, including the growing-dictionary case), falling back to the pure-Python decoder otherwise; add lz4 to the full extra. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Treating "a keymap tree with records exists" as proof of encryption over-reports: an archive whose keymap holds no password-wrapped key would prompt for a password that could never unwrap anything. Confirm the keymap actually carries a password-wrapped data key by locating one structurally -- a known algorithm id, a plausible PBKDF2 iteration exponent, a full salt and an AES-block-multiple wrapped key. No password is involved and no key is derived, so this stays cheap enough to run on every open. AES-GCM archives are still reported as encrypted, since they are: they now fail with a clear "not supported" instead of being mistaken for plaintext. Detection never raises -- a missed one still surfaces as an explicit InvalidPasswordError from the segment reader rather than a silent misread. The keymap region is extracted once and shared with unwrap_data_key, and pycryptodome is imported where it is used rather than at module import time, so detection no longer pulls in an optional dependency. Validated against the real corpus: the encrypted sample is detected and still decrypts file-for-file, with no false positive on any of the ten unencrypted archives. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Regarding the last commit, can the TIBX encryption not be detected via the |
Good point — checked against the full real-sample corpus (12 archives): encr_alg in the ARCH superblock agrees with the keymap detection everywhere (AES_256_CBC on the encrypted sample, NONE on the rest), so yes, it does indicate encryption. The keymap scan answers a narrower question though: not "is this encrypted" but "does a password-wrapped data key exist" — which is what drives the password prompt / keychain lookup. Acronis also supports certificate-wrapped keys (format byte 0x02 next to the 0x01 we locate), where encr_alg would be set but asking for a password can't help. Detection via the keymap also can't disagree with unlock(), since both parse the same structure, and it doesn't depend on the superblock byte layout, which we've only verified for header_version 8. The two compose nicely, and encr_alg is essentially free: happy to use encr_alg != NONE as the archive-level signal and the keymap scan to decide password vs. certificate — that would let a certificate-wrapped archive fail with "encrypted, but not password-protected" instead of a generic missing-password error. |
|
Is there anything I can do to support the PR? |
yunzheng
left a comment
There was a problem hiding this comment.
Hi @falklindner, sorry for the late reply, but I was on holiday.
I left some general feedback.
Also, regarding the synthetic tests, while I do think that's awesome, I think nothing beats real data. It would be nice to add tibx archives created by Acronis itself, for example:
- simple tibx archive
- password protected archive
- full disk vs files
- other edge cases that we want to test
I'm also curious how much you're already using this implementation for your own needs.
Reuse crc32c from dissect.util
dissect/archive/vbk.py already imports dissect.util.hash.crc32c, and the
module is present in the pinned dissect.util floor (3.22), so the vendored
Castagnoli table had no reason to exist. Delete tibx/crc32c.py and move
page_crc32c next to its callers in page.py, folding its test into
test_page.py. The new implementation chains the three runs through the
crc32c value argument instead of building a zeroed 4 KiB copy per page.
Parse the wrapped data key as a structure
Add a wrapped_key struct to c_tibx.py and use it in crypto.py in place of
the hardcoded offset arithmetic, in the module docstring as well as the
code. The scan for the key keeps its cheap format-byte prefilter, so the
structure is only built for actual candidates. tests/_synth.py builds its
fixture blob from the same struct.
Parse page envelopes as structures on the cold paths
SuperBlock, PageStore.__init__ and page_type now go through
c_tibx.page_header / arch_superblock. The three whole-archive scan loops
(live_root, commit_roots, verify) keep direct byte indexing: cstruct costs
~3.99 us per envelope against ~0.24 us, which on a 100 GiB archive is ~100 s
of header parsing instead of ~6 s. A note in page.py records the reason.
SuperBlock now reports a truncated buffer as InvalidArchiveError rather than
letting cstruct's EOFError escape.
Compose encr_alg with the keymap scan
TIBX.encrypted is now the archive-level question, answered by the
superblock's encr_alg with the keymap scan as a fallback for header
versions where encr_alg is not verified. The narrower question moves to
TIBX.password_protected, and unlock() rejects a certificate-wrapped archive
with "encrypted, but not password-protected" instead of a generic wrong
password error.
Fix references and lint
Describe the provenance of the format definitions and the synthetic
fixtures inline, in place of the references to files that are not part of
this repository. Exclude the generated c_tibx.pyi from ruff, which it has
been failing (E501, PYI042) since it was added.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V2NzRCqXtY9bjPJCj5qVp2
The LSM engine, the format findings the structure definitions encode and the synthetic test fixtures all derive from acronis-tibx, a private research project by mniedermaier that is no longer publicly available. Name that source in each of the three docstrings instead of pointing at a THIRD_PARTY_NOTICES.md and a docs/FINDINGS.md that are not part of this repository. Being a private and now unavailable repository there is no URL to link, so the credit is given inline. A THIRD_PARTY_NOTICES.md carrying the original license notice still needs to be added from a local copy of that repository. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V2NzRCqXtY9bjPJCj5qVp2
The LSM engine was credited to acronis-tibx, but acronis-tibx has no LSM engine of its own -- it vendored the MIT-licensed acronis-tib-reader (github.com/TreadingTheTiber/acronis-tib-reader), which is still publicly available. Copyright there is "the tibread contributors", not mniedermaier. Re-checking every module rather than only the three raised in review turned up four more derived files carrying no credit at all: maps.py, segment.py, crypto.py and codecs.py, plus page.py. Add THIRD_PARTY_NOTICES.md, organised by upstream since both projects are woven through the package, list it in license-files, and reference it from every derived module. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Benchmarking the page-envelope struct-vs-offset tradeoff end to end, rather than timing the parse call in isolation, showed the offsetting only pays in one of the three loops it was applied to. Measured on a 949 MiB archive (242,977 pages), offset vs struct: live_root() 1.02x -- stops at the newest ARCH, 3 pages in practice verify() 1.06x -- dominated by CRC-32C commit_roots() 1.92x -- 859 ms vs 1644 ms live_root() and verify() go back to c_tibx.page_header; commit_roots() keeps the direct indexing, and the note at the top now carries the end-to-end figures instead of the per-call ratio. Chasing verify()'s cost also turned up the wrong crc32c import. dissect.util.hash rebinds its crc32c attribute to the native (Rust) implementation, but "from dissect.util.hash.crc32c import crc32c" resolves through sys.modules to the pure-Python submodule and bypasses the rebind -- identical output at 5.7 MB/s instead of 80.9 MB/s. verify() on that archive drops from 172 s to 15.5 s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
exceptions.py -> exception.py, codecs.py -> codec.py, maps.py -> map.py, and tests/test_codecs.py -> tests/test_codec.py alongside. Import-only change; no behaviour is touched. codec.py also stops shadowing the stdlib codecs module. tests/test_exceptions.py and dissect/archive/exceptions.py are left alone -- they predate this branch and cover the package-level exceptions, so renaming them belongs to the wider migration rather than to this PR. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A 512-page LRU sat in front of PageStore.page() on the theory that LSM tree walks and segment header pages get revisited. Measured against the real-sample corpus it hits exactly zero times, on every archive tried: 3000 scattered 4 KiB reads produce only 40 to 500 page() calls in total. Two layers above absorb everything first. TIBX's 256 MiB segment cache catches repeat segment reads, and segment payloads go through read_run(), which never consulted the LRU. What is left for page() is one header page per distinct segment, read once. The LRU only earned hits with the segment cache shrunk until it thrashed (12.6% at 4 MiB, 66% at 256 KiB), which is not a configuration anyone runs. read_run()'s docstring also claimed it honoured pages already resident in the LRU; it never did. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lz4_block_decompress now dispatches across three decoders instead of two:
1. the C lz4 package, the only one taking a dictionary, so it serves every block
2. dissect.util (Rust-backed when built) for blocks needing no dictionary --
the first block of every chain, and whole LZ4 segments
3. the in-tree decoder, now reached only for dictionary blocks without lz4
dissect.util cannot take the dictionary blocks: neither backend exposes a
dictionary parameter, and 931 of 931 non-first blocks in the real-sample corpus
back-reference earlier output, up to 10,690 bytes back. Dropping the dictionary
makes 6 of 17 real archives fail to decode outright.
Where it does apply it is worth having: measured over the 307 no-dictionary
blocks in that corpus it is byte-identical to the in-tree decoder and 14x faster
than it (8.2 ms vs 116 ms), which is what runs when lz4 is not installed.
The two dissect.util backends raise different types on malformed input -- the
Rust one ValueError, the pure-Python one CorruptDataError, which is not a
ValueError -- so both are normalised to CorruptArchiveError.
Verified: with the C package disabled, all 17 real archives decode byte-identical
to the accelerated path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Changes found by testing against real Acronis Cyber Protect 15 archives, checked against acrocmd's own listings: - recovery_points() reads the slices tree (TLV[5]), one record per backup, instead of counting ARCH commit roots. Selection filters extents by slice id. - volumes() leaves out Acronis metadata streams. streams() returns the unfiltered list. - AES-GCM segments are decrypted (IV || tag || ciphertext). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replace the remaining fixed-offset reads with structs: the ARCH header, TLV entries, linked-LZ4 block headers, LSM cell groups, LDIR values and segment_map records. Boot-sector sizing uses its own little-endian c_boot definitions. The synthetic archive builders are built from the same structs. No behaviour change: every stream at every recovery point of the real sample corpus hashes identically before and after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Two behaviour changes in 57fa473, both checked against acrocmd on real Cyber Protect 15 archives. recovery_points() now reads the slices tree, one entry per backup; counting commit roots gave too many. volumes() skips Acronis's own metadata streams, and streams() still lists everything. Older recovery points were verified file by file.
There was a problem hiding this comment.
Since then, volumes() also leaves out allocation bitmaps and the disk's bootstrap region, close() releases the index, and non-overlapping extents are resolved without a sweep. Test results: #30 (comment)
🤖 Generated with Claude Code
There was a problem hiding this comment.
segment_map records now go through segment_map_key / segment_map_value (9b05168). page_count stays the one little-endian field.
There was a problem hiding this comment.
The data_map and segment_map are now decoded in bulk into arrays instead of per-record cstruct objects; a test keeps the precompiled structs in step with c_tibx. segment_map runs are looked up run by run (keep-first) instead of merged. Test results: #30 (comment)
🤖 Generated with Claude Code
There was a problem hiding this comment.
AES-GCM segments (57fa473), as written by Cyber Protect 15; True Image 2026 used AES-CBC: the payload is IV || tag || ciphertext and is not PKCS#7-unpadded.
There was a problem hiding this comment.
A segment's header and continuation pages are now read in one I/O, sized by the segment_map's page count. Test results: #30 (comment)
🤖 Generated with Claude Code
Real samples are ready: 10 archives from Acronis Cyber Protect 15, covering your list (simple, password-protected, whole disk MBR/GPT vs. file-level) plus chains, stored and multi-volume. They're identity-scrubbed, about 6 MB gzipped in total. That's well above the existing |
Nice! Can you put them in LFS under |
Opening the volumes of a large archive took up to a minute and close to 2 GB of memory, nearly all of it spent on the data_map and segment_map records: about a million of each at a few hundred GB. Parsed through c_tibx they cost ~10.7 us a record, and cstruct's int subclasses are tracked by the cyclic GC, so the index also made every full collection pause for most of a second. These two record types are now decoded in bulk with precompiled structs that mirror the c_tibx definitions, which stay the reference (a test keeps the two in step). The segment index is kept in arrays instead of a dict of locators. For an archive with 720k extents, volumes() drops from 40 s to 15 s and from 780 to 470 MiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Without the optional C lz4 package, every block of a linked-LZ4 chain but the first went through the in-tree decoder, which copies matches byte by byte, even where dissect.util is built with Rust. dissect.util takes a dictionary since fox-it/dissect.util#141, so these blocks now go through it as well; the in-tree decoder stays for older versions. The dictionary handed on is the last 64 KiB of output, as far back as a match can reach, instead of all output so far. On a synthetic archive with 1.1M extents, loading the index without the C package takes 6 s instead of 12 s, about the same as with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Resolving the extents of a volume took 24 of the 29 s volumes() needs on a synthetic archive with 1.1M extents: interval painting does dictionary and tuple work at every extent boundary. Extents that do not overlap, as in a full backup, are now taken as they are. Overlapping ones go through a sweep in offset order that keeps the extents covering the current offset in a heap, most recent on top. A randomised test checks both against a brute-force reference. The one difference to painting is that an empty extent is ignored; painting let it cover everything after its offset. The volume span is computed once instead of on every use. Resolving now takes 0.8 s instead of 23 s for the archive above, and 4 s instead of 21 s for a full backup of 1M extents followed by 30 incrementals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Volumes refer back to their archive and the archive keeps its volumes, so a TIBX is only freed by the cyclic garbage collector, whenever that runs. Until then, the index of an archive that was already closed stayed in memory, and a run over many large archives held several of them at once. close() now also drops the index and the segment cache, which breaks the cycle. For a synthetic archive with 1.1M extents, 500 MiB stayed in use after its with block; now 33 MiB do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Opening one real archive took 83,840 reads, 71.5 s over SMB against 0.7 s locally: its newest commit root is 84k pages before EOF, and the backward scan for it read one page at a time. The scan now reads backward in chunks, from 64 KiB up to 4 MiB. Like commit_roots(), it indexes the page envelopes directly instead of going through cstruct, which makes such a long scan 2.8 times slower; the ENVELOPE PARSING note has the numbers. On a synthetic archive laid out the same way, the root is found in 88 reads instead of 84,001. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
A random read took two round trips over SMB: one for the header page of a segment, then one for its continuation pages, whose number only the header gave. The segment_map already says how many pages a segment takes, so they are now read together; the header still decides where the segment ends. A single-file archive is also opened unbuffered. Every read seeks to a page boundary, so a buffer only doubled each single-page read to 8 KiB, and split page runs that are no multiple of it into two reads. Reads that a raw file returns in parts are completed. On a synthetic archive, random 4 KiB reads now make half as many read calls, 0.83 instead of 1.65 each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Any stream of 1 MiB or more counted as a volume, but on real disks two kinds of Acronis metadata reach that size. An allocation bitmap holds a bit per cluster, so it passes for any volume over 32 GiB, and dissect.target reported "Can't identify filesystem" for each one. The bootstrap region before the first partition of a disk is a whole 1 MiB when partitioned the Windows way, and dissect.target's BitLocker and FAT detection failed on it with warnings. A stream without a recognised filesystem now also has to span a whole number of sectors, which a bitmap does not, and must not start with a partition table: an MBR whose partitions all start past the stream, or the protective MBR of a GPT disk. The MBR is parsed with new c_boot definitions. A locked archive is still classified on spans alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
A volume read put its result together piece by piece: a slice of each segment it needed, a fresh run of zeros for each hole, all appended to a growing bytearray and copied once more at the end. It now collects views into the cached segments and the runs of zeros, and joins them once, so segment data is copied once, straight into the result. Read sequentially, 1 MiB at a time, the sample volumes and two synthetic archives take up to two thirds less time, and none takes longer than before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
A tree walk read its pages one at a time. Since single-file archives are opened unbuffered, that is one OS read per 4 KiB page: 20,000 for the index of the largest sample archive, twice what the buffered PR head needed, and each one a round trip when the archive is on a share. A tree is written in one go, so the pages the walk visits next mostly follow on from each other in the archive. The walk now reads such a run of pages, up to 256 at a time, in one I/O. It reads no page it would not visit anyway, and none past the end of the archive. On synthetic archives at the sample's index size, the index walk goes from 12,341 reads to 69. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
resolve_extents returned a (start, stop, extent) tuple per interval. The extents of a full backup never overlap, so their intervals are just the sorted extents, yet each got a tuple of its own: a million more objects for the cyclic GC to traverse on the largest sample archive, at 64 bytes each. It now returns the starts, stops and covering extents as parallel lists. For a full backup those are the lists it builds anyway, so no object is created per interval. On a synthetic archive with 1.1M extents, this halves the objects the GC tracks while the archive is open, from 2.2M to 1.1M, and shortens a full collection from ~430 to ~290 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
The segment index merged the runs of the segment_map, the mem-tree and the ctrees, newest first, record by record. Every segment id of the newer runs went into a set, and every older record was looked up in it. As the newer runs hold the higher ids, the merged index also came out unsorted and was sorted again. Segment ids are handed out in increasing order, so each run covers an id range of its own and no record can shadow another: the runs only need putting in id order. That is what now happens when their ranges, tombstones included, do not overlap. Runs that do share ids are merged with keep-first semantics as before. Every sample archive has two to four segment_map runs. On a synthetic archive with 1.1M segments in four runs, loading the index takes 2.1 s instead of 2.8 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Loading the index was CPU-bound on real archives, cold as well as warm: the largest one spent 6.9 of its 7.4 s in volumes() on the CPU. Most of that went to the extents as objects: 2 million Extent tuples of some 250 bytes each, which the cyclic GC went through in every full collection, namedtuples never being untracked like plain tuples. Keep the extents in an ExtentTable instead, an array per Extent field and a row per extent, in (volume_id, source_offset) order. Each group of live records is unpacked in one call and transposed into the columns, and a volume is the range of rows of its volume id. resolve_extents works on those rows and returns its intervals as arrays, the covering extent as a row number, which the read path takes the segment from. An Extent is only built for whoever asks for one, through TIBX.extents or TibxVolume.extents, which are now ExtentTables. Real archives have one to four data_map runs, and none of the 81 archives of the test set has a tombstone in them, nor a key in two runs (in the 50 that were opened in full). So the runs are only put in offset order, by a sort of the rows, and the keep-first merge, now over the columns, only runs when there are tombstones or a key in more than one run. On synthetic archives of 1.1 million extents, loading the index takes 2.7 instead of 5.3 s for one run and 3.2 instead of 5.3 s for four runs, with a peak of 160 to 190 MiB instead of 450 to 480 MiB. Every stream of the 15 test archives reads back the same as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
The segment_map runs of real archives overlap in segment ids: a newer run's ids fall within the range of the older runs', and in some archives it rewrites the locations of older segments, up to a fifth more records than segments. So the merge of "Merge TIBX segment_map runs without a set" never took its fast path on the 81 archives of the test set, and every load paid for a set of the newer runs' ids and a sort of all of them. There is no need to merge the runs at all. Keep them apart, newest first, each as sorted arrays, and look a segment id up in one run after the other: the first run with a record for it decides, which is LSM keep-first, and a tombstone there means the segment is gone. That is two to five binary searches for each segment a read decompresses. Only iterating over the index still merges, and only when it does. Each group of live records is unpacked in one call and transposed into the arrays. The page_count, the record's one little-endian field, is unpacked as a big-endian int with the rest and its bytes are swapped after, a whole run at once. On a synthetic archive shaped like the largest real one (four runs, 21% of the records rewrites), loading the index takes 3.2 instead of 4.0 s, with a peak of 190 instead of 270 MiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
A volume's allocation bitmap is a bit per cluster, and the rule of "Fix TIBX metadata streams listed as volumes" tells it from a partition by its span, which is no whole number of sectors. The bitmap of a 100 GiB volume with 4 KiB clusters is 3,276,800 bytes, though, and one such bitmap was still listed as a volume in the 81 archives of the test set. Such a bitmap starts with a sector of set bits: the clusters at the start of a volume are always in use. A partition, formatted or not, does not start like that, so a stream that does is no volume. It is still among streams(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Every block of a linked-LZ4 chain starts with an 8-byte header, and cstruct took ~3 us to parse one. An index page chains six to eight blocks, so with dissect.util's native LZ4 the headers were a quarter of the time spent decoding the index's LZ4. The chain loop now unpacks them with a precompiled struct, like the data_map and segment_map records; a test keeps it in step with c_tibx.lz4_block_header. On synthetic archives shaped like the real ones, loading the index takes 7-14% less CPU, and 25-28% less together with dissect.util borrowing its LZ4 input instead of copying it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
"Do not list sector-aligned TIBX bitmaps as volumes" tells such a bitmap by a first sector of set bits. Over 187 real archives, two bitmaps that are a whole number of sectors start otherwise, and each was still listed as a volume, with a "Can't identify filesystem" in dissect.target: one has clear bits in its first sector, the other within its first 16 bytes. Their span gives them away instead: a bit per cluster of a volume in the same archive, rounded up to a byte. 171 metadata streams of those archives have that span for 4 KiB clusters, and none of their 572 volumes has it for another one. So a stream with the span of another stream's bitmap, for a cluster size from 512 bytes to 2 MiB, is no volume, unless it starts with a filesystem. Like the other size tests this needs no read, so it holds for a locked archive too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Below Python 3.14, zstd comes from backports.zstd, which only the full extra of dissect.archive installs. The TIBX codec imported it at the top, so without that extra "import dissect.archive.tibx" failed: no archive could be opened, although the index is LZ4 and listing what an archive holds, or reading stored segments, needs no zstd. It is now imported where a segment is decompressed, as the crypto module imports pycryptodome. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
|
Updated with 18 commits after a test campaign on 187 real-world Acronis Cyber Protect archives (5.6 GiB to 1.27 TiB, up to 12.7M extents), on Windows, locally and over SMB: every archive opened, every volume read back and hashed, and Result: all 187 archives open and every volume reads back, with the same bytes as the previous head wherever it could read them. Fixed along the way: index trees over 8,192 pages were silently truncated (7 volumes in 8 archives unreadable), and metadata streams were listed as volumes. Loading the index is ~9x faster, with up to 90% less memory. Related PRs:
The commits are meant to be squashed; each message has the details. 🤖 Generated with Claude Code |
fox-it/dissect.util#141 adds a dictionary argument to dissect.util's LZ4 decompress, so every TIBX LZ4 block now goes through it, including the linked-LZ4 blocks that refer back into earlier output. This drops the in-tree decoder, the optional C lz4 path and the lz4 dependency. The tests now build their LZ4 blocks by hand. A block without a recorded size is rejected, which keeps the decode bounded on both dissect.util backends. dissect.util 3.25.dev10 is the first build with it, so that is the minimum version until 3.25 is released. Every stream of the test archives reads back identically to before, with both dissect.util backends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
test_close_releases_the_index checks that close() lets reference counting free the index while the cyclic GC is off. PyPy has no reference counting, so the test is now skipped there. The TIBX exceptions are documented in dissect.archive.tibx and in dissect.archive.tibx.exception, which made each bare Raises: entry an ambiguous cross-reference: 12 warnings, and docs-build fails on warnings. The entries now name the exception module and still render as the short name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
|
1a4c4a2 fixes the two CI jobs this PR would turn red: the refcount test is skipped on PyPy, and the 🤖 Generated with Claude Code |
|
|
||
| c_boot = cstruct(endian="<").load(boot_def) | ||
|
|
||
| PAGE_SIZE: int = c_tibx.PAGE_SIZE |
There was a problem hiding this comment.
Type hinters unfortunately require all of these constants to also be defined in the .pyi file. For example: https://github.com/fox-it/dissect.fve/blob/0b3caf20c1e95136cb2eb4494303604e7629616b/dissect/fve/bde/c_bde.pyi#L816-L837
There was a problem hiding this comment.
@Schamper Could we add this to cstruct-stubgen? or at least create an issue for it?
| # Generated by `cstruct-stubgen`; reformatting or rewrapping it here would only be undone | ||
| # the next time the stub is regenerated. | ||
| extend-exclude = ["dissect/archive/tibx/c_tibx.pyi"] |
There was a problem hiding this comment.
That's fine, our current setup is imperfect and requires manual addition anyway (see my other comment). For the time being, it's preferred that you just let this file be autoformatted by ruff.
There was a problem hiding this comment.
@Schamper maybe output some warning in cstruct-stubgen, that you should autoformat it using ruff manually.
Type checkers only see what c_tibx.pyi declares, so it now also lists the 51 module-level constants of c_tibx.py, as dissect.fve's c_bde.pyi does. The stub is no longer excluded from ruff and is formatted by it; like dissect.fve, stubs ignore PYI042 for the snake_case aliases that cstruct-stubgen writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
The tests for dissect/archive/tibx/ are now in tests/tibx/, and the synthetic archive builder is in tests/_tools/tibx/, where dissect.evidence keeps its test tools. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
Each module ported from acronis-tib-reader or acronis-tibx now names the project with its MIT copyright notice in its own docstring, so THIRD_PARTY_NOTICES.md and its license-files entry are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
falklindner
left a comment
There was a problem hiding this comment.
All five points are addressed in 8a0dce9 (stub), 7ca3ba0 (test layout) and 16095d6 (notices).
🤖 Generated with Claude Code
|
|
||
| c_boot = cstruct(endian="<").load(boot_def) | ||
|
|
||
| PAGE_SIZE: int = c_tibx.PAGE_SIZE |
| # Generated by `cstruct-stubgen`; reformatting or rewrapping it here would only be undone | ||
| # the next time the stub is regenerated. | ||
| extend-exclude = ["dissect/archive/tibx/c_tibx.pyi"] |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #30 +/- ##
===========================================
+ Coverage 69.25% 81.64% +12.39%
===========================================
Files 9 20 +11
Lines 1187 2572 +1385
===========================================
+ Hits 822 2100 +1278
- Misses 365 472 +107
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
yunzheng
left a comment
There was a problem hiding this comment.
I have some feedback regarding the Compression. See my inline comments.
The current tests are mostly synthetic, so it would be nice to have fixture-based tests for:
- a small real backup at “normal” (stored + Zstd), decoding every segment and checking the lengths
- a small backup at “low”, which should produce LZ4 segments (none of our current samples have them)
zlib can stay synthetic, since the current writer never produces it. Maybe some of the real tibx archives already meet this, but I haven't checked.
| enum ComprLvl : uint8 { | ||
| NONE = 0, // "none" | ||
| LOW = 1, // "low" | ||
| NORMAL = 2, // "normal" | ||
| HIGH = 3 // "high" | ||
| }; | ||
There was a problem hiding this comment.
Fix indenting of ComprLvl, and added ComprAlg enums. (Zlib was added).
| enum ComprLvl : uint8 { | |
| NONE = 0, // "none" | |
| LOW = 1, // "low" | |
| NORMAL = 2, // "normal" | |
| HIGH = 3 // "high" | |
| }; | |
| enum ComprLvl : uint8 { | |
| NONE = 0, // "none" | |
| LOW = 1, // "low" | |
| NORMAL = 2, // "normal" | |
| HIGH = 3 // "high" | |
| }; | |
| enum ComprAlg : uint8 { | |
| NONE = 0, | |
| LZ4 = 1, | |
| ZLIB = 2, | |
| ZSTD = 3 | |
| }; |
| uint32 length; /* uncompressed payload size */ | ||
| uint32 zlength; /* compressed payload size */ | ||
| uint32 key_id; /* encryption key id, 0 = plaintext */ | ||
| uint16 compression; /* see COMP_* */ |
There was a problem hiding this comment.
I think the compression should be just uint8, and the other uint8 is some flag.
| uint16 compression; /* see COMP_* */ | |
| ComprAlg compression; /* compression codec */ | |
| uint8 _kind; /* unknown, observed 0x00, 0x01, 0x02, 0x03 */ |
See my remarks on COMP_ in other comment
| # segment_header.compression variants | ||
| COMP_NONE = 0x0000 | ||
| COMP_LZ4 = 0x0001 | ||
| # Observed in True Image 2026 metadata streams (2 in partition backups, 3 in device | ||
| # metadata of USB sources), only ever stored (zlength == length); compressed forms of | ||
| # these variants have not been seen in the wild | ||
| COMP_STORED_VARIANTS = frozenset({0x0002, 0x0003}) | ||
| COMP_ZSTD = frozenset({0x0300, 0x0301, 0x0302, 0x0303}) |
There was a problem hiding this comment.
I think these definitions should go away, and solely use the ComprAlg enum.
I'm curious about the COMP_LZ4 = 0x0001, because I think that is incorrect, it should've been COMP_LZ4 = 0x0200. I wonder if some of these Compression codecs can be tested on the real Tibx test data.
It's possible that Zlib is legacy, as I don't see any writer for it in my version only a reader.
| def decompress(raw: bytes, segment: Segment) -> bytes: | ||
| """Decompress a segment's raw payload to exactly ``segment.length`` plaintext bytes.""" |
There was a problem hiding this comment.
decompress() needs a refactor, my clanker came up with something like this, but I have not tested it:
import zlib
def decompress(raw: bytes, segment: Segment) -> bytes:
"""Decompress a segment's raw payload to exactly ``segment.length`` plaintext bytes.
Dispatches only on the codec byte; ``segment.kind`` plays no part in decoding.
"""
alg = segment.compression
if alg == ComprAlg.NONE:
if len(raw) != segment.length:
raise CorruptArchiveError(
f"stored segment at page {segment.page_index} has zlength {len(raw)} "
f"!= length {segment.length}"
)
return raw
try:
match alg:
case ComprAlg.LZ4:
out = lz4_block_decompress(raw, segment.length)
case ComprAlg.ZLIB:
out = zlib.decompress(raw)
case ComprAlg.ZSTD:
out = decompress_zstd(raw, segment.length)
case _:
raise UnsupportedFormatError(
f"segment at page {segment.page_index} uses unknown compression "
f"{int(alg):#04x} (kind {segment.kind})"
)
except (CorruptArchiveError, UnsupportedFormatError):
raise
except Exception as exc: # codec-library errors (LZ4BlockError, zlib.error, ZstdError, ...)
raise CorruptArchiveError(
f"{ComprAlg(alg).name} segment at page {segment.page_index} failed to decompress: {exc}"
) from exc
if len(out) != segment.length:
raise CorruptArchiveError(
f"{alg.name} segment at page {segment.page_index} decoded {len(out)} bytes, "
f"expected {segment.length}"
)
return out
Add a parser for the Acronis CyberBackup/CyberProtect TIBX ("archive3") backup format as a
dissect.archive.tibxsubpackage.Closes #20
Proposed changes
The parser handles the full TIBX format:
TIBX is materially more complex than VBK (page store + LSM index + crypto + chain reconstruction), so a subpackage is proposed rather than a flat module. Happy to consolidate if preferred.
Dependencies
pycryptodomeandbackports.zstdadded to thefulloptional extra; LZ4 uses dissect.util 3.25.dev10 or later (fox-it/dissect.util#141).Testing
Unit tests covering all parser layers (page store, LSM trees, segments, codecs, crypto, volume reconstruction, version sets, tombstone masking), on synthetic CRC-correct archives built by
tests/_tools/tibx/_synth.py;tests/tibx/test_tibx_acronis.pychecks the real Acronis samples intests/_data/acronis/tibx(Git LFS).Validated byte-for-byte against real Acronis True Image 2026 (Build 42980) output across: single-partition, multi-partition, whole-disk (MBR and GPT), multi-disk, split, incremental, differential, and encrypted archives; NTFS, FAT32, and exFAT volumes.
Also validated on Acronis Cyber Protect 15 (build 37414) archives, including AES-GCM encryption, and on 187 real-world Cyber Protect archives: see this comment.
Acknowledgments
Thanks to @mniedermaier for his contributions and support in developing this parser.
Checklist