Skip to content

Add TIBX archive parser - #30

Open
falklindner wants to merge 40 commits into
fox-it:mainfrom
falklindner:feature/tibx-parser
Open

falklindner wants to merge 40 commits into
fox-it:mainfrom
falklindner:feature/tibx-parser

Conversation

@falklindner

@falklindner falklindner commented Jul 13, 2026 •

Copy link
Copy Markdown

Add a parser for the Acronis CyberBackup/CyberProtect TIBX ("archive3") backup format as a dissect.archive.tibx subpackage.

Closes #20

Proposed changes

The parser handles the full TIBX format:

  • Page store: 4 KiB pages with CRC-32C verification, ARCH commit-root superblocks, live-root / commit-root selection
  • LSM B-tree index: TLV directory, L-SB superblocks, LEAF/LDIR cell decoding, data_map / segment_map with mem-tree and on-disk ctree layers, tombstone masking for backup-version cleanup
  • Compression: zstd and linked-LZ4 (dictionary-seeded block chain)
  • Encryption: AES-CBC and AES-GCM (128, 192 and 256 bit) with PBKDF2 key derivation
  • Chain reconstruction: full incremental/differential chain with recovery-point selection
  • Volume reconstruction: NTFS, FAT32, exFAT, FAT12/16 and ext boot-sector detection for accurate volume sizing
  • Archive variants: split archives (auto-stitched), version sets (TLV[18] file table with compacted bases), multi-disk archives

TIBX is materially more complex than VBK (page store + LSM index + crypto + chain reconstruction), so a subpackage is proposed rather than a flat module. Happy to consolidate if preferred.

Dependencies

pycryptodome and backports.zstd added to the full optional extra; LZ4 uses dissect.util 3.25.dev10 or later (fox-it/dissect.util#141).

Testing

Unit tests covering all parser layers (page store, LSM trees, segments, codecs, crypto, volume reconstruction, version sets, tombstone masking), on synthetic CRC-correct archives built by tests/_tools/tibx/_synth.py; tests/tibx/test_tibx_acronis.py checks the real Acronis samples in tests/_data/acronis/tibx (Git LFS).

Validated byte-for-byte against real Acronis True Image 2026 (Build 42980) output across: single-partition, multi-partition, whole-disk (MBR and GPT), multi-disk, split, incremental, differential, and encrypted archives; NTFS, FAT32, and exFAT volumes.

Also validated on Acronis Cyber Protect 15 (build 37414) archives, including AES-GCM encryption, and on 187 real-world Cyber Protect archives: see this comment.

Acknowledgments

Thanks to @mniedermaier for his contributions and support in developing this parser.

Checklist

falklindner and others added 3 commits July 13, 2026 09:06
Add a parser for the Acronis CyberBackup/CyberProtect TIBX
("archive3") backup format as a dissect.archive.tibx subpackage.

The parser handles the 4 KiB page store, CRC-32C verification,
ARCH commit-root superblocks, TLV directory, LSM B-tree index
(data_map / segment_map with mem-tree and on-disk ctree layers),
zstd and linked-LZ4 decompression, AES-256-CBC decryption, and
full incremental/differential chain reconstruction with
recovery-point selection.

Volume reconstruction supports NTFS, FAT32, exFAT, FAT12/16 and
ext boot-sector detection for accurate volume sizing, split
archives, version sets (TLV[18] file table with compacted bases),
multi-disk archives, and tombstone masking for backup-version
cleanup.

Validated byte-for-byte against real Acronis True Image 2026
(Build 42980) output across single-partition, multi-partition,
whole-disk (MBR and GPT), multi-disk, split, incremental,
differential, and encrypted archives.

Closes fox-it#20

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a parser for the Acronis CyberBackup/CyberProtect TIBX
("archive3") backup format as a dissect.archive.tibx subpackage.

The parser handles the 4 KiB page store, CRC-32C verification,
ARCH commit-root superblocks, TLV directory, LSM B-tree index
(data_map / segment_map with mem-tree and on-disk ctree layers),
zstd and linked-LZ4 decompression, AES-256-CBC decryption, and
full incremental/differential chain reconstruction with
recovery-point selection.

Volume reconstruction supports NTFS, FAT32, exFAT, FAT12/16 and
ext boot-sector detection for accurate volume sizing, split
archives, version sets (TLV[18] file table with compacted bases),
multi-disk archives, and tombstone masking for backup-version
cleanup.

Validated byte-for-byte against real Acronis True Image 2026
(Build 42980) output across single-partition, multi-partition,
whole-disk (MBR and GPT), multi-disk, split, incremental,
differential, and encrypted archives.

Closes fox-it#20

Co-Authored-By: Matthias Niedermaier <matthias.niedermaier@hs-augsburg.de>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@yunzheng

Copy link
Copy Markdown
Member

Very nice work, and thanks for sharing it! I haven't started reviewing yet, but I pushed a commit filling in a few of the unknown fields based on some of our own reversing efforts.

falklindner and others added 2 commits July 22, 2026 21:48
Reduce the random-access parsing gap versus a raw disk image with
caching and batched I/O on the segment read path. Output is
byte-identical: every real-sample volume hash and file check is
unchanged.

- Segment cache: replace the fixed 32-entry limit with a 256 MiB
  byte-budget LRU, so random-access workloads (MFT, registry
  hives) that touch thousands of segments stop thrashing.
- Multi-page segments: read the contiguous continuation pages in
  a single I/O instead of one read per page, and retain the
  segment header page so it is not read twice.
- Add a small LRU page cache to the page store.
- Use the C lz4 library for the linked-LZ4 decoder when it is
  importable (byte-identical, including the growing-dictionary
  case), falling back to the pure-Python decoder otherwise; add
  lz4 to the full extra.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Treating "a keymap tree with records exists" as proof of encryption
over-reports: an archive whose keymap holds no password-wrapped key
would prompt for a password that could never unwrap anything.

Confirm the keymap actually carries a password-wrapped data key by
locating one structurally -- a known algorithm id, a plausible PBKDF2
iteration exponent, a full salt and an AES-block-multiple wrapped key.
No password is involved and no key is derived, so this stays cheap
enough to run on every open. AES-GCM archives are still reported as
encrypted, since they are: they now fail with a clear "not supported"
instead of being mistaken for plaintext. Detection never raises -- a
missed one still surfaces as an explicit InvalidPasswordError from the
segment reader rather than a silent misread.

The keymap region is extracted once and shared with unwrap_data_key,
and pycryptodome is imported where it is used rather than at module
import time, so detection no longer pulls in an optional dependency.

Validated against the real corpus: the encrypted sample is detected
and still decrypts file-for-file, with no false positive on any of the
ten unencrypted archives.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@yunzheng

Copy link
Copy Markdown
Member

Regarding the last commit, can the TIBX encryption not be detected via the encr_alg in the superblock header?

@falklindner

Copy link
Copy Markdown
Author

Regarding the last commit, can the TIBX encryption not be detected via the encr_alg in the superblock header?

Good point — checked against the full real-sample corpus (12 archives): encr_alg in the ARCH superblock agrees with the keymap detection everywhere (AES_256_CBC on the encrypted sample, NONE on the rest), so yes, it does indicate encryption.

The keymap scan answers a narrower question though: not "is this encrypted" but "does a password-wrapped data key exist" — which is what drives the password prompt / keychain lookup. Acronis also supports certificate-wrapped keys (format byte 0x02 next to the 0x01 we locate), where encr_alg would be set but asking for a password can't help. Detection via the keymap also can't disagree with unlock(), since both parse the same structure, and it doesn't depend on the superblock byte layout, which we've only verified for header_version 8.

The two compose nicely, and encr_alg is essentially free: happy to use encr_alg != NONE as the archive-level signal and the keymap scan to decide password vs. certificate — that would let a certificate-wrapped archive fail with "encrypted, but not password-protected" instead of a generic missing-password error.

@falklindner

Copy link
Copy Markdown
Author

Is there anything I can do to support the PR?

@yunzheng yunzheng left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @falklindner, sorry for the late reply, but I was on holiday.

I left some general feedback.

Also, regarding the synthetic tests, while I do think that's awesome, I think nothing beats real data. It would be nice to add tibx archives created by Acronis itself, for example:

  • simple tibx archive
  • password protected archive
  • full disk vs files
  • other edge cases that we want to test

I'm also curious how much you're already using this implementation for your own needs.

Comment thread dissect/archive/tibx/crc32c.py Outdated
Comment thread dissect/archive/tibx/c_tibx.py Outdated
Comment thread dissect/archive/tibx/crc32c.py Outdated
Comment thread dissect/archive/tibx/crypto.py Outdated
Comment thread dissect/archive/tibx/lsm.py Outdated
Comment thread dissect/archive/tibx/page.py Outdated
Comment thread tests/_synth.py Outdated
Comment thread dissect/archive/tibx/crypto.py Outdated
Comment thread dissect/archive/tibx/c_tibx.py Outdated
claude and others added 3 commits August 27, 2026 18:30
Reuse crc32c from dissect.util
    dissect/archive/vbk.py already imports dissect.util.hash.crc32c, and the
    module is present in the pinned dissect.util floor (3.22), so the vendored
    Castagnoli table had no reason to exist. Delete tibx/crc32c.py and move
    page_crc32c next to its callers in page.py, folding its test into
    test_page.py. The new implementation chains the three runs through the
    crc32c value argument instead of building a zeroed 4 KiB copy per page.

Parse the wrapped data key as a structure
    Add a wrapped_key struct to c_tibx.py and use it in crypto.py in place of
    the hardcoded offset arithmetic, in the module docstring as well as the
    code. The scan for the key keeps its cheap format-byte prefilter, so the
    structure is only built for actual candidates. tests/_synth.py builds its
    fixture blob from the same struct.

Parse page envelopes as structures on the cold paths
    SuperBlock, PageStore.__init__ and page_type now go through
    c_tibx.page_header / arch_superblock. The three whole-archive scan loops
    (live_root, commit_roots, verify) keep direct byte indexing: cstruct costs
    ~3.99 us per envelope against ~0.24 us, which on a 100 GiB archive is ~100 s
    of header parsing instead of ~6 s. A note in page.py records the reason.
    SuperBlock now reports a truncated buffer as InvalidArchiveError rather than
    letting cstruct's EOFError escape.

Compose encr_alg with the keymap scan
    TIBX.encrypted is now the archive-level question, answered by the
    superblock's encr_alg with the keymap scan as a fallback for header
    versions where encr_alg is not verified. The narrower question moves to
    TIBX.password_protected, and unlock() rejects a certificate-wrapped archive
    with "encrypted, but not password-protected" instead of a generic wrong
    password error.

Fix references and lint
    Describe the provenance of the format definitions and the synthetic
    fixtures inline, in place of the references to files that are not part of
    this repository. Exclude the generated c_tibx.pyi from ruff, which it has
    been failing (E501, PYI042) since it was added.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V2NzRCqXtY9bjPJCj5qVp2
The LSM engine, the format findings the structure definitions encode and the
synthetic test fixtures all derive from acronis-tibx, a private research
project by mniedermaier that is no longer publicly available. Name that source
in each of the three docstrings instead of pointing at a THIRD_PARTY_NOTICES.md
and a docs/FINDINGS.md that are not part of this repository.

Being a private and now unavailable repository there is no URL to link, so the
credit is given inline. A THIRD_PARTY_NOTICES.md carrying the original license
notice still needs to be added from a local copy of that repository.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V2NzRCqXtY9bjPJCj5qVp2
The LSM engine was credited to acronis-tibx, but acronis-tibx has no LSM
engine of its own -- it vendored the MIT-licensed acronis-tib-reader
(github.com/TreadingTheTiber/acronis-tib-reader), which is still publicly
available. Copyright there is "the tibread contributors", not mniedermaier.

Re-checking every module rather than only the three raised in review turned
up four more derived files carrying no credit at all: maps.py, segment.py,
crypto.py and codecs.py, plus page.py.

Add THIRD_PARTY_NOTICES.md, organised by upstream since both projects are
woven through the package, list it in license-files, and reference it from
every derived module.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread dissect/archive/tibx/codec.py Outdated
Comment thread dissect/archive/tibx/exception.py
falklindner and others added 6 commits August 28, 2026 15:04
Benchmarking the page-envelope struct-vs-offset tradeoff end to end, rather
than timing the parse call in isolation, showed the offsetting only pays in
one of the three loops it was applied to. Measured on a 949 MiB archive
(242,977 pages), offset vs struct:

  live_root()      1.02x -- stops at the newest ARCH, 3 pages in practice
  verify()         1.06x -- dominated by CRC-32C
  commit_roots()   1.92x -- 859 ms vs 1644 ms

live_root() and verify() go back to c_tibx.page_header; commit_roots() keeps
the direct indexing, and the note at the top now carries the end-to-end
figures instead of the per-call ratio.

Chasing verify()'s cost also turned up the wrong crc32c import.
dissect.util.hash rebinds its crc32c attribute to the native (Rust)
implementation, but "from dissect.util.hash.crc32c import crc32c" resolves
through sys.modules to the pure-Python submodule and bypasses the rebind --
identical output at 5.7 MB/s instead of 80.9 MB/s. verify() on that archive
drops from 172 s to 15.5 s.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
exceptions.py -> exception.py, codecs.py -> codec.py, maps.py -> map.py, and
tests/test_codecs.py -> tests/test_codec.py alongside. Import-only change; no
behaviour is touched. codec.py also stops shadowing the stdlib codecs module.

tests/test_exceptions.py and dissect/archive/exceptions.py are left alone --
they predate this branch and cover the package-level exceptions, so renaming
them belongs to the wider migration rather than to this PR.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A 512-page LRU sat in front of PageStore.page() on the theory that LSM tree
walks and segment header pages get revisited. Measured against the real-sample
corpus it hits exactly zero times, on every archive tried: 3000 scattered 4 KiB
reads produce only 40 to 500 page() calls in total.

Two layers above absorb everything first. TIBX's 256 MiB segment cache catches
repeat segment reads, and segment payloads go through read_run(), which never
consulted the LRU. What is left for page() is one header page per distinct
segment, read once.

The LRU only earned hits with the segment cache shrunk until it thrashed
(12.6% at 4 MiB, 66% at 256 KiB), which is not a configuration anyone runs.

read_run()'s docstring also claimed it honoured pages already resident in the
LRU; it never did.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lz4_block_decompress now dispatches across three decoders instead of two:

  1. the C lz4 package, the only one taking a dictionary, so it serves every block
  2. dissect.util (Rust-backed when built) for blocks needing no dictionary --
     the first block of every chain, and whole LZ4 segments
  3. the in-tree decoder, now reached only for dictionary blocks without lz4

dissect.util cannot take the dictionary blocks: neither backend exposes a
dictionary parameter, and 931 of 931 non-first blocks in the real-sample corpus
back-reference earlier output, up to 10,690 bytes back. Dropping the dictionary
makes 6 of 17 real archives fail to decode outright.

Where it does apply it is worth having: measured over the 307 no-dictionary
blocks in that corpus it is byte-identical to the in-tree decoder and 14x faster
than it (8.2 ms vs 116 ms), which is what runs when lz4 is not installed.

The two dissect.util backends raise different types on malformed input -- the
Rust one ValueError, the pure-Python one CorruptDataError, which is not a
ValueError -- so both are normalised to CorruptArchiveError.

Verified: with the C package disabled, all 17 real archives decode byte-identical
to the accelerated path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Changes found by testing against real Acronis Cyber Protect 15
archives, checked against acrocmd's own listings:

- recovery_points() reads the slices tree (TLV[5]), one record per
  backup, instead of counting ARCH commit roots. Selection filters
  extents by slice id.
- volumes() leaves out Acronis metadata streams. streams() returns
  the unfiltered list.
- AES-GCM segments are decrypted (IV || tag || ciphertext).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replace the remaining fixed-offset reads with structs: the ARCH
header, TLV entries, linked-LZ4 block headers, LSM cell groups, LDIR
values and segment_map records. Boot-sector sizing uses its own
little-endian c_boot definitions. The synthetic archive builders are
built from the same structs.

No behaviour change: every stream at every recovery point of the real
sample corpus hashes identically before and after.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two behaviour changes in 57fa473, both checked against acrocmd on real Cyber Protect 15 archives. recovery_points() now reads the slices tree, one entry per backup; counting commit roots gave too many. volumes() skips Acronis's own metadata streams, and streams() still lists everything. Older recovery points were verified file by file.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since then, volumes() also leaves out allocation bitmaps and the disk's bootstrap region, close() releases the index, and non-overlapping extents are resolved without a sweep. Test results: #30 (comment)

🤖 Generated with Claude Code

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

segment_map records now go through segment_map_key / segment_map_value (9b05168). page_count stays the one little-endian field.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The data_map and segment_map are now decoded in bulk into arrays instead of per-record cstruct objects; a test keeps the precompiled structs in step with c_tibx. segment_map runs are looked up run by run (keep-first) instead of merged. Test results: #30 (comment)

🤖 Generated with Claude Code

@falklindner falklindner Sep 11, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AES-GCM segments (57fa473), as written by Cyber Protect 15; True Image 2026 used AES-CBC: the payload is IV || tag || ciphertext and is not PKCS#7-unpadded.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A segment's header and continuation pages are now read in one I/O, sized by the segment_map's page count. Test results: #30 (comment)

🤖 Generated with Claude Code

@falklindner

Copy link
Copy Markdown
Author

It would be nice to add tibx archives created by Acronis itself

Real samples are ready: 10 archives from Acronis Cyber Protect 15, covering your list (simple, password-protected, whole disk MBR/GPT vs. file-level) plus chains, stored and multi-volume. They're identity-scrubbed, about 6 MB gzipped in total. That's well above the existing tests/_data (534 KB), so: add them via LFS under tests/_data/, or would you prefer a subset?

@yunzheng

Copy link
Copy Markdown
Member

It would be nice to add tibx archives created by Acronis itself

Real samples are ready: 10 archives from Acronis Cyber Protect 15, covering your list (simple, password-protected, whole disk MBR/GPT vs. file-level) plus chains, stored and multi-volume. They're identity-scrubbed, about 6 MB gzipped in total. That's well above the existing tests/_data (534 KB), so: add them via LFS under tests/_data/, or would you prefer a subset?

Nice! Can you put them in LFS under test/_data/acronis/tibx? The total size is fine.

falklindner and others added 17 commits October 21, 2026 13:44
Opening the volumes of a large archive took up to a minute and close
to 2 GB of memory, nearly all of it spent on the data_map and
segment_map records: about a million of each at a few hundred GB.
Parsed through c_tibx they cost ~10.7 us a record, and cstruct's int
subclasses are tracked by the cyclic GC, so the index also made every
full collection pause for most of a second.

These two record types are now decoded in bulk with precompiled
structs that mirror the c_tibx definitions, which stay the reference
(a test keeps the two in step). The segment index is kept in arrays
instead of a dict of locators. For an archive with 720k extents,
volumes() drops from 40 s to 15 s and from 780 to 470 MiB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Without the optional C lz4 package, every block of a linked-LZ4 chain
but the first went through the in-tree decoder, which copies matches
byte by byte, even where dissect.util is built with Rust. dissect.util
takes a dictionary since fox-it/dissect.util#141, so these blocks now
go through it as well; the in-tree decoder stays for older versions.
The dictionary handed on is the last 64 KiB of output, as far back as
a match can reach, instead of all output so far.

On a synthetic archive with 1.1M extents, loading the index without
the C package takes 6 s instead of 12 s, about the same as with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Resolving the extents of a volume took 24 of the 29 s volumes() needs
on a synthetic archive with 1.1M extents: interval painting does
dictionary and tuple work at every extent boundary.

Extents that do not overlap, as in a full backup, are now taken as
they are. Overlapping ones go through a sweep in offset order that
keeps the extents covering the current offset in a heap, most recent
on top. A randomised test checks both against a brute-force reference.
The one difference to painting is that an empty extent is ignored;
painting let it cover everything after its offset. The volume span is
computed once instead of on every use.

Resolving now takes 0.8 s instead of 23 s for the archive above, and
4 s instead of 21 s for a full backup of 1M extents followed by 30
incrementals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Volumes refer back to their archive and the archive keeps its
volumes, so a TIBX is only freed by the cyclic garbage collector,
whenever that runs. Until then, the index of an archive that was
already closed stayed in memory, and a run over many large archives
held several of them at once.

close() now also drops the index and the segment cache, which breaks
the cycle. For a synthetic archive with 1.1M extents, 500 MiB stayed
in use after its with block; now 33 MiB do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Opening one real archive took 83,840 reads, 71.5 s over SMB against
0.7 s locally: its newest commit root is 84k pages before EOF, and
the backward scan for it read one page at a time.

The scan now reads backward in chunks, from 64 KiB up to 4 MiB. Like
commit_roots(), it indexes the page envelopes directly instead of
going through cstruct, which makes such a long scan 2.8 times slower;
the ENVELOPE PARSING note has the numbers. On a synthetic archive laid
out the same way, the root is found in 88 reads instead of 84,001.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
A random read took two round trips over SMB: one for the header page
of a segment, then one for its continuation pages, whose number only
the header gave. The segment_map already says how many pages a
segment takes, so they are now read together; the header still
decides where the segment ends.

A single-file archive is also opened unbuffered. Every read seeks to
a page boundary, so a buffer only doubled each single-page read to
8 KiB, and split page runs that are no multiple of it into two reads.
Reads that a raw file returns in parts are completed.

On a synthetic archive, random 4 KiB reads now make half as many read
calls, 0.83 instead of 1.65 each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Any stream of 1 MiB or more counted as a volume, but on real disks
two kinds of Acronis metadata reach that size. An allocation bitmap
holds a bit per cluster, so it passes for any volume over 32 GiB, and
dissect.target reported "Can't identify filesystem" for each one. The
bootstrap region before the first partition of a disk is a whole
1 MiB when partitioned the Windows way, and dissect.target's BitLocker
and FAT detection failed on it with warnings.

A stream without a recognised filesystem now also has to span a whole
number of sectors, which a bitmap does not, and must not start with a
partition table: an MBR whose partitions all start past the stream,
or the protective MBR of a GPT disk. The MBR is parsed with new c_boot
definitions. A locked archive is still classified on spans alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
A volume read put its result together piece by piece: a slice of each
segment it needed, a fresh run of zeros for each hole, all appended to
a growing bytearray and copied once more at the end. It now collects
views into the cached segments and the runs of zeros, and joins them
once, so segment data is copied once, straight into the result.

Read sequentially, 1 MiB at a time, the sample volumes and two
synthetic archives take up to two thirds less time, and none takes
longer than before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
A tree walk read its pages one at a time. Since single-file archives
are opened unbuffered, that is one OS read per 4 KiB page: 20,000 for
the index of the largest sample archive, twice what the buffered PR
head needed, and each one a round trip when the archive is on a share.

A tree is written in one go, so the pages the walk visits next mostly
follow on from each other in the archive. The walk now reads such a
run of pages, up to 256 at a time, in one I/O. It reads no page it
would not visit anyway, and none past the end of the archive.

On synthetic archives at the sample's index size, the index walk goes
from 12,341 reads to 69.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
resolve_extents returned a (start, stop, extent) tuple per interval.
The extents of a full backup never overlap, so their intervals are just
the sorted extents, yet each got a tuple of its own: a million more
objects for the cyclic GC to traverse on the largest sample archive,
at 64 bytes each.

It now returns the starts, stops and covering extents as parallel
lists. For a full backup those are the lists it builds anyway, so no
object is created per interval.

On a synthetic archive with 1.1M extents, this halves the objects the
GC tracks while the archive is open, from 2.2M to 1.1M, and shortens a
full collection from ~430 to ~290 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
The segment index merged the runs of the segment_map, the mem-tree and
the ctrees, newest first, record by record. Every segment id of the
newer runs went into a set, and every older record was looked up in
it. As the newer runs hold the higher ids, the merged index also came
out unsorted and was sorted again.

Segment ids are handed out in increasing order, so each run covers an
id range of its own and no record can shadow another: the runs only
need putting in id order. That is what now happens when their ranges,
tombstones included, do not overlap. Runs that do share ids are merged
with keep-first semantics as before.

Every sample archive has two to four segment_map runs. On a synthetic
archive with 1.1M segments in four runs, loading the index takes 2.1 s
instead of 2.8 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Loading the index was CPU-bound on real archives, cold as well as warm:
the largest one spent 6.9 of its 7.4 s in volumes() on the CPU. Most of
that went to the extents as objects: 2 million Extent tuples of some
250 bytes each, which the cyclic GC went through in every full
collection, namedtuples never being untracked like plain tuples.

Keep the extents in an ExtentTable instead, an array per Extent field
and a row per extent, in (volume_id, source_offset) order. Each group of
live records is unpacked in one call and transposed into the columns,
and a volume is the range of rows of its volume id. resolve_extents
works on those rows and returns its intervals as arrays, the covering
extent as a row number, which the read path takes the segment from. An
Extent is only built for whoever asks for one, through TIBX.extents or
TibxVolume.extents, which are now ExtentTables.

Real archives have one to four data_map runs, and none of the 81
archives of the test set has a tombstone in them, nor a key in two runs
(in the 50 that were opened in full). So the runs are only put in
offset order, by a sort of the rows, and the keep-first merge, now over
the columns, only runs when there are tombstones or a key in more than
one run.

On synthetic archives of 1.1 million extents, loading the index takes
2.7 instead of 5.3 s for one run and 3.2 instead of 5.3 s for four runs,
with a peak of 160 to 190 MiB instead of 450 to 480 MiB. Every stream of
the 15 test archives reads back the same as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
The segment_map runs of real archives overlap in segment ids: a newer
run's ids fall within the range of the older runs', and in some
archives it rewrites the locations of older segments, up to a fifth
more records than segments. So the merge of "Merge TIBX segment_map
runs without a set" never took its fast path on the 81 archives of the
test set, and every load paid for a set of the newer runs' ids and a
sort of all of them.

There is no need to merge the runs at all. Keep them apart, newest
first, each as sorted arrays, and look a segment id up in one run after
the other: the first run with a record for it decides, which is LSM
keep-first, and a tombstone there means the segment is gone. That is
two to five binary searches for each segment a read decompresses. Only
iterating over the index still merges, and only when it does.

Each group of live records is unpacked in one call and transposed into
the arrays. The page_count, the record's one little-endian field, is
unpacked as a big-endian int with the rest and its bytes are swapped
after, a whole run at once.

On a synthetic archive shaped like the largest real one (four runs, 21%
of the records rewrites), loading the index takes 3.2 instead of 4.0 s,
with a peak of 190 instead of 270 MiB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
A volume's allocation bitmap is a bit per cluster, and the rule of
"Fix TIBX metadata streams listed as volumes" tells it from a partition
by its span, which is no whole number of sectors. The bitmap of a
100 GiB volume with 4 KiB clusters is 3,276,800 bytes, though, and one
such bitmap was still listed as a volume in the 81 archives of the test
set.

Such a bitmap starts with a sector of set bits: the clusters at the
start of a volume are always in use. A partition, formatted or not,
does not start like that, so a stream that does is no volume. It is
still among streams().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Every block of a linked-LZ4 chain starts with an 8-byte header, and
cstruct took ~3 us to parse one. An index page chains six to eight
blocks, so with dissect.util's native LZ4 the headers were a quarter
of the time spent decoding the index's LZ4.

The chain loop now unpacks them with a precompiled struct, like the
data_map and segment_map records; a test keeps it in step with
c_tibx.lz4_block_header. On synthetic archives shaped like the real
ones, loading the index takes 7-14% less CPU, and 25-28% less together
with dissect.util borrowing its LZ4 input instead of copying it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
"Do not list sector-aligned TIBX bitmaps as volumes" tells such a bitmap
by a first sector of set bits. Over 187 real archives, two bitmaps that
are a whole number of sectors start otherwise, and each was still listed
as a volume, with a "Can't identify filesystem" in dissect.target: one
has clear bits in its first sector, the other within its first 16
bytes.

Their span gives them away instead: a bit per cluster of a volume in the
same archive, rounded up to a byte. 171 metadata streams of those
archives have that span for 4 KiB clusters, and none of their 572
volumes has it for another one. So a stream with the span of another
stream's bitmap, for a cluster size from 512 bytes to 2 MiB, is no
volume, unless it starts with a filesystem. Like the other size tests
this needs no read, so it holds for a locked archive too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
Below Python 3.14, zstd comes from backports.zstd, which only the full
extra of dissect.archive installs. The TIBX codec imported it at the
top, so without that extra "import dissect.archive.tibx" failed: no
archive could be opened, although the index is LZ4 and listing what an
archive holds, or reading stored segments, needs no zstd.

It is now imported where a segment is decompressed, as the crypto module
imports pycryptodome.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
@falklindner

Copy link
Copy Markdown
Author

Updated with 18 commits after a test campaign on 187 real-world Acronis Cyber Protect archives (5.6 GiB to 1.27 TiB, up to 12.7M extents), on Windows, locally and over SMB: every archive opened, every volume read back and hashed, and target-query -f hostname run through the loader of fox-it/dissect.target#1882.

Result: all 187 archives open and every volume reads back, with the same bytes as the previous head wherever it could read them. Fixed along the way: index trees over 8,192 pages were silently truncated (7 volumes in 8 archives unreadable), and metadata streams were listed as volumes. Loading the index is ~9x faster, with up to 90% less memory.

Related PRs:

The commits are meant to be squashed; each message has the details.

🤖 Generated with Claude Code

falklindner and others added 2 commits October 26, 2026 13:23
fox-it/dissect.util#141 adds a dictionary argument to dissect.util's
LZ4 decompress, so every TIBX LZ4 block now goes through it, including
the linked-LZ4 blocks that refer back into earlier output. This drops
the in-tree decoder, the optional C lz4 path and the lz4 dependency.
The tests now build their LZ4 blocks by hand.

A block without a recorded size is rejected, which keeps the decode
bounded on both dissect.util backends.

dissect.util 3.25.dev10 is the first build with it, so that is the
minimum version until 3.25 is released. Every stream of the test
archives reads back identically to before, with both dissect.util
backends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
test_close_releases_the_index checks that close() lets reference
counting free the index while the cyclic GC is off. PyPy has no
reference counting, so the test is now skipped there.

The TIBX exceptions are documented in dissect.archive.tibx and in
dissect.archive.tibx.exception, which made each bare Raises: entry an
ambiguous cross-reference: 12 warnings, and docs-build fails on
warnings. The entries now name the exception module and still render
as the short name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
@falklindner

Copy link
Copy Markdown
Author

1a4c4a2 fixes the two CI jobs this PR would turn red: the refcount test is skipped on PyPy, and the Raises: entries no longer give docs-build ambiguous cross-references.

🤖 Generated with Claude Code


c_boot = cstruct(endian="<").load(boot_def)

PAGE_SIZE: int = c_tibx.PAGE_SIZE

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Type hinters unfortunately require all of these constants to also be defined in the .pyi file. For example: https://github.com/fox-it/dissect.fve/blob/0b3caf20c1e95136cb2eb4494303604e7629616b/dissect/fve/bde/c_bde.pyi#L816-L837

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8a0dce9.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Schamper Could we add this to cstruct-stubgen? or at least create an issue for it?

Comment thread pyproject.toml Outdated
Comment on lines +78 to +80
# Generated by `cstruct-stubgen`; reformatting or rewrapping it here would only be undone
# the next time the stub is regenerated.
extend-exclude = ["dissect/archive/tibx/c_tibx.pyi"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fine, our current setup is imperfect and requires manual addition anyway (see my other comment). For the time being, it's preferred that you just let this file be autoformatted by ruff.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8a0dce9.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Schamper maybe output some warning in cstruct-stubgen, that you should autoformat it using ruff manually.

Comment thread tests/tibx/test_codec.py
Comment thread tests/_tools/tibx/_synth.py
Comment thread THIRD_PARTY_NOTICES.md Outdated
falklindner and others added 3 commits October 27, 2026 13:22
Type checkers only see what c_tibx.pyi declares, so it now also lists
the 51 module-level constants of c_tibx.py, as dissect.fve's c_bde.pyi
does. The stub is no longer excluded from ruff and is formatted by it;
like dissect.fve, stubs ignore PYI042 for the snake_case aliases that
cstruct-stubgen writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
The tests for dissect/archive/tibx/ are now in tests/tibx/, and the
synthetic archive builder is in tests/_tools/tibx/, where
dissect.evidence keeps its test tools.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
Each module ported from acronis-tib-reader or acronis-tibx now names
the project with its MIT copyright notice in its own docstring, so
THIRD_PARTY_NOTICES.md and its license-files entry are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE

@falklindner falklindner left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All five points are addressed in 8a0dce9 (stub), 7ca3ba0 (test layout) and 16095d6 (notices).

🤖 Generated with Claude Code


c_boot = cstruct(endian="<").load(boot_def)

PAGE_SIZE: int = c_tibx.PAGE_SIZE

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8a0dce9.

Comment thread pyproject.toml Outdated
Comment on lines +78 to +80
# Generated by `cstruct-stubgen`; reformatting or rewrapping it here would only be undone
# the next time the stub is regenerated.
extend-exclude = ["dissect/archive/tibx/c_tibx.pyi"]

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8a0dce9.

Comment thread tests/tibx/test_codec.py
Comment thread tests/_tools/tibx/_synth.py
Comment thread THIRD_PARTY_NOTICES.md Outdated
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.27437% with 107 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.64%. Comparing base (e2f7b9f) to head (16095d6).

Files with missing lines Patch % Lines
dissect/archive/tibx/lsm.py 86.17% 34 Missing ⚠️
dissect/archive/tibx/tibx.py 94.67% 22 Missing ⚠️
dissect/archive/tibx/segment.py 82.75% 15 Missing ⚠️
dissect/archive/tibx/map.py 95.60% 12 Missing ⚠️
dissect/archive/tibx/crypto.py 91.34% 9 Missing ⚠️
dissect/archive/tibx/page.py 93.38% 8 Missing ⚠️
dissect/archive/tibx/codec.py 88.70% 7 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main      #30       +/-   ##
===========================================
+ Coverage   69.25%   81.64%   +12.39%     
===========================================
  Files           9       20       +11     
  Lines        1187     2572     +1385     
===========================================
+ Hits          822     2100     +1278     
- Misses        365      472      +107     
Flag Coverage Δ
unittests 81.64% <92.27%> (+12.39%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@yunzheng yunzheng left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some feedback regarding the Compression. See my inline comments.

The current tests are mostly synthetic, so it would be nice to have fixture-based tests for:

  • a small real backup at “normal” (stored + Zstd), decoding every segment and checking the lengths
  • a small backup at “low”, which should produce LZ4 segments (none of our current samples have them)

zlib can stay synthetic, since the current writer never produces it. Maybe some of the real tibx archives already meet this, but I haven't checked.

Comment on lines +52 to +58
enum ComprLvl : uint8 {
NONE = 0, // "none"
LOW = 1, // "low"
NORMAL = 2, // "normal"
HIGH = 3 // "high"
};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix indenting of ComprLvl, and added ComprAlg enums. (Zlib was added).

Suggested change
enum ComprLvl : uint8 {
NONE = 0, // "none"
LOW = 1, // "low"
NORMAL = 2, // "normal"
HIGH = 3 // "high"
};
enum ComprLvl : uint8 {
NONE = 0, // "none"
LOW = 1, // "low"
NORMAL = 2, // "normal"
HIGH = 3 // "high"
};
enum ComprAlg : uint8 {
NONE = 0,
LZ4 = 1,
ZLIB = 2,
ZSTD = 3
};

uint32 length; /* uncompressed payload size */
uint32 zlength; /* compressed payload size */
uint32 key_id; /* encryption key id, 0 = plaintext */
uint16 compression; /* see COMP_* */

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the compression should be just uint8, and the other uint8 is some flag.

Suggested change
uint16 compression; /* see COMP_* */
ComprAlg compression; /* compression codec */
uint8 _kind; /* unknown, observed 0x00, 0x01, 0x02, 0x03 */

See my remarks on COMP_ in other comment

Comment on lines +329 to +336
# segment_header.compression variants
COMP_NONE = 0x0000
COMP_LZ4 = 0x0001
# Observed in True Image 2026 metadata streams (2 in partition backups, 3 in device
# metadata of USB sources), only ever stored (zlength == length); compressed forms of
# these variants have not been seen in the wild
COMP_STORED_VARIANTS = frozenset({0x0002, 0x0003})
COMP_ZSTD = frozenset({0x0300, 0x0301, 0x0302, 0x0303})

@yunzheng yunzheng Oct 28, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these definitions should go away, and solely use the ComprAlg enum.

I'm curious about the COMP_LZ4 = 0x0001, because I think that is incorrect, it should've been COMP_LZ4 = 0x0200. I wonder if some of these Compression codecs can be tested on the real Tibx test data.

It's possible that Zlib is legacy, as I don't see any writer for it in my version only a reader.

Comment on lines +117 to +118
def decompress(raw: bytes, segment: Segment) -> bytes:
"""Decompress a segment's raw payload to exactly ``segment.length`` plaintext bytes."""

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

decompress() needs a refactor, my clanker came up with something like this, but I have not tested it:

import zlib


def decompress(raw: bytes, segment: Segment) -> bytes:
    """Decompress a segment's raw payload to exactly ``segment.length`` plaintext bytes.

    Dispatches only on the codec byte; ``segment.kind`` plays no part in decoding.
    """
    alg = segment.compression

    if alg == ComprAlg.NONE:
        if len(raw) != segment.length:
            raise CorruptArchiveError(
                f"stored segment at page {segment.page_index} has zlength {len(raw)} "
                f"!= length {segment.length}"
            )
        return raw

    try:
        match alg:
            case ComprAlg.LZ4:
                out = lz4_block_decompress(raw, segment.length)
            case ComprAlg.ZLIB:
                out = zlib.decompress(raw)
            case ComprAlg.ZSTD:
                out = decompress_zstd(raw, segment.length)
            case _:
                raise UnsupportedFormatError(
                    f"segment at page {segment.page_index} uses unknown compression "
                    f"{int(alg):#04x} (kind {segment.kind})"
                )
    except (CorruptArchiveError, UnsupportedFormatError):
        raise
    except Exception as exc:  # codec-library errors (LZ4BlockError, zlib.error, ZstdError, ...)
        raise CorruptArchiveError(
            f"{ComprAlg(alg).name} segment at page {segment.page_index} failed to decompress: {exc}"
        ) from exc

    if len(out) != segment.length:
        raise CorruptArchiveError(
            f"{alg.name} segment at page {segment.page_index} decoded {len(out)} bytes, "
            f"expected {segment.length}"
        )
    return out

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for TIB and TIBX format (Acronis backup)

4 participants