Skip to content

feat(io): detect input compression by magic bytes, optional bz2/zstd/xz - #272

Open
BenjaminDEMAILLE wants to merge 1 commit into
mainfrom
feat/compressed-input-detection
Open

BenjaminDEMAILLE wants to merge 1 commit into
mainfrom
feat/compressed-input-detection

Conversation

@BenjaminDEMAILLE

Copy link
Copy Markdown
Contributor

Closes #218.

  • Input format is detected from the first bytes (files, pipes, stdin), not the extension. --readFilesCommand still takes precedence.
  • gzip is always decoded with MultiGzDecoder (multi-member input is read in full).
  • Optional pure-Rust features: bz2 (bzip2 0.6), zstd (ruzstd 0.9, with a multi-frame adapter), xz (lzma-rust2 0.21), and compressed-input for all three. Default features are unchanged.
  • niffler was not used: it only offers C backends for zstd and xz.
  • A detected format whose feature isn't built stops with an error naming the feature and a --readFilesCommand alternative.

Behaviour change to review: a plain file named *.gz is now read as plain text, and gzip is decoded whatever the filename. Documented in DIVERGENCE.md §4.3. The Parameters_openReadsFiles.cpp citation there should be double-checked.

Conflicts expected: #219 (same MultiGzDecoder change in fastq.rs), #225 (rapidgzip would move into the Gzip arm of compression.rs; both add [features]).

Tests: fmt, clippy 0 warnings, cargo test with default and all features. Test data is generated at runtime.

🤖 Generated with Claude Code

FASTQ input is now identified by content instead of extension. gzip is
always decoded with MultiGzDecoder; bzip2, zstd and xz are optional
pure-Rust features (bz2, zstd, xz, compressed-input). A detected format
whose feature is not built fails with an error naming the feature and a
--readFilesCommand alternative.

Closes #218

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Read other compressed input formats (bz2 / zstd / xz): evaluate niffler

1 participant