Skip to content

fix(gzip): bound decompression memory - #581

Draft
CyMule wants to merge 8 commits into
mainfrom
fix/gzip-decompression-memory
Draft

fix(gzip): bound decompression memory#581
CyMule wants to merge 8 commits into
mainfrom
fix/gzip-decompression-memory

Conversation

@CyMule

@CyMule CyMule commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Decompress gzip uploads incrementally in 1 MiB chunks.
  • Keep decompressed outputs up to 1 MiB in memory and spool larger outputs to disk.
  • Proxy disk-backed spools so downstream MIME detection and partitioning do not copy them back into BytesIO.
  • Reuse the request-owned UploadFile so FastAPI closes the decompressed temporary file after the response.
  • Add coverage for memory-backed and disk-backed outputs, stream cleanup, and proxy file behavior.

Why

ungz_file currently reads the complete expanded payload into a bytes object before wrapping it in BytesIO. Peak Python memory therefore grows with the uncompressed file size.

A raw SpooledTemporaryFile is not sufficient by itself because existing downstream code recognizes that concrete type and copies its complete contents back into memory. Incremental decompression plus a normal file proxy avoids both document-sized allocations.

This does not introduce a file-size limit or change the partition API contract.

Impact

Local end-to-end benchmarks with valid documents produced byte-identical responses:

  • A 54.9 MB DOCX reduced median peak process RSS from 828.6 MiB to 759.9 MiB (-68.7 MiB, 8.3%) while median latency changed from 1.348s to 1.370s (+1.6%).
  • A 70.4 MB PDF reduced median peak process RSS from 1,029.5 MiB to 951.6 MiB (-77.9 MiB, 7.6%) while median latency changed from 0.532s to 0.529s (-0.6%).

In decompression-and-reread benchmarks, request-attributable RSS fell from 108.1 MiB to 5.3 MiB for the DOCX and from 133.6 MiB to 4.8 MiB for the PDF. That isolated stage was 4.0% and 8.3% slower respectively, but decompression was a small portion of the complete request.

Large expanded outputs use temporary disk space rather than equivalent heap memory.

Risk

Disk-backed outputs add temporary-file I/O. The 1 MiB copy buffer keeps this overhead small while maintaining bounded heap use.

Temporary-disk capacity remains the external bound; this change does not impose a maximum expanded size.

Validation

  • uv run pytest test_general/api/test_gzip.py -q — 3 passed, 28 xfailed.
  • uv run ruff check --select I prepline_general/api/general.py test_general/api/test_gzip.py — passed.
  • Valid 54.9 MB DOCX and 70.4 MB PDF responses were byte-identical before and after.
  • Paired decompression-and-reread and complete API benchmarks covered memory-backed and disk-backed behavior.

Review in cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant