Skip to content

fix(partition): avoid copying spooled uploads - #4419

Draft
CyMule wants to merge 2 commits into
mainfrom
fix/avoid-spooled-upload-copies
Draft

fix(partition): avoid copying spooled uploads#4419
CyMule wants to merge 2 commits into
mainfrom
fix/avoid-spooled-upload-copies

Conversation

@CyMule

@CyMule CyMule commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Reuse SpooledTemporaryFile uploads directly on supported Python versions.
  • Rewind spooled streams before passing them to downstream consumers.
  • Remove document-sized BytesIO copies from DOCX and PPTX partitioning.
  • Update regression tests to verify that the original stream is reused.

Why

Python 3.11 and newer provide the buffered-I/O methods required by zipfile and other consumers. Converting a spooled upload to BytesIO therefore adds a document-sized heap allocation without providing compatibility value.

This change removes that allocation without introducing a file-size limit or changing the partition API.

Impact

In a representative end-to-end partition API benchmark using a 54.9 MB DOCX upload:

  • Peak process RSS decreased by 51.2 MiB.
  • Peak container memory decreased by 52.6 MiB.
  • Median latency changed from 9.343s to 9.197s (-1.6%).

This removes approximately one document-sized allocation without a measured performance penalty.

Validation

  • Focused common, DOCX, and PPTX tests — 3 passed.
  • uv run ruff check unstructured/partition/common/common.py unstructured/partition/docx.py unstructured/partition/pptx.py test_unstructured/partition/common/test_common.py test_unstructured/partition/test_docx.py test_unstructured/partition/test_pptx.py — passed.

Review in cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant