Repository navigation
Borrow the input of native functions - #146
Merged
Merged
Conversation
4 tasks done
Schamper
previously approved these changes
Oct 27, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #146 +/- ##
=====================================
Coverage 0.00% 0.00%
=====================================
Files 31 31
Lines 2179 2179
=====================================
Misses 2179 2179
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will not alter performance
Performance Changes
Comparing 
|
yunzheng
force-pushed
the
feature/native-borrow-input
branch
from
October 27, 2026 12:50
b45da01 to
94b241d
Compare
yunzheng
requested changes
Oct 27, 2026
yunzheng
left a comment
yunzheng
left a comment
Member
There was a problem hiding this comment.
@falklindner can you rebase and resolve conflicts and also check+fix for the same issues in crc64c.rs that was merged in #138 ?
The native LZ4 and LZO decompress and the CRC32C functions take their input as a Vec<u8>. pyo3 fills one from a bytes object as from any sequence, one Python int per byte, which costs about 10 ns a byte and is most of the time of a call: the CRC32C of a 4 KiB page took 34 us, of which 8 us is computing it. A bytes is now borrowed and a bytearray copied in one go, through a Buffer type the three share. Other sequences of ints, a memoryview among them, still take the old path, so every input that worked before still does. LZO also no longer copies the body after a header. Measured on Linux with Python 3.13, before and after: crc32c, 13 bytes 0.22 us 0.10 us crc32c, 4 KiB 34.3 us 8.2 us crc32c, 1 MiB 8.1 ms 2.1 ms lzo, 64 KiB block (20 KB in) 266 us 78 us lz4, 256 KiB block (95 KB in) 747 us 162 us Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNZJRJFni88V8JGtP2tj29
The native crc64c from fox-it#138 takes its input as a Vec<u8>, which pyo3 fills from a bytes object one Python int at a time. It now takes the same Buffer argument as crc32c. On Linux with Python 3.13, a 4 KiB input takes 7.7 us instead of 39.6 us. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE
falklindner
force-pushed
the
feature/native-borrow-input
branch
from
October 28, 2026 05:31
94b241d to
9235a51
Compare
Contributor
Author
|
Done: rebased onto main, and crc64c now borrows its input too (9235a51). The commit that moved the LZO test is dropped, as #143 is merged. 🤖 Generated with Claude Code |
yunzheng
approved these changes
Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The native
lz4.decompress,lzo.decompress,crc32candcrc64cnow borrow abytesinput and copy abytearrayin one go, through a sharedBufferargument type. Any other sequence of ints is still accepted as before. LZO also no longer copies the body after a header.Linux, Python 3.13, native:
crc64c(from #138) gains as much ascrc32c: on a second machine, a 4 KiB input takes 7.7 µs instead of 39.6 µs, and 1 MiB 1.9 ms instead of 10.2 ms.Test with
tox -e py3,native. New tests coverbytes,bytearrayandmemoryviewinput; a benchmark coverscrc32con a 4 KiB page.Closes #145
🤖 Generated with Claude Code
https://claude.ai/code/session_01HJvjcGbjsG74UunzfT5zNE