Repository navigation
Flush the allocator when freed memory could push RSS past the limit - #9
Merged
Merged
Conversation
The buffer pool accounts for allocated buffers, but freed buffers stay resident in jemalloc as dirty pages until its one second decay runs on the arena that owns them. During spilling this retained memory grew to roughly 70% of the memory limit on TPC-DS q67 and got the process killed inside a cgroup while the tracked usage was within the limit. Count the bytes handed back to the block allocator, which is where every buffer and operator allocation is freed, and purge the allocator on the next buffer pool allocation once at least a sixteenth of the memory limit has been freed and the tracked usage plus that amount exceeds the limit. Resident memory then stays within the memory limit plus a sixteenth, while queries that run well below the limit never pay for a purge. Counting at the allocator rather than in the pool's accounting keeps buffer reuse during eviction, tag changes and failed reservations from being mistaken for frees. The allocator_bulk_deallocation_flush_threshold setting keeps its single-deallocation meaning. (cherry picked from commit 71e7ab8)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The buffer pool accounts for allocated buffers, but freed buffers stay resident in jemalloc as dirty pages until its one second decay runs on the arena that owns them. During spilling this retained memory grew to roughly 70% of the memory limit on TPC-DS q67 and got the process killed inside a cgroup while the tracked usage was within the limit. Count the bytes handed back to the block allocator, which is where every buffer and operator allocation is freed, and purge the allocator on the next buffer pool allocation once at least a sixteenth of the memory limit has been freed and the tracked usage plus that amount exceeds the limit. Resident memory then stays within the memory limit plus a sixteenth, while queries that run well below the limit never pay for a purge. Counting at the allocator rather than in the pool's accounting keeps buffer reuse during eviction, tag changes and failed reservations from being mistaken for frees. The allocator_bulk_deallocation_flush_threshold setting keeps its single-deallocation meaning.
(cherry picked from commit 71e7ab8)