Repository navigation
tp: have the executor end every step's run when its stream ends - #8332
Open
LalitMaganti wants to merge 1 commit into
Open
LalitMaganti wants to merge 1 commit into
LalitMaganti wants to merge 1 commit into
Conversation
LalitMaganti
force-pushed
the
dev/lalitm/step-state-lifetimes
branch
3 times, most recently
from
October 28, 2026 07:13
aae62b5 to
4295b58
Compare
LalitMaganti
force-pushed
the
dev/lalitm/step-state-lifetimes
branch
2 times, most recently
from
October 29, 2026 18:17
84c0e83 to
0487efd
Compare
LalitMaganti
marked this pull request as ready for review
October 29, 2026 18:17
LalitMaganti
force-pushed
the
dev/lalitm/step-state-lifetimes
branch
from
October 29, 2026 19:13
0487efd to
4eec006
Compare
LalitMaganti
force-pushed
the
dev/lalitm/step-state-lifetimes
branch
from
October 29, 2026 20:56
4eec006 to
0ccac95
Compare
LalitMaganti
force-pushed
the
dev/lalitm/step-state-lifetimes
branch
2 times, most recently
from
October 30, 2026 20:20
5eb9bc3 to
baca647
Compare
LalitMaganti
force-pushed
the
dev/lalitm/step-state-lifetimes
branch
from
October 30, 2026 21:20
baca647 to
422e2cb
Compare
**Why** A step let go of what it held only if it remembered to clear itself, and almost none did. GraphTree, the interval operators and the tree transforms kept everything they built until the next run. One `OperatorState` mixed per-call scratch, rows held for the whole input and the step's status, so nothing said what a step may keep or for how long. **Changes** - `operator.h` states the rules every step keeps to. Each field of a state declares its lifetime: `Held<T>` or `RunValue<T>` for the run, `CallScratch<T>` for one call, and anything else is set when the state is made. - `Held<T>` and `RunValue<T>` register with their state. The pipeline ends their run as soon as the step's stream ends, and again before a source is restarted. `Reset()` and `ResetEachRun` are gone. - States are made once and reused, because the same pipeline is often rerun for small queries. Ending a run keeps up to 64 KiB of each container's storage and releases anything larger. - The pipeline merges its states' registries by segment, so ending a run is one pass with no call per state. - A batch a step outputs never points into its state, so a run can end while its batches are still read. `GroupedSort` copies its group numbers into a buffer instead of lending them. - Each state's fields are grouped by what they do. **Results** `INCLUDE` of `heap_graph_dominator_tree` on `test/data/heap_graph_object_for_benchmarks.pftrace`, release build: | | Peak RSS | Buffers | |---|---:|---:| | #8331 | 305 MB | 57 MB | | This PR | 270 MB | 53 MB | Every heap graph table is byte-identical. Rerunning a one-row pipeline costs about 2-3 ns more per step that holds state. Large runs are unchanged. **Testing** - New `StateTest.AStepsRunEndsWithItsStream`. - The affected unit test suites pass.
LalitMaganti
force-pushed
the
dev/lalitm/step-state-lifetimes
branch
from
October 30, 2026 21:37
422e2cb to
e89c486
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A step let go of what it held only if it remembered to clear itself,
and almost none did. GraphTree, the interval operators and the tree
transforms kept everything they built until the next run. One
OperatorStatemixed per-call scratch, rows held for the whole inputand the step's status, so nothing said what a step may keep or for how
long.
Changes
operator.hstates the rules every step keeps to. Each field of astate declares its lifetime:
Held<T>orRunValue<T>for the run,CallScratch<T>for one call, and anything else is set when thestate is made.
Held<T>andRunValue<T>register with their state. The pipelineends their run as soon as the step's stream ends, and again before a
source is restarted.
Reset()andResetEachRunare gone.rerun for small queries. Ending a run keeps up to 64 KiB of each
container's storage and releases anything larger.
run is one pass with no call per state.
while its batches are still read.
GroupedSortcopies its groupnumbers into a buffer instead of lending them.
Results
INCLUDEofheap_graph_dominator_treeontest/data/heap_graph_object_for_benchmarks.pftrace, release build:Every heap graph table is byte-identical. Rerunning a one-row pipeline
costs about 2-3 ns more per step that holds state. Large runs are
unchanged.
Testing
StateTest.AStepsRunEndsWithItsStream.