Draft
Conversation
…t latency Pre-initializes lazy components before streaming starts and uses first N real requests to warm up KV store connections, CatalystUtil.session, Janino codegen, and derivation UDF on executor JVMs — eliminating the 5-10 minute timeout spike on new deploys. - Add PoolMap.warmup() and PooledCatalystUtil.warmup() to pre-populate the CatalystUtil pool beyond the default initialSize=2 - Add driver-side warmupDriver(): forces CatalystUtil.session, TTLCaches (GroupByServingInfo, JoinCodec), deriveFunc + CatalystUtil pool - Add executor-side warm-up in enrichBaseJoin: runs first N real rows through fetchBaseJoin (60s timeout) then invokes deriveFunc with real base values to warm up UDF lazy state and JIT; results discarded and all rows re-processed normally - Fallback: if fetchBaseJoin times out, still force deriveFunc init so enrichModelTransforms avoids CatalystUtil cold-start timeout Config (spark.chronon.stream.chain.*): warmup.enabled=true, warmup.request_count=10, warmup.timeout_seconds=60, warmup.pool_size=4
pengyu-hou
force-pushed
the
pengyu--chaining-warm-up
branch
from
2cc35e2 to
e848f48
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR will add warm up methods for the JoinSourceRunner for the model transform pipelines. We are seeing timeout errors and traffic spikes for the first ten mins of the streaming job.
Why / Goal
Resolve the timeout issue for model transform
Test Plan
Checklist
Reviewers