fix(db): prevent nested-loop observation scoring - #4168
Closed
r266-tech wants to merge 1 commit into
Closed
Conversation
koriyoshi2041
approved these changes
Sep 6, 2026
koriyoshi2041
left a comment
Contributor
There was a problem hiding this comment.
Validated the exact head locally against PostgreSQL: the fused-query and scoring suites pass 10/10, including distinct shared-source scores, wide source arrays, per-entity caps, ordering, and the new plan-shape guard. Ruff check/format and diff hygiene also pass. The FULL JOIN preserves matched candidate scores while preventing an equality full join from degrading to the reported nested-loop shape.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scoredCTE and use an equalityFULL OUTER JOIN, which PostgreSQL must execute as a hash or merge join even whenconnected_sourcesis estimated at one rowThis intentionally does not add
MATERIALIZEDtoconnected_sources: it is already referenced twice and therefore materialized by PostgreSQL, and an explicit keyword did not change the bad cardinality estimate in PostgreSQL 18.Closes #4163
Validation
3.0and1.0EXPLAIN ANALYZE, normal statistics: 73ms total versus 75ms for the original query in the same 37.7k-unit/150.8k-edge fixtureHash Full Joinretained; scoring join ~1.3msuv run --extra embedded-db pytest tests/test_observation_expansion_single_fetch.py tests/test_observation_expansion_scoring.py -q -n0(3 passed, 7 skipped because local ML extras are unavailable)uv run --frozen ruff check hindsight_api/engine/db/ops_postgresql.py tests/test_observation_expansion_single_fetch.pyuv run --frozen ruff format --check hindsight_api/engine/db/ops_postgresql.py tests/test_observation_expansion_single_fetch.pyuv run --frozen ty check hindsight_apigit diff --checkThe repository-wide lint hook also ran; all Python checks passed, while the unrelated control-plane ESLint task could not start because
@eslint/jsis not installed in this checkout.CI follow-up
The only red job is the free-threaded 3.14 full-suite run. Two of its three failures are the same untouched OpenCode/named-result failures present on recent unrelated runs for #4164, #4159, and #4158. The remaining
test_parallel_fans_out_across_schemasshort count is also outside this diff and passed 3/3 isolated repetitions locally. All other 14 executed jobs passed, including generated-file verification and Python 3.11–3.14 builds; no changed SQL/test failure was reported, so the verified head is unchanged.