Skip to content

[fix] Remove deleted sources from search - #452

Merged
bvolpato merged 4 commits into
mainfrom
bvolpato/fix-deleted-source-reuse
Oct 13, 2026
Merged

bvolpato merged 4 commits into
mainfrom
bvolpato/fix-deleted-source-reuse

Conversation

@bvolpato

@bvolpato bvolpato commented Oct 13, 2026 •

Copy link
Copy Markdown
Owner

Why

The recurring stale-deletion evidence in #441 includes a deleted file in SQLite and the Merkle snapshot, returned by an uncached query. The clean-checkout shortcut could certify transient source content even when both Git checks saw the same clean tree, then suppress reconciliation.

What changed

  • Certify indexed paths against tracked files and live metadata before recording clean reuse. Reconcile legacy markers once and apply explicit watcher deltas even when the checkout marker matches.
  • Exclude confirmed-missing sources before relevance filtering and result limits, and from cached daemon results, symbol definitions, and current context relationships. Recompute a cached query if deletion removes a hit, so live candidates can fill the requested limit. Preserve safe indexed-text fallback for other read errors.
  • Use request-local presence checks, borrowed path keys, and a reusable buffer to avoid descriptor traversal and repeated allocation for existing files. Unix F_OK success only confirms presence; every error still requires contained confirmation, and already-open readers keep their pinned root identity.
  • Cover inherited deletion tombstones, overlay-only removal, divergent restoration, base-identical restoration, and preservation of sibling/base results.
  • Add a required bounded daemon mutation correctness job to the protected CI gate. Check cached and uncached queries together with SQLite, Merkle snapshots, and worktree tombstones, retaining diagnostics on failure.
  • Correct merge-base semantics, unreleased safeguard availability, installation scope, and deletion behavior in the README, architecture, website, integration docs, and changelog.

Validation

  • ./test.sh: passed on 430286c, including formatting, ShellCheck, Clippy, frontend checks/build, Rust library/binary/integration tests, and Python harness tests.
  • cargo build --release --locked: passed on 430286c. Focused workspace-file, cached-search, deletion, overlay, unsafe-fallback, and all-indices regressions also passed.
  • uv run scripts/soak_daemon.py --binary <built-ig> --correctness-only --output <report>: passed with the final default release binary and current-head hash-only CI binary. Cached and uncached deleted stages return no hits; the stores reconcile and inherited deletions retain tombstones.
  • Current-head paired CI benchmark and soaks: passed the existing performance, retrieval-quality, resource, daemon mutation/restart, and MCP lifecycle/load/churn gates. All expected-recall losses are zero; daemon and MCP workloads report zero errors.
  • Initial local cold and distinct-query tail regressions did not reproduce in the prescribed reverse-order confirmation. Local tail intervals remain wide, so no local speedup or equivalence is claimed.
  • All current-head cross-platform CI and required protection checks passed. Publication and full public retrieval jobs intentionally skip on pull requests.
  • Markdown lint, actionlint, zizmor, local documentation link checks, and a local browser inspection: passed.
  • Independent implementation review and direct CLI checks: passed.

Evidence

Deterministic tracked edit/read/restore and untracked create/read/delete fixtures failed before the fix with a transient indexed source was certified as current, then passed after it. Search regressions also reproduced missing files occupying result limits and raising the relevance cutoff enough to suppress a live result.

A cached limit=1 search with two matches failed before the cache refill fix with cached deletion lost the live runner-up, then passed after recomputation was added. The regression also verifies unchanged warm-cache reuse and reuse of the refreshed result.

The final paired CI report compares main 048fc938 with the synthetic merge of 430286c. Its source tree matches the reviewed head exactly. Cache-replay p95 ratio is 1.126, with a 95% interval of 1.081 to 1.151. The existing guard rejects a regression whose confidence lower bound exceeds 1.10, so this passes the guard while retaining measurable overhead. No speedup or latency equivalence is claimed.

Risk

Existing clean markers require reconciliation once. Certification checks snapshot paths and source metadata when recording a reusable state; unchanged certified indexing retains its shortcut. Live query filtering checks file presence on every request, treats only contained NotFound as deletion, and adds measured cache latency. Further cache-path tuning remains a performance follow-up. Explicit Git-history context can still include deleted content labeled git_deleted. The fix and home/root confirmation safeguard remain unreleased relative to v1.3.0.

Checklist

  • Change is focused and follows existing project patterns.
  • Tests cover changed behavior or regression, or testing is not applicable.
  • ./test.sh --quick passed during iteration.
  • ./test.sh passes, or Validation explains narrower proportional checks.
  • Relevant CLI, daemon, MCP, browser, neural, or artifact E2E passes, or E2E is not applicable.
  • User-facing docs and changelog are updated when needed.
  • Performance and relevance claims include reproducible evidence.
  • No private code, credentials, generated caches, or local paths are committed.

Certify indexed inputs before clean-checkout reuse, reconcile legacy markers, and apply explicit watcher deltas. Exclude confirmed-missing sources from fresh and cached results before relevance filtering.

Cover worktree tombstones and restoration, require daemon mutation correctness in CI, and correct documentation of merge-base behavior and unreleased safeguards.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-13T03:41:46.055462Z 56a4de2 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 56a4de2f03

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/daemon.rs
Use successful metadata only to establish that absence is unconfirmed. Errors still require contained NotFound confirmation, and opened Unix readers preserve their pinned root identity. Cover deletion, restoration, unsafe paths, and replaced roots.
@bvolpato
bvolpato merged commit 756ff19 into main Oct 13, 2026
33 checks passed
@bvolpato
bvolpato deleted the bvolpato/fix-deleted-source-reuse branch October 13, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant