Repository navigation
fix(conserver): storage fallback for missing vCons + halt observability - #169
Merged
Merged
Conversation
When a vCon is no longer in Redis (TTL expired or evicted), VconRedis now falls back to the configured storage backends, returning the first hit and re-caching it back into Redis with VCON_REDIS_EXPIRY. The behavior is on by default and can be disabled with the new VCON_STORAGE_FALLBACK_ENABLED setting. Also refactors get_vcon to delegate to get_vcon_dict (single lookup path), and consolidates the read metric counters into one family: get_vcon_redis_miss / get_vcon_storage_hit / get_vcon_not_found. This makes every link that calls VconRedis.get_vcon resilient to the "vCon UUID hits the chain after Redis TTL" pattern without needing per-link guards. Refs CONSERVER-B7.
Guard against VconRedis.get_vcon returning None. With the new storage fallback this only happens when the vCon is truly gone (missed Redis and every configured storage), or when VCON_STORAGE_FALLBACK_ENABLED is off. In either case, dereferencing .dialog crashed the worker with 'NoneType' object has no attribute 'dialog'. Return None to halt the chain, matching the docstring contract and the pattern already used by transcribe, tag_router, jq_link, and check_and_tag. Fixes CONSERVER-B7.
The conserver.link.count counter previously fired with outcome="success" for both "return uuid" (chain continues) and "return None" (chain halts) — making it impossible to track which links are halting chains. Split into outcome="success" vs outcome="halt" based on whether the link returned a truthy value. Dashboards already filtering outcome="success" remain accurate (halts are simply excluded). New dashboards can pivot on outcome="halt" with link_name to see e.g. "deepgram halted N chains today."
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three small, layered changes addressing Sentry CONSERVER-B7 (
AttributeError: 'NoneType' object has no attribute 'dialog'indeepgram_link.run):Storage fallback in
VconRedis— when a vCon is no longer in Redis (TTL expired or evicted),get_vcon/get_vcon_dictnow iterate the configured storage backends, return the first hit, and re-cache it back into Redis withVCON_REDIS_EXPIRY. On by default; opt out withVCON_STORAGE_FALLBACK_ENABLED=false. This makes every link that callsVconRedis.get_vconresilient to the "vCon UUID hits the chain after Redis TTL" pattern without per-link changes.get_vconis also refactored to delegate toget_vcon_dictfor a single lookup path.deepgram_linkNone-guard — defense-in-depth for the case where the storage fallback also misses (or is disabled). Halts the chain cleanly per the docstring contract (return None) instead of dereferencing the missing vCon. Mirrors the existing guard pattern intranscribe,tag_router,jq_link, andcheck_and_tag.Halt-vs-success metric split —
conserver.link.countpreviously fired withoutcome="success"for both "return uuid" (continue) and "return None" (halt), making it impossible to track which links halt chains. Now usesoutcome="halt"when the link returned a falsy value. Existing dashboards filtering onoutcome="success"remain accurate; new dashboards can pivot onoutcome="halt"withlink_name.Fixes CONSERVER-B7.
Test plan
pytest tests/core/test_vcon_redis.py— 10 passing (4 pre-existing + 6 new fallback-path tests: Redis hit, Redis miss + storage hit (re-cache + sorted-set add), Redis miss + storage miss, fallback disabled, first-storage-errors-second-succeeds, dict-variant fallback)pytest tests/core/test_vcon_redis_ttl.py— 24 passing (no regression)pytest conserver/links/deepgram_link/tests/— 11 passing (1 newtest_run_halts_chain_when_vcon_not_found)pytest conserver/tests/test_link_metrics.py conserver/tests/test_vcon_concurrency.py— 34 passing (1 newtest_link_halt_records_halt_outcome)All tests run inside the
vcon-server-conserver-1container.🤖 Generated with Claude Code