blog: What We Got Wrong About Agent Memory

[benfrank241]

Six decisions we shipped and later reversed. Every one is traced to a commit in this repository rather than invented for the piece — the commit bodies here are unusually candid, so the material was already written.

The six

1. We optimized the thing that looked expensive. Embeddings were the obvious culprit for retain's memory profile. Measurement showed two whole-document operations dominated, both running before a single fact or embedding existed.
2. We published a speedup we hadn't earned. perf(embeddings): non-coalesced callers issue their batches serially #4039 attributed a retain throughput figure to embedding-request serialization and predicted 3.1x end-to-end. The follow-up says the attribution was wrong and claims no end-to-end number.
3. We engineered a workaround we could have deleted. count_tokens_windowed, its helpers, and six call sites, all removed when a tokenizer change made the problem disappear and gave exact counts for less memory than the approximation cost.
4. We made failure quiet, in three places. Silent config defaults, Memory Defense skipping screening on a malformed policy, and reflect storing a placeholder when a run produced no answer.
5. We treated every source as the same kind of claim. Diffs and transcripts consolidated into one belief set, so an idea floated in chat and never implemented became indistinguishable from committed behaviour. Not fixable by configuration.
6. We inferred a mode instead of being told it. Passthrough detection by noticing identical scores, which a real cross-encoder can also produce.

Why it closes the way it does

Read together they're two mistakes wearing different clothes: three were believing a plausible story instead of measuring one, three were making failure invisible.

The second is the sharper lesson for this domain specifically. In a stateless service a placeholder is one bad request. In a memory system it's a wrong fact that persists and gets recalled — the artifacts of graceful degradation don't end when the request does.

Notes for review

• Dated 2026-09-04 to avoid a third post on 09-03.
• Item 4 groups three separate reversals under one heading. Keeps it tight, but one of the "six" is really three — flagging in case you'd rather split it.
• All three cross-links verified live.

Content score

Composite 88.2/100 (humanity 100, specificity 100, readability 90, zero long paragraphs).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AE4NCnCKXUzNDF3N9FaQja