Skip to content

feat(dream): add reversible lifecycle observation and maintenance - #2561

Open
Cookie0103 wants to merge 8 commits into
MemTensor:mainfrom
Cookie0103:feat/dream-lifecycle-maintenance
Open

Cookie0103 wants to merge 8 commits into
MemTensor:mainfrom
Cookie0103:feat/dream-lifecycle-maintenance

Conversation

@Cookie0103

Copy link
Copy Markdown

Description

Dream archive/update/merge actions currently call the graph writer with an incompatible signature, while the maintenance entry point is unimplemented. This proposal repairs those native writes and adds one opt-in lifecycle shared by Dream persistence, final retrieval observation, explicitly affected successful feedback and bounded maintenance.

Related design proposal: #2560. The issue is published; maintainer acceptance and the contribution guide's target branch are pending. The documented bare dev branch does not exist, so this contribution is provisionally based on current main; it can be retargeted to the requested versioned development branch.

The initial atomic backend is PostgreSQL. Scoped row-lock reductions, transactional rollback and ordinary-write protection preserve committed lifecycle evidence and protected archive status across stale updates and Context upserts. Retention recomputes from fixed event-time anchors, separates retrieval from attributed usefulness, bounds deduplication by age/capacity, rejects policy reinterpretation/corrupt state, respects pins for passive retention and requires deliberate recovery for invalidated evidence. Original confidence, content and embeddings remain separate from retention state.

Maintenance reads metadata-only keyset pages, revalidates current evidence under locks and publishes scoped/versioned JSON cursors after committed pages. Reports retain bounded decisions/revisions/reasons and exact page retry IDs; dry runs create no state or checkpoint files. Explicit cube registration and optional periodic execution allow quiet cubes to age. Notifications observe detached final results after pipes, preserve the single-provider version hook and report delivery separately from business success. No public route, request/response model, database schema or core dependency is added.

Type of change

  • Bug fix: native Dream graph write contract and protection of committed evidence.
  • New opt-in feature: lifecycle observation, reversible retention and resumable maintenance.

Validation

  • Native PostgreSQL 17.11 + pgvector 0.8.1, Python 3.12.13, disposable test-owned schemas: 200 Dream/plugin/search-hook/feedback/graph/textual tests passed. The external socket guard prohibited model/external requests.
  • make format passed over 644 Python files. Repository pre-commit hooks on changed files passed.
  • Unrestricted make test cannot collect two optional Torch test files in this environment. The identical missing-Torch errors occur on the untouched complete base snapshot.
  • Broader suite excluding those two files: branch 872 passed, 12 failed, 64 skipped, 9 errors; untouched base 844 passed, 12 failed, 9 skipped, 9 errors. The 21 failed/error node IDs are identical. These include unavailable optional dependencies/services; this is not a claim of a fully green repository suite. Backend lifecycle tests explicitly skip without MEMOS_TEST_PG_DSN; the native run above supplied it.
  • Controlled native metadata sweeps of 100/1,000/10,000 old memories completed in 0.040/0.366/4.069 seconds with pages/batches bounded at 100. Replay archived zero additional memories and preserved metadata hashes/revisions. A legal 1,000-row page produced a 179,339-byte cursor and replayed successfully. These single local measurements exclude PostgreSQL server memory and are not agent-quality or production-throughput claims.
  • Native observation/outcome deduplication, explicit archive/restore and source/content/vector preservation were exercised. Tests cover rollback, scope, stale writers, protected archive, malformed/time-invalid metadata, receipt capacity, delayed delivery clock, cursor corruption, page failure, checkpoint failure and active-work shutdown acknowledgment.
  • An independent automated whole-module review reproduced two P2 findings. Both now have native RED→GREEN regressions: unknown/malformed envelope versions cannot authorize ordinary reactivation, and newly allocated feedback/version-archive identities do not inherit old observations/receipts. Follow-up actual writer-to-model tests also repaired native Dream source/UUID-prefix compatibility while retaining UUID validation. The original three review proofs pass with corrected assertions. Independent automated fix-diff verification passed 203 native/affected/textual tests including the adapted original proofs, with no new blocker.

Example native test command:

MEMOS_TEST_PG_DSN='host=localhost port=5432 dbname=disposable_lifecycle user=tester' \
  pytest tests/dream -q

Install the existing tree/scheduler extras and the PostgreSQL backend's documented psycopg2-binary dependency. The DSN must point to a disposable database where tests can create/drop their own schemas. No model API key is needed.

Limits and review context

Notifications are best effort, not a durable outbox or transaction with recall/feedback writes. Only committed observations protect a racing archive. Explicit SDK events need stable producer IDs and original timestamps; automatic observation covers the decorated API search path only. Receipt age/capacity is bounded, and source IDs cannot be recycled into an unlimited exactly-once guarantee. Cooperative limits/shutdown do not forcibly terminate arbitrary blocked backend calls. The current capability is PostgreSQL only; unsupported backends fail clearly when enabled. No improved QA, causal attribution, calibrated confidence or privacy-unlearning claim is made.

AI assistance disclosure: Codex automation produced the implementation, tests, documentation and local checks. Automated self-review and independent automated review are reported separately; the submitting user is not claimed to have personally reviewed or run them. No CLA statement or maintainer approval is asserted.

Contribution checklist

  • Added tests and comments for the implemented behavior.
  • Linked the design issue and included reproduction/configuration details.
  • Added repository-local usage documentation. No change to the separately hosted documentation site is proposed here.
  • Disclosed automated authorship and validation limitations.
  • Maintainer review and any requested target-branch alignment remain pending.

Production Python diff at dc03795: +2,158/-150 physical lines; +1,917/-137 nonblank; +1,765/-85 code-token lines excluding comments/docstrings. Tests +1,398 and Markdown +90 are separate. The caller's actual module scope is reported without padding. No-envelope legacy upsert semantics remain the existing baseline; this change does not certify general tenant isolation for all legacy writer paths. Explicit archive is a deliberate caller operation, including ownership of an already archived target; it is not a passive retention decision.

@Memtensor-AI Memtensor-AI added area:database graph_db + vector_db | 图数据库与向量数据库 area:memory 记忆存储、检索、更新、召回逻辑 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:database graph_db + vector_db | 图数据库与向量数据库 area:memory 记忆存储、检索、更新、召回逻辑 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants