Repository navigation
Commit 64388b5
authored
fix(plugin): isolate Windows DLLs and resolve DSH memory issues (#2519)
## Description
Fix problems found while investigating #2495: Windows native model
dependencies can conflict with host Sharp/libvips, a recalled memory can
abort a DSH V4 turn, malformed import rows are silently reported as
skipped, and local model source settings are unavailable or ignored.
- Isolate Windows local embeddings in a shared, lazy child process.
Transformers/Sharp/ONNX stay outside the host process; requests,
cancellation and progress use IPC. Remove inherited host preloads,
restart failed children, release the child after the last provider
closes, and reclaim idle processes. macOS/Linux retain the in-process
path. No DLL replacement, production dependency changes or host-version
pinning.
- Give both recall and auxiliary host-LLM messages the producer-owned
`plugin:memos-local-memory` source kind accepted by DSH V4.
- Validate bundle rows before writing, report duplicates/invalid
rows/database failures separately, and roll back a rejected trace
together with any synthetic parents. Keep the existing
`imported`/`skipped` response fields.
- Add write-free import previews, explicit preserve/target-profile
ownership, and active-profile visibility counts. Existing IDs remain
duplicates and sharing scopes are preserved.
- Add local embedding mirror/model/cache directories, offline mode, a
bounded initial wait, and download/load logs. Explicit mirror
configuration overrides `HF_ENDPOINT`; default model caching lives
outside installed package files. Serialize differently configured loads
and restore Transformers' shared environment afterward.
- Expose these options in Viewer Settings/Import, test unsaved local
model settings, and document configuration and import semantics. Remove
an accidental real-model dependency from the Hermes summarizer-status
unit test.
No production dependency or host peer-range changes. V4 regression tests
add the pinned official codec as a dev dependency, align its Cordis dev
peer, and pin existing brand/protocol dev peers; both lockfiles are
updated. Changes are confined to `apps/memos-local-plugin/`.
Related Issue (Required): #2495 (partial follow-up). **Windows
sharp/libvips isolation is now included. Native Windows validation is
pending; keep the issue open until both load orders and the original
long-session case are confirmed on Windows.**
Reviewers: @hijzy and @whipser030 (assigned by the repository).
## Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] Documentation update
## How Has This Been Tested?
Run from the repository root:
```sh
make format
cd apps/memos-local-plugin
npm run lint
npm test
npm run build:package
node node_modules/typescript/bin/tsc --noEmit --strict --target ES2022 --module ES2022 --moduleResolution Bundler --jsx react-jsx --jsxImportSource preact --lib ES2022,DOM --esModuleInterop --skipLibCheck viewer/src/main.tsx
```
Results: Ruff checks passed (630 Python files unchanged); TypeScript,
strict Viewer type checking, and package/Viewer builds passed. Vitest:
**190 test files passed; 1638 tests passed, 3 skipped**. Repository
pre-commit checks passed over the changed files before committing.
- [x] Unit Test — structured world-model round trips (including
evidence/legacy strings), malformed structure, session/episode ownership
conflicts in preview and writes across batches, stable UI/headless
preset ownership, explicit-profile isolation, and backup/transactional
legacy-profile migration; Windows process isolation without host
Transformers import, concurrent response/progress routing, cancellation,
crash/restart, startup/IPC failure, invalid vectors and shared-provider
cleanup; source attribution, real V4 codec admission/encoding for
recall, pre-step inbox batches and host-LLM output, plus rejection of
retired sources; local mirror precedence, offline paths, stable cache,
shared-environment restoration, serialized loads, progress and
timeout/warmup; SQLite import validation, rollback, preview and
ownership.
- [x] Test Script Or Test Steps (please provide) — On macOS with
official DSH **0.2.0-rc.2**, use an isolated DSH profile/data home and
the logged-in `deepseek-flash` account. Load this build, import a
synthetic deployment-region memory, and ask `deployment region` without
including the stored answer. Recall injected the matching memory,
`memos_get` returned its contents, and the model answered the stored
value with a completed turn and no V4 admission error. Restore the
original test installation afterward and verify its hash. A real cached
local model also returned a 384-dimensional embedding in offline mode.
- [x] Pipeline Automated API Test (please provide) —
`tests/unit/server/http.test.ts` verifies import options and
invalid-option responses. An isolated HTTP smoke test verified zero
writes during preview; one valid, one duplicate and one invalid
timestamp row were classified correctly; target ownership made the
imported row visible.
## Checklist
- [x] I have performed a self-review of my own code | 我已自行检查了自己的代码
- [x] I have commented my code in hard-to-understand areas |
我已在难以理解的地方对代码进行了注释
- [x] I have added tests that prove my fix is effective or that my
feature works | 我已添加测试以证明我的修复有效或功能正常
- [x] I have created related documentation issue/PR in
[MemOS-Docs](https://github.com/MemTensor/MemOS-Docs) (if applicable) |
我已在 [MemOS-Docs](https://github.com/MemTensor/MemOS-Docs) 中创建了相关的文档
issue/PR(如果适用) — N/A: local-plugin configuration and import behavior are
documented in this package's README.
- [x] I have linked the issue to this PR (if applicable) | 我已将 issue
链接到此 PR(如果适用)
- [x] I have mentioned the person who will review this PR | 我已提及将审查此 PR
的人
## Reviewer Checklist
- [x] Issue linkage and scope reviewed: relates to #2495; keep the issue
open pending native Windows DLL verification.
- [x] Made sure local checks passed; GitHub CI results are tracked
separately on this PR.
- [x] Tests have been provided
### V4 regression coverage
Bridge tests now use the production recall factory and assert
producer-owned source kinds. The published
`@deepseek-ai/dsh-session-format-v3-to-v4@0.2.0-rc.2` validates both
individual messages and the query/recall inbox batch through
`assertV4RowAdmission` and `encodeEvent`. Host-LLM auxiliary messages
pass without `form`; both old source shapes are rejected. A mutation
check restoring `kind: "plugin"` in the real factory made the admission
regression fail with the original error. Tests require no desktop
installation or account.
### Windows isolation validation
- **Cross-platform checks completed:** real Node and official DSH
Electron subprocess smoke tests each returned two concurrent
384-dimensional vectors, with a separate worker PID and no
Sharp/libvips/ONNX native libraries added to the parent process. `npm
pack --dry-run --ignore-scripts` confirmed all three worker runtime
modules are included.
- **Native Windows check pending:** build the branch and run
`scripts/verify-windows-dll-isolation.mjs --host-sharp <host sharp
package path> [--cache-dir <existing model cache>] [--offline]`. The
README includes PowerShell instructions for executing with DSH's
Electron executable when host Sharp is in ASAR. Each load order runs in
a fresh process and verifies both embedding output and real PNG encoding
using host Sharp. No native Windows pass is claimed from macOS.
- Restart DSH completely before testing the plugin; confirm new and long
conversations plus host image processing after local-model
initialization. Electron hosts need
`ELECTRON_RUN_AS_NODE`/`child_process.fork` support; failure does not
fall back to loading native model libraries into the host.
### QA report follow-up
Address the verified findings from the independent QA report:
- Preserve imported world-model `structure`, including labelled evidence
entries and legacy string entries. Reject malformed structures with
field diagnostics instead of silently clearing them.
- Reject traces whose existing/planned episode belongs to another
session or whose parent records belong to another memory owner. Preview
tracks accepted parents across batches and applies the same checks
without writing records.
- Use the configured MemOS `profileId` consistently for DSH recall,
capture, tools and Viewer; `agentPreset` no longer changes memory
ownership. Explicit profiles stay isolated. Older preset-owned records
are not silently merged: the packaged `migrate-dsh-preset-profile.mjs`
tool defaults to read-only preview and requires a new SQLite backup
before an atomic, explicitly scoped move. Skill-name/constraint
conflicts roll back the entire move.
- Clarify that the Viewer button **Repair embeddings** invokes `POST
/api/v1/embeddings/rebuild` with `mode: repair`.
Validation on the final changes: **190 test files passed; 1638 tests
passed, 3 skipped**; `npm run lint`, package/Viewer build, `make format`
(630 Python files unchanged), pre-commit, and npm package-content checks
passed. Added regressions failed against the old implementation before
the fixes. No schema migration or automatic modification of existing
user databases is introduced. Windows native verification remains
pending as documented above.56 files changed
Lines changed: 4031 additions & 1658 deletions
File tree
- .github/workflows
- apps/memos-local-plugin
- adapters
- deepseek-harness
- hermes
- agent-contract
- core
- config
- embedding
- providers
- pipeline
- scripts
- server/routes
- tests
- helpers
- unit
- adapters
- embedding
- install
- pipeline
- scripts
- server
- viewer/src
- stores
- views
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
388 | 388 | | |
389 | 389 | | |
390 | 390 | | |
391 | | - | |
392 | | - | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | | - | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
397 | 395 | | |
398 | 396 | | |
399 | 397 | | |
400 | 398 | | |
401 | 399 | | |
402 | 400 | | |
403 | | - | |
| 401 | + | |
404 | 402 | | |
405 | 403 | | |
406 | 404 | | |
| |||
492 | 490 | | |
493 | 491 | | |
494 | 492 | | |
495 | | - | |
496 | | - | |
497 | | - | |
498 | | - | |
| 493 | + | |
499 | 494 | | |
500 | | - | |
501 | | - | |
502 | | - | |
503 | | - | |
504 | | - | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
505 | 499 | | |
506 | 500 | | |
507 | 501 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
3 | 67 | | |
4 | 68 | | |
5 | 69 | | |
| |||
261 | 325 | | |
262 | 326 | | |
263 | 327 | | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
264 | 435 | | |
265 | 436 | | |
266 | 437 | | |
| |||
0 commit comments