diff --git a/docs/en/guides/cli.md b/docs/en/guides/cli.md index 553f974..870274e 100644 --- a/docs/en/guides/cli.md +++ b/docs/en/guides/cli.md @@ -178,7 +178,9 @@ Features: | Command | Key args | Description | |---------|----------|-------------| | `add` | `--content`(req) `--source` `--tags` | Write a context item, returns `{id, stage}` | -| `retrieve` | `--query`(req) `--k`(10) `--full` `--json` `--tags` | Ranked SearchHits; L1 summaries by default, `--full` for L0. `--tags a,b` requires returned items to carry all listed tags | +| `retrieve` | `--query`(req) `--k`(10) `--full` `--json` `--tags` `--tag-match` | Ranked SearchHits; L1 summaries by default, `--full` for L0. `--tags a,b` filters by tag; `--tag-match all` (default) requires every tag, `any` requires at least one | +| `sync` | `path`(req) `--scope` `--dry-run` | Import notes/documents from a file or directory (auto-detect format) | +| `sync-obsidian` | `path`(req) `--scope` `--dry-run` | Incremental Obsidian vault sync: skips unchanged files, maps `[[wikilinks]]` to `Link` edges, soft-deletes vanished files | | `expand` | `--ids`(req, comma-separated) | Expand retrieved ids to L0 full content | | `items` | `--stage`(raw/extracted/knowledge/skill) | List all items in a scope | diff --git a/docs/en/reference/api.md b/docs/en/reference/api.md index 391506b..92e3a48 100644 --- a/docs/en/reference/api.md +++ b/docs/en/reference/api.md @@ -115,7 +115,7 @@ See [DataPlugs](../guides/integrations/dataplugs.md) for available plug types. ## Read -### `retrieve(query, *, scope, k=10, full=False, stage=None, tags=None, filters=None, include_deleted=False) → RetrieveResponse` +### `retrieve(query, *, scope, k=10, full=False, stage=None, tags=None, tag_match='all', filters=None, include_deleted=False) → RetrieveResponse` Ranked semantic search. Returns a `RetrieveResponse` iterable of `SearchHit` objects. @@ -128,8 +128,9 @@ By default returns **L1 summaries** (token-efficient). Pass `full=True` to recei | `k` | `10` | Maximum hits to return | | `full` | `False` | `True` → L0 bodies; `False` → L1 summaries (call `expand()` for L0) | | `stage` | `None` | Filter by `Stage` enum value | -| `tags` | `None` | All listed tags must match (AND filter) | -| `filters` | `None` | Dict bag: may include `stage`, `tags`, `min_confidence` | +| `tags` | `None` | Tag filter list; matching controlled by `tag_match` | +| `tag_match` | `'all'` | `'all'` — item must have every tag (AND); `'any'` — at least one tag (OR) | +| `filters` | `None` | Dict bag: may include `stage`, `tags`, `tag_match`, `min_confidence` | | `include_deleted` | `False` | Whether soft-deleted items appear in results | **Returns:** `RetrieveResponse` — iterable as `for hit in response`. Each `hit` has: diff --git a/docs/zh/guides/cli.md b/docs/zh/guides/cli.md index e50f526..4563d8b 100644 --- a/docs/zh/guides/cli.md +++ b/docs/zh/guides/cli.md @@ -178,7 +178,9 @@ sync 会跳过写入时的冲突检测以保证批量导入速度；导入后用 | 命令 | 关键参数 | 说明 | |------|----------|------| | `add` | `--content`(必填) `--source` `--tags` | 写入一条上下文，返回 `{id, stage}` | -| `retrieve` | `--query`(必填) `--k`(10) `--full` `--json` `--tags` | 检索排序后的 SearchHit；默认 L1 摘要，`--full` 返回 L0 全文。`--tags a,b` 要求返回项同时包含列出的所有标签 | +| `retrieve` | `--query`(必填) `--k`(10) `--full` `--json` `--tags` `--tag-match` | 检索排序后的 SearchHit；默认 L1 摘要，`--full` 返回 L0 全文。`--tags a,b` 按标签过滤；`--tag-match all`（默认）要求包含所有标签，`any` 只需匹配一个 | +| `sync` | `path`(必填) `--scope` `--dry-run` | 从文件或目录导入笔记/文档（自动检测格式） | +| `sync-obsidian` | `path`(必填) `--scope` `--dry-run` | 增量同步 Obsidian vault：跳过未变文件，将 `[[wikilink]]` 映射为 `Link` 边，对消失文件做软删除 | | `expand` | `--ids`(必填，逗号分隔) | 把已检索 id 升档到 L0 全文 | | `items` | `--stage`(raw/extracted/knowledge/skill) | 列举 scope 内全部 item | diff --git a/docs/zh/reference/api.md b/docs/zh/reference/api.md index 93bcad7..9992352 100644 --- a/docs/zh/reference/api.md +++ b/docs/zh/reference/api.md @@ -114,7 +114,7 @@ ctx.plug(RAGPlug(results=my_rag_results), scope="acme/kb/general") ## 读取 -### `retrieve(query, *, scope, k=10, full=False, stage=None, tags=None, filters=None, include_deleted=False) → RetrieveResponse` +### `retrieve(query, *, scope, k=10, full=False, stage=None, tags=None, tag_match='all', filters=None, include_deleted=False) → RetrieveResponse` 排名语义检索，返回 `SearchHit` 的可迭代 `RetrieveResponse`。 @@ -127,8 +127,9 @@ ctx.plug(RAGPlug(results=my_rag_results), scope="acme/kb/general") | `k` | `10` | 最多返回命中数 | | `full` | `False` | `True` 返回 L0 正文；`False` 返回 L1 摘要 | | `stage` | `None` | 按 Stage 枚举值过滤 | -| `tags` | `None` | AND 过滤：所有标签必须全部匹配 | -| `filters` | `None` | 字典包：可含 `stage`、`tags`、`min_confidence` | +| `tags` | `None` | 标签过滤列表，匹配方式由 `tag_match` 控制 | +| `tag_match` | `'all'` | `'all'`：条目必须包含所有标签（AND）；`'any'`：至少匹配一个标签（OR） | +| `filters` | `None` | 字典包：可含 `stage`、`tags`、`tag_match`、`min_confidence` | | `include_deleted` | `False` | 是否包含软删除条目 | **返回：** `RetrieveResponse` — 可用 `for hit in response` 遍历。每个 `hit`： diff --git a/src/contextseek/cli/doctor_cmd.py b/src/contextseek/cli/doctor_cmd.py new file mode 100644 index 0000000..2e7aeb0 --- /dev/null +++ b/src/contextseek/cli/doctor_cmd.py @@ -0,0 +1,581 @@ +"""``contextseek doctor`` — config & connectivity self-check. + +Loads ``ContextSeekSettings``, reports which backend/embedder/LLM are resolved, +runs a lightweight liveness check on each (e.g. a tiny embedding call, a storage +ping), and prints clear PASS/FAIL/SKIP/WARN lines with actionable hints pointing +at ``.env.example``. + +Never prints secrets. Exits non-zero if any required component fails. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from typing import Any + +from contextseek.config.factory import ( + build_embedder, + build_llm, + resolve_embedding_dims, +) +from contextseek.config.settings import ContextSeekSettings + + +# --------------------------------------------------------------------------- +# Data structures +# --------------------------------------------------------------------------- + +PASS = "PASS" +FAIL = "FAIL" +SKIP = "SKIP" +WARN = "WARN" + + +@dataclass +class CheckResult: + """Outcome of a single doctor check.""" + + status: str + component: str + message: str + hint: str = "" + env_section: str = "" + + +# --------------------------------------------------------------------------- +# Error message sanitisation — never leak secrets +# --------------------------------------------------------------------------- + +_SECRET_PATTERNS: list[tuple[re.Pattern[str], str]] = [ + (re.compile(r"sk-[A-Za-z0-9\-_]{10,}"), "sk-***"), + (re.compile(r"(?i)(password\s*=\s*)\S+"), r"\1***"), + (re.compile(r"(?i)(api[_-]?key\s*=\s*)\S+"), r"\1***"), + (re.compile(r"(?i)(token\s*=\s*)\S+"), r"\1***"), +] + + +def _sanitize_error_message(msg: str, *, max_len: int = 200) -> str: + """Remove secrets from an error message and cap its length.""" + sanitized = msg + for pattern, replacement in _SECRET_PATTERNS: + sanitized = pattern.sub(replacement, sanitized) + if len(sanitized) > max_len: + sanitized = sanitized[:max_len] + "…" + return sanitized + + +# --------------------------------------------------------------------------- +# Error classification +# --------------------------------------------------------------------------- + +def _classify_exception(exc: Exception, component: str) -> tuple[str, str, str]: + """Return (message, hint, env_section) for a caught exception. + + The message is sanitised before being returned. + """ + raw = str(exc) + exc_type = type(exc).__name__ + msg = _sanitize_error_message(raw) + + # Build a short type prefix for the message + short = f"{exc_type}: {msg}" if msg else exc_type + + # --- Import / missing dependency --- + if isinstance(exc, ImportError): + if "pyseekdb" in raw.lower(): + return ( + short, + "Install pyseekdb: pip install pyseekdb", + "section 1.0", + ) + if "pyobvector" in raw.lower() or "sqlalchemy" in raw.lower(): + return ( + short, + "Install OceanBase deps: pip install 'contextseek[oceanbase]'", + "section 1.1", + ) + if "langchain" in raw.lower(): + pkg_hint = "pip install 'contextseek[langchain]'" + if "openai" in raw.lower(): + pkg_hint += " && pip install 'contextseek[openai]'" + elif "dashscope" in raw.lower(): + pkg_hint += " && pip install 'contextseek[dashscope]'" + elif "ollama" in raw.lower(): + pkg_hint += " && pip install 'contextseek[ollama]'" + elif "huggingface" in raw.lower(): + pkg_hint += " && pip install 'contextseek[huggingface]'" + return ( + short, + f"Missing LangChain package: {pkg_hint}", + f"section {'2' if component == 'embedding' else '3'}", + ) + return (short, "Install the missing dependency", f"section {'2' if component == 'embedding' else '3'}") + + # --- Authentication --- + raw_lower = raw.lower() + if ( + "auth" in raw_lower + or "api key" in raw_lower + or "api_key" in raw_lower + or "unauthorized" in raw_lower + or "401" in raw_lower + or "forbidden" in raw_lower + or "403" in raw_lower + ): + if component == "embedding": + return (short, "Set the correct API key (e.g. OPENAI_API_KEY) in .env", "section 2") + return (short, "Set the correct API key (e.g. OPENAI_API_KEY) in .env", "section 3") + + # --- Connection / network --- + if ( + isinstance(exc, (ConnectionError, TimeoutError, OSError)) + or "connection refused" in raw_lower + or "timed out" in raw_lower + or "timeout" in raw_lower + or "unreachable" in raw_lower + or "network" in raw_lower + ): + if component == "storage": + return (short, "Check that the storage service is running and host/port are correct", "section 1 / 1.0 / 1.1") + if component == "embedding": + return (short, "Check EMBEDDING_BASE_URL and network connectivity", "section 2") + return (short, "Check LLM_BASE_URL and network connectivity", "section 3") + + # --- Configuration / value errors --- + if isinstance(exc, ValueError): + if "dims" in raw_lower or "embedding_dims" in raw_lower: + return (short, "Set EMBEDDING_DIMS when using oceanbase backend", "section 1.1 / 2") + if "unknown" in raw_lower and "provider" in raw_lower: + if component == "embedding": + return (short, "Check EMBEDDING_PROVIDER spelling; see supported providers", "section 2") + return (short, "Check LLM_PROVIDER spelling; see supported providers", "section 3") + return (short, "Check configuration values in .env", "section 1 / 2 / 3") + + # --- Generic fallback --- + return (short, "See .env.example for configuration reference", "section 1 / 2 / 3") + + +# --------------------------------------------------------------------------- +# Individual checks +# --------------------------------------------------------------------------- + +def _check_storage(settings: ContextSeekSettings) -> CheckResult: + """Build and initialise the storage backend to verify connectivity.""" + storage = settings.storage + backend_name = storage.backend + + # --- oceanbase requires EMBEDDING_DIMS --- + if backend_name == "oceanbase": + vector_dims = settings.embedding.dims + if not vector_dims: + return CheckResult( + FAIL, + "storage", + "EMBEDDING_DIMS must be set when STORAGE_BACKEND=oceanbase", + "Set EMBEDDING_DIMS (e.g. 1536 for OpenAI text-embedding-3-small)", + "section 1.1 / 2", + ) + + # --- Construct the backend (mirrors client/contextseek.py:261-318) --- + backend: Any + try: + if backend_name == "oceanbase": + from contextseek.storage.ob_backend import OceanBaseBackend + + ob = settings.ob + geo = getattr(settings, "geo", None) + if geo is not None and getattr(geo, "enabled", False): + from contextseek.storage.ob_geo_backend import OceanBaseGeoBackend + + backend = OceanBaseGeoBackend( + table_name=ob.table_name, + vector_dims=vector_dims, + host=ob.host, + port=ob.port, + user=ob.user, + password=ob.password, + db_name=ob.db_name, + geo_table_name=geo.geo_table_name, + distance_decay_km=geo.distance_decay_km, + route_sample_interval_km=geo.route_sample_interval_km, + ) + else: + backend = OceanBaseBackend( + table_name=ob.table_name, + vector_dims=vector_dims, + host=ob.host, + port=ob.port, + user=ob.user, + password=ob.password, + db_name=ob.db_name, + ) + elif backend_name == "sqlite": + from contextseek.storage.sqlite_backend import SQLiteBackend + + backend = SQLiteBackend(path=settings.sqlite.path) + elif backend_name == "seekdb": + from contextseek.storage.seekdb_backend import SeekDBBackend + + seekdb = settings.seekdb + backend = SeekDBBackend( + path=seekdb.path, + database=seekdb.database, + host=seekdb.host, + port=seekdb.port, + ) + elif backend_name == "file": + from contextseek.storage.file_backend import FileBackend + + backend = FileBackend(root_dir=storage.path) + elif backend_name == "memory": + from contextseek.storage.in_memory_backend import InMemoryBackend + + backend = InMemoryBackend() + else: + return CheckResult( + FAIL, + "storage", + f"Unknown storage backend: {backend_name!r}", + "Set STORAGE_BACKEND to one of: memory, file, sqlite, seekdb, oceanbase", + "section 1", + ) + except Exception as exc: + msg, hint, section = _classify_exception(exc, "storage") + return CheckResult(FAIL, "storage", msg, hint, section) + + # --- Initialise (this is the actual connectivity check) --- + try: + backend.initialize() + except Exception as exc: + msg, hint, section = _classify_exception(exc, "storage") + return CheckResult(FAIL, "storage", msg, hint, section) + + # --- Clean up --- + close = getattr(backend, "close", None) + if callable(close): + try: + close() + except Exception: + pass + + return CheckResult(PASS, "storage", f"{backend_name} backend initialized") + + +def _check_embedding(settings: ContextSeekSettings) -> CheckResult: + """Build embedder and do a tiny probe call.""" + emb_settings = settings.embedding + provider = emb_settings.provider.strip().lower() if emb_settings.provider else "none" + + if provider in {"", "none"}: + return CheckResult( + SKIP, + "embedding", + "provider=none — vector search disabled", + "Set EMBEDDING_PROVIDER to enable vector retrieval", + "section 2", + ) + + # langchain provider requires class_path + if provider == "langchain" and not emb_settings.class_path: + return CheckResult( + SKIP, + "embedding", + "provider=langchain but EMBEDDING_CLASS_PATH not set", + "Set EMBEDDING_CLASS_PATH to your custom LangChain embeddings class", + "section 2", + ) + + # Build the embedder + try: + embedder = build_embedder(emb_settings) + except Exception as exc: + msg, hint, section = _classify_exception(exc, "embedding") + return CheckResult(FAIL, "embedding", msg, hint, section) + + if embedder is None: + return CheckResult( + SKIP, + "embedding", + f"provider={provider} resolved to None — vector search disabled", + "Check EMBEDDING_PROVIDER / EMBEDDING_CLASS_PATH configuration", + "section 2", + ) + + # Probe call + try: + vec = embedder("test") + except Exception as exc: + msg, hint, section = _classify_exception(exc, "embedding") + return CheckResult(FAIL, "embedding", msg, hint, section) + + if not vec or not isinstance(vec, list): + return CheckResult( + FAIL, + "embedding", + f"embedder returned empty/invalid result for probe: {type(vec).__name__}", + "Check EMBEDDING_MODEL and provider configuration", + "section 2", + ) + + actual_dims = len(vec) + configured_dims = resolve_embedding_dims(emb_settings) + if configured_dims and configured_dims != actual_dims: + return CheckResult( + WARN, + "embedding", + f"embedder returned {actual_dims}-dim vector (configured dims={configured_dims})", + f"Update EMBEDDING_DIMS to {actual_dims} or verify EMBEDDING_MODEL", + "section 2", + ) + + return CheckResult( + PASS, + "embedding", + f"embedder returned {actual_dims}-dim vector for probe \"test\"", + ) + + +def _check_llm(settings: ContextSeekSettings) -> CheckResult: + """Build LLM and do a tiny invoke call.""" + llm_settings = settings.llm + provider = llm_settings.provider.strip().lower() if llm_settings.provider else "none" + + if provider in {"", "none"}: + return CheckResult( + SKIP, + "llm", + "provider=none — rerank/summarize/evolution LLM disabled", + "Set LLM_PROVIDER to enable LLM-powered features", + "section 3", + ) + + # langchain provider requires class_path + if provider == "langchain" and not llm_settings.class_path: + return CheckResult( + SKIP, + "llm", + "provider=langchain but LLM_CLASS_PATH not set", + "Set LLM_CLASS_PATH to your custom LangChain chat model class", + "section 3", + ) + + # Build the LLM + try: + llm = build_llm(llm_settings) + except Exception as exc: + msg, hint, section = _classify_exception(exc, "llm") + return CheckResult(FAIL, "llm", msg, hint, section) + + if llm is None: + return CheckResult( + SKIP, + "llm", + f"provider={provider} resolved to None — LLM disabled", + "Check LLM_PROVIDER / LLM_CLASS_PATH configuration", + "section 3", + ) + + # Probe call — directly invoke to get precise error reporting + try: + from langchain_core.messages import HumanMessage + + resp = llm.invoke([HumanMessage(content="hello")]) + except Exception as exc: + msg, hint, section = _classify_exception(exc, "llm") + return CheckResult(FAIL, "llm", msg, hint, section) + + if resp is None: + return CheckResult( + FAIL, + "llm", + "LLM invoke returned None", + "Check LLM_MODEL and provider configuration", + "section 3", + ) + + return CheckResult(PASS, "llm", f"LLM responded to probe (provider={provider})") + + +def _check_cross( + settings: ContextSeekSettings, + results: list[CheckResult], +) -> list[CheckResult]: + """Cross-dependency checks that produce WARN (never FAIL).""" + warnings: list[CheckResult] = [] + + backend_name = settings.storage.backend + emb_provider = settings.embedding.provider.strip().lower() if settings.embedding.provider else "none" + llm_provider = settings.llm.provider.strip().lower() if settings.llm.provider else "none" + + # seekdb + no external embedder + if backend_name == "seekdb" and emb_provider in {"", "none"}: + warnings.append( + CheckResult( + WARN, + "cross", + "seekdb backend with no external embedder — using built-in 384-dim embedding", + "Set EMBEDDING_PROVIDER for higher-quality vectors", + "section 2", + ) + ) + + # LLM configured but embedding not — retrieval will lack vector route + if llm_provider not in {"", "none"} and emb_provider in {"", "none"}: + warnings.append( + CheckResult( + WARN, + "cross", + "LLM configured but embedding is not — retrieval recall routes will not include vector search", + "Set EMBEDDING_PROVIDER to enable vector retrieval", + "section 2", + ) + ) + + return warnings + + +# --------------------------------------------------------------------------- +# Configuration report (no side effects) +# --------------------------------------------------------------------------- + +def _describe_storage(settings: ContextSeekSettings) -> str: + """Human-readable description of the resolved storage config (no secrets).""" + s = settings.storage + backend = s.backend + if backend == "sqlite": + return f"sqlite (path={settings.sqlite.path})" + if backend == "seekdb": + seekdb = settings.seekdb + if seekdb.host: + return f"seekdb (host={seekdb.host}:{seekdb.port}, database={seekdb.database})" + return f"seekdb embedded (path={seekdb.path}, database={seekdb.database})" + if backend == "oceanbase": + ob = settings.ob + return f"oceanbase (host={ob.host}:{ob.port}, user={ob.user}, db={ob.db_name}, table={ob.table_name})" + if backend == "file": + return f"file (path={s.path})" + if backend == "memory": + return "memory" + return f"{backend} (unknown)" + + +def _describe_embedding(settings: ContextSeekSettings) -> str: + """Human-readable description of the resolved embedding config (no secrets).""" + emb = settings.embedding + provider = emb.provider or "none" + if provider in {"", "none"}: + return "none" + parts = [f"provider={provider}"] + if emb.model: + parts.append(f"model={emb.model}") + dims = resolve_embedding_dims(emb) + if dims: + parts.append(f"dims={dims}") + if emb.base_url: + parts.append(f"base_url={emb.base_url}") + return ", ".join(parts) + + +def _describe_llm(settings: ContextSeekSettings) -> str: + """Human-readable description of the resolved LLM config (no secrets).""" + llm = settings.llm + provider = llm.provider or "none" + if provider in {"", "none"}: + return "none" + parts = [f"provider={provider}"] + if llm.model: + parts.append(f"model={llm.model}") + if llm.base_url: + parts.append(f"base_url={llm.base_url}") + return ", ".join(parts) + + +# --------------------------------------------------------------------------- +# Output rendering +# --------------------------------------------------------------------------- + +_STATUS_LABELS = { + PASS: "[PASS]", + FAIL: "[FAIL]", + SKIP: "[SKIP]", + WARN: "[WARN]", +} + + +def _print(text: str = "") -> None: + print(text) + + +def _render_results(results: list[CheckResult]) -> None: + """Print check results in human-readable format.""" + _print() + _print("Checks:") + for r in results: + label = _STATUS_LABELS.get(r.status, f"[{r.status}]") + _print(f" {label} {r.component:<10} {r.message}") + if r.hint: + section = f"; see .env.example ({r.env_section})" if r.env_section else "" + indent = " ↳ fix: " if r.status == FAIL else " ↳ hint: " + _print(f"{indent}{r.hint}{section}") + _print() + + +def _render_summary(results: list[CheckResult]) -> int: + """Print summary line and return exit code.""" + counts = {PASS: 0, FAIL: 0, SKIP: 0, WARN: 0} + for r in results: + counts[r.status] = counts.get(r.status, 0) + 1 + + parts = [] + if counts[PASS]: + parts.append(f"{counts[PASS]} PASS") + if counts[FAIL]: + parts.append(f"{counts[FAIL]} FAIL") + if counts[SKIP]: + parts.append(f"{counts[SKIP]} SKIP") + if counts[WARN]: + parts.append(f"{counts[WARN]} WARN") + + exit_code = 1 if counts[FAIL] > 0 else 0 + suffix = f" — exit {exit_code}" if exit_code else "" + _print(f"Result: {', '.join(parts) if parts else 'no checks'}{suffix}") + return exit_code + + +# --------------------------------------------------------------------------- +# Main entry point +# --------------------------------------------------------------------------- + +def run_doctor(settings: ContextSeekSettings) -> int: + """Run the doctor diagnostics and return an exit code. + + Exit code is 1 if any check FAILs, 0 otherwise. + """ + # --- Configuration report (no side effects) --- + _print() + _print("ContextSeek doctor — configuration diagnostics") + _print() + _print("Configuration resolved:") + _print(f" storage : {_describe_storage(settings)}") + _print(f" embedding : {_describe_embedding(settings)}") + _print(f" llm : {_describe_llm(settings)}") + + # --- Run checks --- + results: list[CheckResult] = [] + + # Storage check — wrap in suppress_backend_noise to avoid pyseekdb etc. + from contextseek.cli.ui import suppress_backend_noise + + with suppress_backend_noise(): + results.append(_check_storage(settings)) + + results.append(_check_embedding(settings)) + results.append(_check_llm(settings)) + + # Cross-dependency warnings + results.extend(_check_cross(settings, results)) + + # --- Render --- + _render_results(results) + return _render_summary(results) \ No newline at end of file diff --git a/src/contextseek/cli/main.py b/src/contextseek/cli/main.py index 340f74f..45a611d 100644 --- a/src/contextseek/cli/main.py +++ b/src/contextseek/cli/main.py @@ -97,7 +97,13 @@ def build_parser() -> argparse.ArgumentParser: retrieve_parser.add_argument( "--tags", default="", - help="comma-separated tag filter; returned items must contain all tags", + help="comma-separated tag filter (use --tag-match to control all/any semantics)", + ) + retrieve_parser.add_argument( + "--tag-match", + choices=["all", "any"], + default="all", + help="tag match mode: 'all' requires every tag, 'any' requires at least one (default: all)", ) retrieve_parser.add_argument( "--trace", @@ -311,6 +317,12 @@ def build_parser() -> argparse.ArgumentParser: help="initialize ~/.contextseek/: generate config.env, mcp.json, register system service", ) + # doctor — config & connectivity self-check + subparsers.add_parser( + "doctor", + help="diagnose storage / embedding / LLM configuration and connectivity", + ) + # daemon daemon_parser = subparsers.add_parser("daemon", help="manage the background daemon") daemon_sub = daemon_parser.add_subparsers(dest="daemon_command", required=True) @@ -341,6 +353,19 @@ def build_parser() -> argparse.ArgumentParser: help="detect format and count items without writing", ) + # sync-obsidian + obsidian_parser = subparsers.add_parser( + "sync-obsidian", + help="incremental sync of an Obsidian vault (wikilinks → Link edges)", + ) + obsidian_parser.add_argument("path", help="path to Obsidian vault root") + obsidian_parser.add_argument("--scope", default=None) + obsidian_parser.add_argument( + "--dry-run", + action="store_true", + help="detect changed files without writing", + ) + plug_replay_parser = subparsers.add_parser( "plug-outbox-replay", help="replay pending/failed PlugGateway outbox events" ) @@ -500,6 +525,11 @@ def run_cli( run_init(pathlib.Path.home() / ".contextseek") return 0 + if args.command == "doctor": + from contextseek.cli.doctor_cmd import run_doctor + + return run_doctor(settings) + if args.command == "daemon" and args.daemon_command in {"stop", "status"}: from contextseek.daemon.process import DaemonProcess import pathlib @@ -594,6 +624,7 @@ def run_cli( k=args.k, full=args.full, tags=tags or None, + tag_match=args.tag_match, with_trace=args.trace, ) output = { @@ -1188,6 +1219,60 @@ def run_cli( print_success(f"done: added {report.added} skipped {report.skipped}") return 0 + if args.command == "sync-obsidian": + from contextseek.plugs.obsidian import ObsidianVaultPlug + import pathlib as _opl + + _vault = _opl.Path(args.path).expanduser() + _obs_scope = args.scope or "obsidian/vault" + subtitle = "dry-run" if args.dry_run else None + print_panel( + "ContextSeek sync-obsidian", + f"vault : {_vault}\nscope : {_obs_scope}", + subtitle=subtitle, + ) + + plug = ObsidianVaultPlug(vault_path=_vault, scope=_obs_scope) + + if args.dry_run: + # Consume stream to count changes without writing + events = list(plug.stream()) + print_success( + f"dry-run: would import {plug.added} items " + f"({plug.skipped} unchanged, " + f"{len(plug.vanished_files)} deleted)" + ) + return 0 + + # Consume the plug's stream via ctx.plug() + # We intercept to record item IDs for the sync state + events_to_process = list(plug.stream()) + for event in events_to_process: + item = ctx.add( + event.content, + scope=_obs_scope, + source=event.source, + source_type="document", + tags=event.tags, + check_conflicts=False, + ) + plug.record_item_id(event.source, item.id) + + # Persist sync state + plug.persist_state() + + # Resolve wikilinks → Link edges + links = plug.resolve_links(ctx) + + # Handle deletes (soft-delete vanished files) + deletes = plug.handle_deletes(ctx) + + print_success( + f"done: added {plug.added} skipped {plug.skipped} " + f"deleted {deletes} links {links}" + ) + return 0 + return 1 diff --git a/src/contextseek/client/contextseek.py b/src/contextseek/client/contextseek.py index 774a988..bcfad77 100644 --- a/src/contextseek/client/contextseek.py +++ b/src/contextseek/client/contextseek.py @@ -18,7 +18,7 @@ from contextvars import ContextVar from dataclasses import dataclass, field, replace from datetime import datetime, timezone -from typing import TYPE_CHECKING, Any, Callable, Iterator +from typing import TYPE_CHECKING, Any, Callable, Iterator, Literal from uuid import NAMESPACE_URL, uuid4, uuid5 if TYPE_CHECKING: @@ -565,6 +565,7 @@ def retrieve( full: bool = False, stage: Stage | None = None, tags: list[str] | None = None, + tag_match: Literal["all", "any"] = "all", filters: dict[str, Any] | None = None, include_deleted: bool = False, include_expired: bool = False, @@ -586,8 +587,11 @@ def retrieve( (default) L1 summaries replace content to save tokens—call :meth:`expand` to upgrade to full text. stage: Optional stage filter. - tags: Optional tag filter (all tags must match). - filters: Compatibility bag; may include ``stage`` / ``tags`` / ``min_confidence``. + tags: Optional tag filter; ``tag_match`` controls matching semantics. + tag_match: Tag match mode — ``"all"`` (default) requires the item to + have *every* tag in ``tags``; ``"any"`` requires at least one. + filters: Compatibility bag; may include ``stage`` / ``tags`` / + ``tag_match`` / ``min_confidence``. include_deleted: Whether soft-deleted items are visible. include_expired: Whether items whose bi-temporal validity window has closed (``valid_to`` in the past) are visible. Default False, so @@ -599,12 +603,15 @@ def retrieve( """ stage_filter = stage tag_filter = tags + tag_match_filter = tag_match min_conf = None if filters: if filters.get("stage"): stage_filter = Stage(filters["stage"]) if not tag_filter: tag_filter = filters.get("tags") + if tag_match_filter == "all" and filters.get("tag_match"): + tag_match_filter = filters["tag_match"] min_conf = filters.get("min_confidence") prefix = self.resolver.prefix_for(scope) @@ -633,6 +640,7 @@ def retrieve( k=k, stage=stage_filter, tags=tag_filter, + tag_match=tag_match_filter, include_deleted=include_deleted, include_expired=include_expired, geo_query=geo_query, @@ -648,6 +656,7 @@ def retrieve( k=k, stage=stage_filter, tags=tag_filter, + tag_match=tag_match_filter, include_deleted=include_deleted, include_expired=include_expired, geo_query=geo_query, diff --git a/src/contextseek/domain/tools.py b/src/contextseek/domain/tools.py index 5e0b9ba..cfaa06a 100644 --- a/src/contextseek/domain/tools.py +++ b/src/contextseek/domain/tools.py @@ -77,6 +77,18 @@ def to_anthropic(self) -> dict[str, Any]: "description": "If true, return L0 full content instead of L1 summaries.", "default": False, }, + "tags": { + "type": "array", + "items": {"type": "string"}, + "description": "Optional tag filter list.", + "default": [], + }, + "tag_match": { + "type": "string", + "enum": ["all", "any"], + "description": "Tag match mode: 'all' requires every tag, 'any' requires at least one.", + "default": "all", + }, }, "required": ["query", "scope"], }, diff --git a/src/contextseek/mcp/server.py b/src/contextseek/mcp/server.py index 388c869..f0c0562 100644 --- a/src/contextseek/mcp/server.py +++ b/src/contextseek/mcp/server.py @@ -174,6 +174,16 @@ def _meta_tools(self) -> list[dict[str, Any]]: "query": {"type": "string", "required": True}, "k": {"type": "integer", "default": 10}, "full": {"type": "boolean", "default": False}, + "tags": { + "type": "array", + "items": {"type": "string"}, + "default": [], + }, + "tag_match": { + "type": "string", + "enum": ["all", "any"], + "default": "all", + }, "include_expired": {"type": "boolean", "default": False}, "include_trace": {"type": "boolean", "default": False}, }, @@ -329,6 +339,8 @@ def call_tool(self, name: str, arguments: dict[str, Any]) -> dict[str, Any]: scope=arguments["scope"], k=arguments.get("k", 10), full=bool(arguments.get("full", False)), + tags=arguments.get("tags") or None, + tag_match=arguments.get("tag_match", "all"), include_expired=bool(arguments.get("include_expired", False)), with_trace=bool(arguments.get("include_trace", False)), ) diff --git a/src/contextseek/plugs/__init__.py b/src/contextseek/plugs/__init__.py index bb8c2bc..47332ad 100644 --- a/src/contextseek/plugs/__init__.py +++ b/src/contextseek/plugs/__init__.py @@ -3,6 +3,7 @@ __all__ = [ "HermesSkillImporter", "MCPToolImporter", + "ObsidianVaultPlug", "OpenAIFunctionImporter", "PowerMemPlug", "PowerMemProxyPlug", @@ -23,6 +24,10 @@ def __getattr__(name: str): from contextseek.plugs.rag import RAGPlug return RAGPlug + if name == "ObsidianVaultPlug": + from contextseek.plugs.obsidian import ObsidianVaultPlug + + return ObsidianVaultPlug if name in {"HermesSkillImporter", "MCPToolImporter", "OpenAIFunctionImporter"}: from contextseek.plugs.skills import ( HermesSkillImporter, diff --git a/src/contextseek/plugs/obsidian/__init__.py b/src/contextseek/plugs/obsidian/__init__.py new file mode 100644 index 0000000..f87f3f4 --- /dev/null +++ b/src/contextseek/plugs/obsidian/__init__.py @@ -0,0 +1,13 @@ +"""Incremental Obsidian vault DataPlug.""" + +from contextseek.plugs.obsidian.plug import ObsidianVaultPlug +from contextseek.plugs.obsidian.sync_state import FileSyncState, SyncStateStore +from contextseek.plugs.obsidian.wikilinks import extract_wikilinks, resolve_wikilink_targets + +__all__ = [ + "ObsidianVaultPlug", + "FileSyncState", + "SyncStateStore", + "extract_wikilinks", + "resolve_wikilink_targets", +] \ No newline at end of file diff --git a/src/contextseek/plugs/obsidian/plug.py b/src/contextseek/plugs/obsidian/plug.py new file mode 100644 index 0000000..d6e2491 --- /dev/null +++ b/src/contextseek/plugs/obsidian/plug.py @@ -0,0 +1,353 @@ +"""Incremental Obsidian vault DataPlug. + +Syncs an Obsidian vault (Markdown + wikilinks) into ContextSeek with +three key capabilities: + +1. **Incremental sync** — tracks mtime + content hash per file so + unchanged files are skipped on re-sync. +2. **Wikilink → Link** — resolves ``[[wikilinks]]`` to typed + ``Link(relation=LinkType.related_to)`` edges between ContextItems. +3. **Delete handling** — soft-deletes ContextItems whose source files + have vanished from the vault. + +Usage:: + + from contextseek.plugs import ObsidianVaultPlug + + plug = ObsidianVaultPlug(vault_path=Path("/path/to/vault"), scope="obsidian/vault") + ctx.plug(plug, scope="obsidian/vault") + links = plug.resolve_links(ctx) + deletes = plug.handle_deletes(ctx) +""" + +from __future__ import annotations + +import hashlib +import pathlib +import re +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Iterator + +from contextseek.domain.links import Link, LinkType +from contextseek.plugs.core.protocols import PlugMeta, RawEvent +from contextseek.plugs.obsidian.sync_state import FileSyncState, SyncStateStore +from contextseek.plugs.obsidian.wikilinks import resolve_wikilink_targets + +if TYPE_CHECKING: + from contextseek.client.contextseek import ContextSeek + +# Directories to skip when scanning the vault. +_IGNORED_DIR_NAMES: set[str] = { + ".obsidian", + ".contextseek", + ".git", + ".trash", + "__pycache__", + "node_modules", +} + +# YAML front-matter pattern: --- \n key: value \n --- +_FRONTMATTER_RE = re.compile(r"\A---\s*\n(.*?)\n---\s*\n?", re.DOTALL) +# Tags line in front-matter: tags: [a, b] or tags:\n - a\n - b +_FRONTMATTER_TAGS_RE = re.compile( + r"^tags:\s*(?:\[([^\]]*)\]|(.+))$", re.MULTILINE +) + + +def _file_hash(p: pathlib.Path) -> str: + """Streamed SHA256 of a file's bytes.""" + h = hashlib.sha256() + with p.open("rb") as fh: + for chunk in iter(lambda: fh.read(65536), b""): + h.update(chunk) + return h.hexdigest() + + +def _extract_frontmatter_tags(text: str) -> list[str]: + """Extract tags from YAML front-matter (if present).""" + fm_match = _FRONTMATTER_RE.match(text) + if not fm_match: + return [] + fm_text = fm_match.group(1) + + # List-style: tags:\n - a\n - b + list_match = re.search(r"^tags:\s*\n((?:\s*-\s+.+\n?)+)", fm_text, re.MULTILINE) + if list_match: + items = re.findall(r"^\s*-\s+(.+)$", list_match.group(1), re.MULTILINE) + return [item.strip().strip('"').strip("'") for item in items if item.strip()] + + # Bracket-style: tags: [a, b, c] + bracket_match = _FRONTMATTER_TAGS_RE.search(fm_text) + if bracket_match: + bracket_content = bracket_match.group(1) + if bracket_content is not None: + return [ + t.strip().strip('"').strip("'") + for t in bracket_content.split(",") + if t.strip() + ] + # Inline-style: tags: a b c (space-separated) + inline = bracket_match.group(2) + if inline and not inline.startswith("-"): + return [t.strip() for t in inline.split() if t.strip()] + return [] + + +def _strip_frontmatter(text: str) -> str: + """Remove YAML front-matter from markdown text.""" + return _FRONTMATTER_RE.sub("", text, count=1) + + +def _iter_markdown_files(vault_path: pathlib.Path) -> list[pathlib.Path]: + """Return all .md files under vault, skipping config/cache directories.""" + import os + + files: list[pathlib.Path] = [] + for dirpath, dirnames, filenames in os.walk(vault_path): + dirnames[:] = [ + name for name in dirnames if name not in _IGNORED_DIR_NAMES + ] + base = pathlib.Path(dirpath) + for filename in filenames: + fp = base / filename + if fp.suffix.lower() == ".md": + files.append(fp) + return sorted(files) + + +@dataclass +class ObsidianVaultPlug: + """DataPlug for incremental Obsidian vault sync. + + Attributes: + vault_path: Path to the Obsidian vault root. + scope: Destination scope for imported items. + source_name: Plug name for PlugMeta. + """ + + vault_path: pathlib.Path + scope: str = "obsidian/vault" + source_name: str = "obsidian_vault" + + # Runtime state populated during stream() + _added: int = field(default=0, init=False, repr=False) + _skipped: int = field(default=0, init=False, repr=False) + _vanished_files: list[str] = field(default_factory=list, init=False, repr=False) + _state_store: SyncStateStore = field(init=False, repr=False) + _new_states: dict[str, FileSyncState] = field( + default_factory=dict, init=False, repr=False + ) + _stream_meta: dict[str, tuple[str, float]] = field( + default_factory=dict, init=False, repr=False + ) + _old_states: dict[str, FileSyncState] = field( + default_factory=dict, init=False, repr=False + ) + + def __post_init__(self) -> None: + self.vault_path = pathlib.Path(self.vault_path) + state_path = self.vault_path / ".contextseek" / "sync_state.json" + self._state_store = SyncStateStore(state_path) + + # ------------------------------------------------------------------ + # DataPlug protocol + # ------------------------------------------------------------------ + + def stream(self) -> Iterator[RawEvent]: + """Yield RawEvents for new/modified .md files (incremental).""" + old_states = self._state_store.load() + self._old_states = dict(old_states) + current_files = _iter_markdown_files(self.vault_path) + current_paths = {f.as_posix() for f in current_files} + + self._vanished_files = sorted(set(old_states.keys()) - current_paths) + self._added = 0 + self._skipped = 0 + + for fp in current_files: + key = fp.as_posix() + try: + mtime = fp.stat().st_mtime + except OSError: + continue + + old = old_states.get(key) + + # mtime fast-path + if old is not None and abs(old.mtime - mtime) < 1e-6: + self._skipped += 1 + self._new_states[key] = old + continue + + # SHA256 authoritative check + fhash = _file_hash(fp) + if old is not None and old.content_hash == fhash: + # mtime changed but content didn't — refresh mtime, skip + updated = FileSyncState( + mtime=mtime, content_hash=fhash, context_item_id=old.context_item_id + ) + self._new_states[key] = updated + self._skipped += 1 + continue + + # Read and parse the file + try: + raw_text = fp.read_text(encoding="utf-8", errors="replace") + except OSError: + continue + + tags = _extract_frontmatter_tags(raw_text) + tags = list(tags) + ["obsidian"] + + yield RawEvent( + content=raw_text, + source=key, + tags=tags, + metadata={"_file_stem": fp.stem, "_content_hash": fhash, "_mtime": mtime}, + ) + self._stream_meta[key] = (fhash, mtime) + self._added += 1 + + def metadata(self) -> PlugMeta: + """Return plug metadata.""" + return PlugMeta( + name=self.source_name, + source_type="document", + description="Incremental Obsidian vault sync (Markdown + wikilinks)", + ) + + # ------------------------------------------------------------------ + # Post-sync operations (called after ctx.plug()) + # ------------------------------------------------------------------ + + def record_item_id(self, file_path: str, item_id: str) -> None: + """Record the ContextItem ID assigned to a synced file. + + Called by the CLI handler after each ``ctx.add()`` succeeds so + that ``resolve_links()`` and ``handle_deletes()`` can use it. + """ + # Check if we already have stream metadata for this file + # (from the RawEvent we yielded in stream()) + cached = self._stream_meta.get(file_path) + if cached is not None: + self._new_states[file_path] = FileSyncState( + mtime=cached[1], content_hash=cached[0], context_item_id=item_id + ) + return + + # Fallback: compute from disk + p = pathlib.Path(file_path) + try: + mtime = p.stat().st_mtime + fhash = _file_hash(p) + except OSError: + mtime = 0.0 + fhash = "" + self._new_states[file_path] = FileSyncState( + mtime=mtime, content_hash=fhash, context_item_id=item_id + ) + + def persist_state(self) -> None: + """Save the sync cursor to disk. Call after sync completes.""" + self._state_store.save(self._new_states) + + def resolve_links(self, ctx: "ContextSeek") -> int: + """Post-sync pass: resolve ``[[wikilinks]]`` to typed Link edges. + + Builds a ``{file_stem: item_id}`` mapping from the sync state, + then walks every imported item, extracts wikilinks from its + content, and appends ``Link(relation=LinkType.related_to)`` + edges to matching targets. + + Returns the number of links created. + """ + states = self._new_states or self._state_store.load() + if not states: + return 0 + + # Build stem → item_id mapping + stem_to_id: dict[str, str] = {} + for file_path, state in states.items(): + stem = pathlib.Path(file_path).stem + stem_to_id[stem] = state.context_item_id + + links_created = 0 + for file_path, state in states.items(): + item_id = state.context_item_id + ref = ctx.resolver.ref_for(self.scope, item_id) + payload = ctx.adapter.read(ref) + if payload is None: + continue + + from contextseek.domain.serialization import deserialize_context_item + + item = deserialize_context_item(payload) + content = item.content_text + + target_ids = resolve_wikilink_targets(content, stem_to_id) + existing_targets = {link.target_id for link in item.links} + + changed = False + for target_id in target_ids: + if target_id == item_id or target_id in existing_targets: + continue + item.links.append( + Link( + target_id=target_id, + relation=LinkType.related_to, + strength=0.8, + ) + ) + links_created += 1 + changed = True + + if changed: + ctx._write_item(item) + + return links_created + + def handle_deletes(self, ctx: "ContextSeek") -> int: + """Soft-delete ContextItems whose source files vanished from the vault. + + Returns the number of items soft-deleted. + """ + if not self._vanished_files: + return 0 + + # Use old_states captured during stream() — not the freshly + # saved state (which no longer contains the vanished files). + old_states = self._old_states or self._state_store.load() + deleted = 0 + for file_path in self._vanished_files: + old = old_states.get(file_path) + if not old or not old.context_item_id: + continue + ref = ctx.resolver.ref_for(self.scope, old.context_item_id) + try: + ctx.forget( + ref, + scope=self.scope, + reason=f"source file deleted: {file_path}", + propagate=False, + ) + deleted += 1 + except ValueError: + # Already deleted or not found — skip + pass + return deleted + + # ------------------------------------------------------------------ + # Convenience properties + # ------------------------------------------------------------------ + + @property + def added(self) -> int: + return self._added + + @property + def skipped(self) -> int: + return self._skipped + + @property + def vanished_files(self) -> list[str]: + return self._vanished_files \ No newline at end of file diff --git a/src/contextseek/plugs/obsidian/sync_state.py b/src/contextseek/plugs/obsidian/sync_state.py new file mode 100644 index 0000000..0f8a050 --- /dev/null +++ b/src/contextseek/plugs/obsidian/sync_state.py @@ -0,0 +1,69 @@ +"""JSON-backed per-file sync cursor for incremental Obsidian vault sync. + +Records ``mtime``, ``content_hash`` and ``context_item_id`` for every +``.md`` file so the next sync can short-circuit via mtime and detect +vanished files for soft-delete. +""" + +from __future__ import annotations + +import json +import pathlib +from dataclasses import asdict, dataclass + + +@dataclass +class FileSyncState: + """Per-file sync state — the sync cursor's atomic unit.""" + + mtime: float + content_hash: str + context_item_id: str + + +class SyncStateStore: + """JSON file-backed store for ``{file_path: FileSyncState}`` mappings. + + The state file lives at ``{vault}/.contextseek/sync_state.json`` and is + created lazily on first ``save()``. + """ + + def __init__(self, state_path: pathlib.Path) -> None: + self._path = state_path + + # ------------------------------------------------------------------ + # Bulk operations + # ------------------------------------------------------------------ + + def load(self) -> dict[str, FileSyncState]: + """Load all file states from disk. Returns ``{}`` if missing.""" + if not self._path.exists(): + return {} + try: + raw = json.loads(self._path.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + return {} + return { + k: FileSyncState(**v) + for k, v in raw.items() + if isinstance(v, dict) and "mtime" in v and "content_hash" in v + } + + def save(self, states: dict[str, FileSyncState]) -> None: + """Persist all file states to disk (creates parent dirs).""" + self._path.parent.mkdir(parents=True, exist_ok=True) + payload = {k: asdict(v) for k, v in states.items()} + self._path.write_text( + json.dumps(payload, indent=2, ensure_ascii=False), + encoding="utf-8", + ) + + # ------------------------------------------------------------------ + # Convenience helpers + # ------------------------------------------------------------------ + + def get(self, file_path: str) -> FileSyncState | None: + return self.load().get(file_path) + + def all_paths(self) -> set[str]: + return set(self.load().keys()) \ No newline at end of file diff --git a/src/contextseek/plugs/obsidian/wikilinks.py b/src/contextseek/plugs/obsidian/wikilinks.py new file mode 100644 index 0000000..ad83b82 --- /dev/null +++ b/src/contextseek/plugs/obsidian/wikilinks.py @@ -0,0 +1,63 @@ +"""Parse Obsidian ``[[wikilinks]]`` and resolve them to typed ``Link`` edges. + +Obsidian wikilink syntax:: + + [[PageName]] — link to ``PageName`` + [[PageName|Alias]] — link to ``PageName`` with display text ``Alias`` + [[PageName#Heading]] — link to a heading within ``PageName`` + +This module extracts the **target page name** (stripping aliases and +heading anchors) so it can be matched against the file-name → item-id +mapping built during sync. +""" + +from __future__ import annotations + +import re + +# Matches [[Target]] or [[Target|Alias]] or [[Target#Heading]] or +# [[Target#Heading|Alias]] — captures the target name in group 1. +WIKILINK_RE = re.compile( + r"\[\[" + r"([^\]|#]+)" # target page name (no ], |, or #) + r"(?:#[^\]|]*)?" # optional #heading anchor + r"(?:\|[^\]]*)?" # optional |alias + r"\]\]" +) + + +def extract_wikilinks(content: str) -> list[str]: + """Return a list of target page names from all ``[[wikilinks]]`` in *content*. + + Aliases and heading anchors are stripped — only the page name is kept. + Duplicates are removed while preserving first-seen order. + """ + seen: set[str] = set() + results: list[str] = [] + for match in WIKILINK_RE.finditer(content): + target = match.group(1).strip() + if target and target not in seen: + seen.add(target) + results.append(target) + return results + + +def resolve_wikilink_targets( + content: str, + name_to_item_id: dict[str, str], +) -> list[str]: + """Resolve ``[[wikilinks]]`` in *content* to target ContextItem IDs. + + Only wikilinks whose target page name exists in *name_to_item_id* + are returned. The returned list contains **item IDs** (not page + names), with duplicates removed. + """ + targets = extract_wikilinks(content) + seen: set[str] = set() + item_ids: list[str] = [] + for target in targets: + item_id = name_to_item_id.get(target) + if item_id and item_id not in seen: + seen.add(item_id) + item_ids.append(item_id) + return item_ids \ No newline at end of file diff --git a/src/contextseek/retrieval/orchestrator.py b/src/contextseek/retrieval/orchestrator.py index bb437b5..c695fbc 100644 --- a/src/contextseek/retrieval/orchestrator.py +++ b/src/contextseek/retrieval/orchestrator.py @@ -6,7 +6,7 @@ from datetime import datetime, timezone import math from time import perf_counter -from typing import Any, Callable +from typing import Any, Callable, Literal from contextseek.storage.protocol import SeekVFSAdapter from contextseek.config import RetrievalStrategy @@ -125,6 +125,7 @@ def search( k: int, stage: Stage | None = None, tags: list[str] | None = None, + tag_match: Literal["all", "any"] = "all", include_deleted: bool = False, include_expired: bool = False, with_stats: bool = False, @@ -139,7 +140,11 @@ def search( query: User query string. k: Maximum number of results to return. stage: Optional stage filter — only include items matching this stage. - tags: Optional tags filter — only include items having ALL these tags. + tags: Optional tags filter — only include items matching the tag + predicate controlled by ``tag_match``. + tag_match: Tag match mode — ``"all"`` (default) requires the item to + have *every* tag in ``tags``; ``"any"`` requires at least one + matching tag. include_deleted: Whether to include soft-deleted items. with_stats: If True, return (hits, stats) tuple. min_score: Optional threshold applied to the reranker's raw ``_score`` @@ -230,8 +235,13 @@ def _keep(h: dict[str, object]) -> bool: if stage is not None and h.get("stage") != stage.value: return False if tags: - if not set(tags).issubset(set(h.get("tags") or [])): - return False + item_tags = set(h.get("tags") or []) + if tag_match == "any": + if set(tags).isdisjoint(item_tags): + return False + else: # "all" (default) + if not set(tags).issubset(item_tags): + return False return True ranked_streams = [ diff --git a/tests/integration_tests/test_full_pipeline_file_example.py b/tests/integration_tests/test_full_pipeline_file_example.py index 47baf39..8212347 100644 --- a/tests/integration_tests/test_full_pipeline_file_example.py +++ b/tests/integration_tests/test_full_pipeline_file_example.py @@ -72,6 +72,36 @@ def test_hits_tag_filter_requires_all_tags(tmp_path: Path) -> None: assert hits[0].item.id == vector_item_id +def test_hits_tag_filter_any_semantics(tmp_path: Path) -> None: + with file_backend_demo_stack(tmp_path) as s: + ocean_item_id = s.item_ids[0] + vector_item_id = s.item_ids[1] + response = s.ctx.retrieve( + "分布式 向量", + scope=s.scope, + k=5, + filters={"tags": ["database", "vector"], "tag_match": "any"}, + ) + hits = list(response) + hit_ids = {h.item.id for h in hits} + assert ocean_item_id in hit_ids + assert vector_item_id in hit_ids + + +def test_hits_tag_filter_any_single_tag(tmp_path: Path) -> None: + with file_backend_demo_stack(tmp_path) as s: + langchain_item_id = s.item_ids[2] + response = s.ctx.retrieve( + "embedding", + scope=s.scope, + k=5, + filters={"tags": ["embedding"], "tag_match": "any"}, + ) + hits = list(response) + assert len(hits) == 1 + assert hits[0].item.id == langchain_item_id + + def test_response_meta_carries_layer(tmp_path: Path) -> None: with file_backend_demo_stack(tmp_path) as s: response = s.ctx.retrieve("LangChain", scope=s.scope, k=3) diff --git a/tests/integration_tests/test_obsidian_vault_sync.py b/tests/integration_tests/test_obsidian_vault_sync.py new file mode 100644 index 0000000..af26cf5 --- /dev/null +++ b/tests/integration_tests/test_obsidian_vault_sync.py @@ -0,0 +1,340 @@ +"""Integration tests for incremental Obsidian vault sync. + +Tests the full sync flow: import → re-sync (no writes) → edit → re-sync +(changed only) → wikilink → Link edges → delete → soft-delete. +""" + +from __future__ import annotations + +import pathlib +import time + +import pytest + +from contextseek import ContextSeek +from contextseek.domain.links import LinkType +from contextseek.plugs.obsidian import ObsidianVaultPlug + + +@pytest.fixture +def vault(tmp_path: pathlib.Path) -> pathlib.Path: + """Create a minimal Obsidian vault with wikilinks.""" + v = tmp_path / "vault" + v.mkdir() + + (v / "PageA.md").write_text( + "# Page A\n\n" + "This page references [[PageB]] and [[PageC|Custom Alias]].\n", + encoding="utf-8", + ) + (v / "PageB.md").write_text( + "# Page B\n\n" + "Backlink to [[PageA]].\n", + encoding="utf-8", + ) + (v / "PageC.md").write_text( + "---\ntags: [reference, guide]\n---\n\n" + "# Page C\n\n" + "Plain content with no wikilinks.\n", + encoding="utf-8", + ) + return v + + +def _sync_vault(ctx: ContextSeek, vault: pathlib.Path, scope: str) -> ObsidianVaultPlug: + """Run a full sync cycle and return the plug instance.""" + plug = ObsidianVaultPlug(vault_path=vault, scope=scope) + for event in plug.stream(): + item = ctx.add( + event.content, + scope=scope, + source=event.source, + source_type="document", + tags=event.tags, + check_conflicts=False, + ) + plug.record_item_id(event.source, item.id) + plug.persist_state() + plug.resolve_links(ctx) + plug.handle_deletes(ctx) + return plug + + +def test_first_sync_imports_all_files(vault: pathlib.Path) -> None: + ctx = ContextSeek() + plug = _sync_vault(ctx, vault, "obs/vault") + assert plug.added == 3 + assert plug.skipped == 0 + + items = ctx.items(scope="obs/vault") + assert len(items) == 3 + + +def test_second_sync_unchanged_no_writes(vault: pathlib.Path) -> None: + ctx = ContextSeek() + _sync_vault(ctx, vault, "obs/vault") + + # Second sync — no changes + plug = ObsidianVaultPlug(vault_path=vault, scope="obs/vault") + events = list(plug.stream()) + assert len(events) == 0 + assert plug.added == 0 + assert plug.skipped == 3 + + +def test_edited_file_produces_update(vault: pathlib.Path) -> None: + ctx = ContextSeek() + _sync_vault(ctx, vault, "obs/vault") + + # Edit PageA + time.sleep(0.01) # ensure mtime differs + (vault / "PageA.md").write_text("# Page A\n\nEdited content here.\n", encoding="utf-8") + + plug = ObsidianVaultPlug(vault_path=vault, scope="obs/vault") + events = list(plug.stream()) + assert len(events) == 1 + assert plug.added == 1 + assert plug.skipped == 2 + assert "PageA" in events[0].source + + +def test_wikilinks_become_links(vault: pathlib.Path) -> None: + ctx = ContextSeek() + plug = _sync_vault(ctx, vault, "obs/vault") + + # PageA has [[PageB]] and [[PageC|Custom Alias]] + # PageB has [[PageA]] + # After resolve_links, those should be Link(relation=related_to) edges + items = {pathlib.Path(it.provenance.source_id).stem: it for it in ctx.items(scope="obs/vault")} + + assert "PageA" in items + assert "PageB" in items + assert "PageC" in items + + page_a = items["PageA"] + link_targets = {link.target_id for link in page_a.links if link.relation == LinkType.related_to} + assert items["PageB"].id in link_targets + assert items["PageC"].id in link_targets + + page_b = items["PageB"] + link_targets_b = {link.target_id for link in page_b.links if link.relation == LinkType.related_to} + assert items["PageA"].id in link_targets_b + + +def test_deleted_file_soft_deleted(vault: pathlib.Path) -> None: + ctx = ContextSeek() + _sync_vault(ctx, vault, "obs/vault") + + # Get PageC's item ID from sync state before deleting + from contextseek.plugs.obsidian.sync_state import SyncStateStore + + state_file = vault / ".contextseek" / "sync_state.json" + states = SyncStateStore(state_file).load() + page_c_path = (vault / "PageC.md").as_posix() + page_c_state = states[page_c_path] + page_c_id = page_c_state.context_item_id + + # Delete PageC from vault + (vault / "PageC.md").unlink() + + # Re-sync + plug = ObsidianVaultPlug(vault_path=vault, scope="obs/vault") + list(plug.stream()) + + deletes = plug.handle_deletes(ctx) + assert deletes == 1 + + # PageC's item should be soft-deleted (searchable=False) + from contextseek.domain.serialization import deserialize_context_item + + ref = ctx.resolver.ref_for("obs/vault", page_c_id) + payload = ctx.adapter.read(ref) + assert payload is not None + item = deserialize_context_item(payload) + assert item.is_deleted is True + assert item.searchable is False + + +def test_frontmatter_tags_imported(vault: pathlib.Path) -> None: + ctx = ContextSeek() + _sync_vault(ctx, vault, "obs/vault") + + # PageC has front-matter tags: [reference, guide] + items = ctx.items(scope="obs/vault") + page_c = next( + it for it in items + if pathlib.Path(it.provenance.source_id).stem == "PageC" + ) + assert "reference" in page_c.tags + assert "guide" in page_c.tags + assert "obsidian" in page_c.tags + + +def test_sync_state_persisted(vault: pathlib.Path) -> None: + ctx = ContextSeek() + _sync_vault(ctx, vault, "obs/vault") + + state_file = vault / ".contextseek" / "sync_state.json" + assert state_file.exists() + + # State should contain all 3 files + from contextseek.plugs.obsidian.sync_state import SyncStateStore + + states = SyncStateStore(state_file).load() + assert len(states) == 3 + assert all(s.context_item_id for s in states.values()) + + +def test_no_self_links(vault: pathlib.Path) -> None: + """A file should not get a Link pointing to itself even if it + contains a [[wikilink]] to its own page name.""" + ctx = ContextSeek() + _sync_vault(ctx, vault, "obs/vault") + + items = {pathlib.Path(it.provenance.source_id).stem: it for it in ctx.items(scope="obs/vault")} + for item in items.values(): + for link in item.links: + assert link.target_id != item.id + + +def test_resolve_links_idempotent(vault: pathlib.Path) -> None: + """Re-running resolve_links should not create duplicate Link edges.""" + ctx = ContextSeek() + plug = _sync_vault(ctx, vault, "obs/vault") + + # Count links after first resolve + items_before = ctx.items(scope="obs/vault") + links_before = sum(len(it.links) for it in items_before) + + # Run resolve_links again + plug.resolve_links(ctx) + + items_after = ctx.items(scope="obs/vault") + links_after = sum(len(it.links) for it in items_after) + + assert links_after == links_before + + +def test_wikilink_to_nonexistent_page_no_link(vault: pathlib.Path) -> None: + """A [[wikilink]] to a page that doesn't exist in the vault should + not create a Link edge (only resolved wikilinks become links).""" + ctx = ContextSeek() + # Add a wikilink to a non-existent page in PageA + (vault / "PageA.md").write_text( + "# Page A\n\n[[PageB]] and [[NonExistentPage]].\n", encoding="utf-8" + ) + _sync_vault(ctx, vault, "obs/vault") + + items = {pathlib.Path(it.provenance.source_id).stem: it for it in ctx.items(scope="obs/vault")} + page_a = items["PageA"] + link_targets = {link.target_id for link in page_a.links if link.relation == LinkType.related_to} + # Should link to PageB but NOT to any item for NonExistentPage + assert items["PageB"].id in link_targets + # There should be exactly 1 link (PageB only, not NonExistentPage) + assert len(link_targets) == 1 + + +def test_add_new_file_then_sync(vault: pathlib.Path) -> None: + """Adding a new file to the vault and re-syncing imports only the new file.""" + ctx = ContextSeek() + _sync_vault(ctx, vault, "obs/vault") + assert len(ctx.items(scope="obs/vault")) == 3 + + # Add a new file + (vault / "PageD.md").write_text("# Page D\n\nNew page.\n", encoding="utf-8") + plug = _sync_vault(ctx, vault, "obs/vault") + + assert plug.added == 1 + assert plug.skipped == 3 + assert len(ctx.items(scope="obs/vault")) == 4 + + +def test_edit_and_add_then_sync(vault: pathlib.Path) -> None: + """Editing one file and adding another in a single re-sync cycle.""" + ctx = ContextSeek() + _sync_vault(ctx, vault, "obs/vault") + + time.sleep(0.01) + (vault / "PageA.md").write_text("# Page A\n\nEdited.\n", encoding="utf-8") + (vault / "PageD.md").write_text("# Page D\n\nNew.\n", encoding="utf-8") + + plug = _sync_vault(ctx, vault, "obs/vault") + assert plug.added == 2 + assert plug.skipped == 2 + + +def test_delete_then_readd(vault: pathlib.Path) -> None: + """Deleting a file, syncing, then re-adding it creates a new item.""" + ctx = ContextSeek() + _sync_vault(ctx, vault, "obs/vault") + + # Delete PageC and sync (with persist) + (vault / "PageC.md").unlink() + _sync_vault(ctx, vault, "obs/vault") + + # Re-add PageC + (vault / "PageC.md").write_text("# Page C\n\nReborn.\n", encoding="utf-8") + plug2 = ObsidianVaultPlug(vault_path=vault, scope="obs/vault") + events = list(plug2.stream()) + assert len(events) == 1 + assert plug2.added == 1 + + +def test_nested_directories_sync(tmp_path: pathlib.Path) -> None: + """Files in nested subdirectories are synced correctly.""" + v = tmp_path / "vault" + v.mkdir() + (v / "PageA.md").write_text("[[notes/PageB]]", encoding="utf-8") + (v / "notes").mkdir() + (v / "notes" / "PageB.md").write_text("[[PageA]]", encoding="utf-8") + + ctx = ContextSeek() + plug = _sync_vault(ctx, v, "obs/vault") + assert plug.added == 2 + assert len(ctx.items(scope="obs/vault")) == 2 + + # Check wikilinks resolved — note: stem is "PageB" not "notes/PageB" + items = {pathlib.Path(it.provenance.source_id).stem: it for it in ctx.items(scope="obs/vault")} + # PageA has [[notes/PageB]] which won't match stem "PageB" + # This is a known limitation — wikilinks with paths are not resolved + # to the file stem. Only simple [[PageName]] wikilinks are resolved. + page_b = items["PageB"] + link_targets_b = {link.target_id for link in page_b.links if link.relation == LinkType.related_to} + assert items["PageA"].id in link_targets_b # PageB links to [[PageA]] + + +def test_full_cycle_sync(vault: pathlib.Path) -> None: + """Complete cycle: first sync → second sync (no writes) → edit → + re-sync (changed only) → delete → re-sync (soft-delete) → add → + re-sync (new only).""" + ctx = ContextSeek() + + # 1. First sync + plug1 = _sync_vault(ctx, vault, "obs/vault") + assert plug1.added == 3 + + # 2. Second sync — no changes + plug2 = ObsidianVaultPlug(vault_path=vault, scope="obs/vault") + events2 = list(plug2.stream()) + assert len(events2) == 0 + assert plug2.skipped == 3 + + # 3. Edit PageA and re-sync (with persist) + time.sleep(0.01) + (vault / "PageA.md").write_text("# Page A\n\nUpdated.\n[[PageB]]\n", encoding="utf-8") + plug3 = _sync_vault(ctx, vault, "obs/vault") + assert plug3.added == 1 + assert plug3.skipped == 2 + + # 4. Delete PageC and re-sync (with persist — _sync_vault handles deletes) + (vault / "PageC.md").unlink() + plug4 = _sync_vault(ctx, vault, "obs/vault") + # _sync_vault already called handle_deletes internally + + # 5. Add PageD and re-sync + (vault / "PageD.md").write_text("# Page D\n\nNew.\n", encoding="utf-8") + plug5 = ObsidianVaultPlug(vault_path=vault, scope="obs/vault") + events5 = list(plug5.stream()) + assert len(events5) == 1 + assert plug5.added == 1 + assert plug5.skipped == 2 # PageA (updated in step 3), PageB — PageC vanished \ No newline at end of file diff --git a/tests/unit_tests/test_cli.py b/tests/unit_tests/test_cli.py index 11fa121..be9c9d4 100644 --- a/tests/unit_tests/test_cli.py +++ b/tests/unit_tests/test_cli.py @@ -84,6 +84,8 @@ def test_openai_format_outputs_valid_tool_definitions(self, capsys): "scope", "k", "full", + "tags", + "tag_match", } assert expand_tool["parameters"]["required"] == ["ids", "scope"] assert expand_tool["parameters"]["properties"]["ids"]["type"] == "array" @@ -108,6 +110,8 @@ def test_anthropic_format_outputs_valid_tool_definitions(self, capsys): "scope", "k", "full", + "tags", + "tag_match", } assert expand_tool["input_schema"]["required"] == ["ids", "scope"] assert expand_tool["input_schema"]["properties"]["ids"]["items"] == { @@ -159,8 +163,60 @@ def test_retrieve_filters_results_by_all_tags(self) -> None: assert code == 0 assert [item["id"] for item in payload["items"]] == [kept.id] + def test_retrieve_accepts_tag_match_flag(self) -> None: + parser = build_parser() + args = parser.parse_args( + ["retrieve", "--scope", "t", "--query", "q", "--tags", "a,b", "--tag-match", "any"] + ) + + assert args.tag_match == "any" + + def test_retrieve_tag_match_any_returns_items_with_any_tag(self) -> None: + ctx = ContextSeek() + kept_a = ctx.add( + "database backup runbook", + scope="t/p", + source="test", + tags=["ops", "database"], + ) + kept_b = ctx.add( + "database onboarding guide", + scope="t/p", + source="test", + tags=["docs", "database"], + ) + out = StringIO() + + with redirect_stdout(out): + code = run_cli( + [ + "retrieve", + "--scope", + "t/p", + "--query", + "database", + "--tags", + "ops,docs", + "--tag-match", + "any", + "--json", + ], + client=ctx, + ) + + payload = json.loads(out.getvalue()) + assert code == 0 + ids = {item["id"] for item in payload["items"]} + assert ids == {kept_a.id, kept_b.id} + + def test_retrieve_tag_match_all_still_default(self) -> None: + parser = build_parser() + args = parser.parse_args( + ["retrieve", "--scope", "t", "--query", "q", "--tags", "a,b"] + ) + + assert args.tag_match == "all" -class TestExpandOutput: def test_expand_reports_missing_ids(self) -> None: ctx = ContextSeek() item = ctx.add("expand target", scope="t/p", source="test") @@ -182,3 +238,564 @@ def test_expand_reports_missing_ids(self) -> None: assert code == 0 assert [it["id"] for it in payload["items"]] == [item.id] assert payload["missing_ids"] == ["missing-id"] + + +class TestDoctor: + """Tests for the ``contextseek doctor`` subcommand.""" + + # ------------------------------------------------------------------ + # Parser registration + # ------------------------------------------------------------------ + + def test_doctor_parser_registered(self) -> None: + """``doctor`` subcommand is registered and parseable.""" + parser = build_parser() + args = parser.parse_args(["doctor"]) + assert args.command == "doctor" + + def test_doctor_in_help(self) -> None: + """``doctor`` appears in the main parser help text.""" + parser = build_parser() + help_text = parser.format_help() + assert "doctor" in help_text + assert "diagnose" in help_text + + # ------------------------------------------------------------------ + # Storage checks + # ------------------------------------------------------------------ + + def test_doctor_all_none_passes(self, capsys) -> None: + """Default isolated env (memory/none/none) → exit 0 with PASS + SKIPs.""" + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[PASS] storage" in out + assert "[SKIP] embedding" in out + assert "[SKIP] llm" in out + assert "[FAIL]" not in out + assert "exit 0" not in out # no failure, no exit suffix + # Should mention .env.example in the SKIP hints + assert ".env.example" in out + + def test_doctor_storage_memory_pass(self, monkeypatch, capsys) -> None: + """Memory backend → PASS.""" + monkeypatch.setenv("STORAGE_BACKEND", "memory") + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[PASS] storage" in out + assert "memory backend initialized" in out + + def test_doctor_storage_sqlite_pass(self, monkeypatch, tmp_path, capsys) -> None: + """SQLite backend with a temp path → PASS.""" + db_path = str(tmp_path / "test.sqlite3") + monkeypatch.setenv("STORAGE_BACKEND", "sqlite") + monkeypatch.setenv("SQLITE_PATH", db_path) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[PASS] storage" in out + assert "sqlite backend initialized" in out + + def test_doctor_storage_file_pass(self, monkeypatch, tmp_path, capsys) -> None: + """File backend with a temp path → PASS.""" + file_path = str(tmp_path / "store") + monkeypatch.setenv("STORAGE_BACKEND", "file") + monkeypatch.setenv("STORAGE_PATH", file_path) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[PASS] storage" in out + assert "file backend initialized" in out + + def test_doctor_storage_oceanbase_missing_dims_fails( + self, monkeypatch, capsys + ) -> None: + """OceanBase backend without EMBEDDING_DIMS → storage FAIL, exit 1.""" + monkeypatch.setenv("STORAGE_BACKEND", "oceanbase") + monkeypatch.setenv("EMBEDDING_PROVIDER", "none") + monkeypatch.setenv("EMBEDDING_DIMS", "0") + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert "[FAIL] storage" in out + assert "EMBEDDING_DIMS" in out + assert ".env.example" in out + + def test_doctor_storage_unknown_backend_fails(self, monkeypatch, capsys) -> None: + """Unknown storage backend → FAIL.""" + monkeypatch.setenv("STORAGE_BACKEND", "unknown_backend") + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert "[FAIL] storage" in out + assert "Unknown storage backend" in out + assert "memory, file, sqlite, seekdb, oceanbase" in out + + def test_doctor_storage_empty_backend_fails(self, monkeypatch, capsys) -> None: + """Empty storage backend string → FAIL (unknown).""" + monkeypatch.setenv("STORAGE_BACKEND", "") + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert "[FAIL] storage" in out + + # ------------------------------------------------------------------ + # Embedding checks + # ------------------------------------------------------------------ + + def test_doctor_embedding_none_skips(self, capsys) -> None: + """Embedding provider=none → SKIP.""" + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[SKIP] embedding" in out + assert "vector search disabled" in out + + def test_doctor_embedding_langchain_no_class_path_skips( + self, monkeypatch, capsys + ) -> None: + """Embedding provider=langchain without class_path → SKIP.""" + monkeypatch.setenv("EMBEDDING_PROVIDER", "langchain") + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[SKIP] embedding" in out + assert "EMBEDDING_CLASS_PATH" in out + + def test_doctor_embedding_build_failure(self, monkeypatch, capsys) -> None: + """Embedding provider configured but build_embedder raises → FAIL.""" + monkeypatch.setenv("EMBEDDING_PROVIDER", "openai") + monkeypatch.setenv("EMBEDDING_MODEL", "text-embedding-3-small") + + def _fake_build_embedder(settings): + raise PermissionError("AuthenticationError: Incorrect API key provided") + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_embedder", _fake_build_embedder + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert "[FAIL] embedding" in out + assert "API key" in out + assert ".env.example" in out + + def test_doctor_embedding_probe_success(self, monkeypatch, capsys) -> None: + """Embedder successfully returns a vector → PASS.""" + monkeypatch.setenv("EMBEDDING_PROVIDER", "openai") + monkeypatch.setenv("EMBEDDING_MODEL", "text-embedding-3-small") + monkeypatch.setenv("EMBEDDING_DIMS", "4") + + def _fake_embedder(text): + return [0.1, 0.2, 0.3, 0.4] + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_embedder", lambda s: _fake_embedder + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[PASS] embedding" in out + assert "4-dim" in out + + def test_doctor_embedding_probe_returns_empty_fails( + self, monkeypatch, capsys + ) -> None: + """Embedder returns empty list → FAIL.""" + monkeypatch.setenv("EMBEDDING_PROVIDER", "openai") + + def _fake_embedder(text): + return [] + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_embedder", lambda s: _fake_embedder + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert "[FAIL] embedding" in out + + def test_doctor_embedding_probe_raises_fails(self, monkeypatch, capsys) -> None: + """Embedder probe call raises → FAIL.""" + monkeypatch.setenv("EMBEDDING_PROVIDER", "openai") + + def _fake_embedder(text): + raise ConnectionError("Connection refused to embedding server") + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_embedder", lambda s: _fake_embedder + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert "[FAIL] embedding" in out + assert "Connection" in out or "connection" in out + + def test_doctor_embedding_dims_mismatch_warns(self, monkeypatch, capsys) -> None: + """Embedder returns dims different from configured → WARN.""" + monkeypatch.setenv("EMBEDDING_PROVIDER", "openai") + monkeypatch.setenv("EMBEDDING_DIMS", "1536") + + def _fake_embedder(text): + return [0.1] * 768 # returns 768 but configured 1536 + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_embedder", lambda s: _fake_embedder + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + # WARN does not cause failure + assert code == 0 + assert "[WARN] embedding" in out + assert "768-dim" in out + assert "1536" in out + + def test_doctor_embedding_probe_returns_non_list_fails( + self, monkeypatch, capsys + ) -> None: + """Embedder returns non-list (e.g. None) → FAIL.""" + monkeypatch.setenv("EMBEDDING_PROVIDER", "openai") + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_embedder", lambda s: (lambda t: None) + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert "[FAIL] embedding" in out + + # ------------------------------------------------------------------ + # LLM checks + # ------------------------------------------------------------------ + + def test_doctor_llm_none_skips(self, capsys) -> None: + """LLM provider=none → SKIP.""" + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[SKIP] llm" in out + assert "rerank/summarize/evolution" in out + + def test_doctor_llm_langchain_no_class_path_skips( + self, monkeypatch, capsys + ) -> None: + """LLM provider=langchain without class_path → SKIP.""" + monkeypatch.setenv("LLM_PROVIDER", "langchain") + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[SKIP] llm" in out + assert "LLM_CLASS_PATH" in out + + def test_doctor_llm_build_failure(self, monkeypatch, capsys) -> None: + """LLM build_llm raises → FAIL.""" + monkeypatch.setenv("LLM_PROVIDER", "openai") + + def _fake_build_llm(settings): + raise ImportError("No module named 'langchain_openai'") + + monkeypatch.setattr("contextseek.cli.doctor_cmd.build_llm", _fake_build_llm) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert "[FAIL] llm" in out + assert "langchain" in out.lower() + + def test_doctor_llm_invoke_failure(self, monkeypatch, capsys) -> None: + """LLM provider configured but invoke raises → FAIL.""" + monkeypatch.setenv("LLM_PROVIDER", "openai") + monkeypatch.setenv("LLM_MODEL", "gpt-4o-mini") + + class _FakeLLM: + def invoke(self, messages): + raise ConnectionError("Connection refused to api.openai.com") + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_llm", lambda s: _FakeLLM() + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert "[FAIL] llm" in out + assert ".env.example" in out + + def test_doctor_llm_invoke_success(self, monkeypatch, capsys) -> None: + """LLM responds to probe → PASS.""" + monkeypatch.setenv("LLM_PROVIDER", "openai") + monkeypatch.setenv("LLM_MODEL", "gpt-4o-mini") + + class _FakeLLM: + def invoke(self, messages): + return "hello back" + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_llm", lambda s: _FakeLLM() + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[PASS] llm" in out + + def test_doctor_llm_invoke_returns_none_fails(self, monkeypatch, capsys) -> None: + """LLM invoke returns None → FAIL.""" + monkeypatch.setenv("LLM_PROVIDER", "openai") + + class _FakeLLM: + def invoke(self, messages): + return None + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_llm", lambda s: _FakeLLM() + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert "[FAIL] llm" in out + assert "None" in out + + # ------------------------------------------------------------------ + # Cross-dependency checks + # ------------------------------------------------------------------ + + def test_doctor_cross_dependency_seekdb_warn(self, monkeypatch, capsys) -> None: + """seekdb + embedding=none → WARN about built-in embedding.""" + monkeypatch.setenv("STORAGE_BACKEND", "seekdb") + monkeypatch.setenv("EMBEDDING_PROVIDER", "none") + + from contextseek.cli.doctor_cmd import CheckResult, PASS + + def _fake_check_storage(settings): + return CheckResult(PASS, "storage", "seekdb backend initialized") + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd._check_storage", _fake_check_storage + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[WARN] cross" in out + assert "seekdb" in out.lower() + + def test_doctor_cross_dependency_llm_without_embedding_warn( + self, monkeypatch, capsys + ) -> None: + """LLM configured but embedding=none → WARN about missing vector search.""" + monkeypatch.setenv("EMBEDDING_PROVIDER", "none") + monkeypatch.setenv("LLM_PROVIDER", "openai") + + from contextseek.cli.doctor_cmd import CheckResult, PASS + + def _fake_check_llm(settings): + return CheckResult(PASS, "llm", "LLM responded") + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd._check_llm", _fake_check_llm + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[WARN] cross" in out + assert "vector search" in out.lower() + + def test_doctor_no_cross_warning_when_both_configured( + self, monkeypatch, capsys + ) -> None: + """Both embedding and LLM configured → no cross WARN.""" + monkeypatch.setenv("EMBEDDING_PROVIDER", "openai") + monkeypatch.setenv("LLM_PROVIDER", "openai") + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_embedder", + lambda s: (lambda t: [0.1, 0.2]), + ) + + class _FakeLLM: + def invoke(self, messages): + return "ok" + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_llm", lambda s: _FakeLLM() + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 0 + assert "[WARN] cross" not in out + + # ------------------------------------------------------------------ + # Secret sanitisation + # ------------------------------------------------------------------ + + def test_doctor_no_secrets_leaked_sk_key(self, monkeypatch, capsys) -> None: + """Doctor output must never contain raw sk- keys from error messages.""" + fake_secret = "sk-FAKESECRET1234567890abcdef" + + def _fake_build_embedder(settings): + raise PermissionError( + f"AuthenticationError: Incorrect API key provided: {fake_secret}" + ) + + monkeypatch.setenv("EMBEDDING_PROVIDER", "openai") + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_embedder", _fake_build_embedder + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert fake_secret not in out + assert "sk-***" in out + + def test_doctor_no_secrets_leaked_password(self, monkeypatch, capsys) -> None: + """Doctor output must not leak password= values from error messages.""" + fake_password = "SuperSecret123!" + + def _fake_build_embedder(settings): + raise ValueError(f"Connection failed: password={fake_password} rejected") + + monkeypatch.setenv("EMBEDDING_PROVIDER", "openai") + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_embedder", _fake_build_embedder + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert fake_password not in out + assert "password=***" in out + + def test_doctor_no_secrets_leaked_token(self, monkeypatch, capsys) -> None: + """Doctor output must not leak token= values from error messages.""" + fake_token = "tok_abcDEF123456" + + def _fake_build_embedder(settings): + raise RuntimeError(f"Auth failed: token={fake_token} is invalid") + + monkeypatch.setenv("EMBEDDING_PROVIDER", "openai") + monkeypatch.setattr( + "contextseek.cli.doctor_cmd.build_embedder", _fake_build_embedder + ) + code = run_cli(["doctor"]) + out = capsys.readouterr().out + + assert code == 1 + assert fake_token not in out + assert "token=***" in out + + def test_doctor_config_report_no_password(self, monkeypatch, capsys) -> None: + """OceanBase config report must not include the password field.""" + monkeypatch.setenv("STORAGE_BACKEND", "oceanbase") + monkeypatch.setenv("OB_PASSWORD", "should_not_appear_in_output") + monkeypatch.setenv("EMBEDDING_DIMS", "1536") + + # Patch storage check to avoid real OB connection + from contextseek.cli.doctor_cmd import CheckResult, PASS + + monkeypatch.setattr( + "contextseek.cli.doctor_cmd._check_storage", + lambda s: CheckResult(PASS, "storage", "ok"), + ) + run_cli(["doctor"]) + out = capsys.readouterr().out + + assert "should_not_appear_in_output" not in out + + # ------------------------------------------------------------------ + # Sanitisation unit tests + # ------------------------------------------------------------------ + + def test_sanitize_truncates_long_messages(self) -> None: + """_sanitize_error_message truncates messages over 200 chars.""" + from contextseek.cli.doctor_cmd import _sanitize_error_message + + long_msg = "x" * 300 + result = _sanitize_error_message(long_msg) + assert len(result) <= 201 # 200 + ellipsis + assert result.endswith("…") + + def test_sanitize_preserves_short_messages(self) -> None: + """_sanitize_error_message preserves short messages unchanged.""" + from contextseek.cli.doctor_cmd import _sanitize_error_message + + assert _sanitize_error_message("short error") == "short error" + + def test_sanitize_sk_key_pattern(self) -> None: + """_sanitize_error_message masks sk- prefixed keys.""" + from contextseek.cli.doctor_cmd import _sanitize_error_message + + result = _sanitize_error_message("error: sk-ABCDEFGHIJKLMNOPQRSTUVWXYZ1234") + assert "sk-***" in result + assert "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234" not in result + + # ------------------------------------------------------------------ + # Output format structure + # ------------------------------------------------------------------ + + def test_doctor_output_has_header(self, capsys) -> None: + """Doctor output starts with the diagnostic header.""" + run_cli(["doctor"]) + out = capsys.readouterr().out + + assert "ContextSeek doctor" in out + assert "configuration diagnostics" in out.lower() + + def test_doctor_output_has_config_resolved_section(self, capsys) -> None: + """Doctor output includes a 'Configuration resolved:' section.""" + run_cli(["doctor"]) + out = capsys.readouterr().out + + assert "Configuration resolved:" in out + assert "storage" in out + assert "embedding" in out + assert "llm" in out + + def test_doctor_output_has_checks_section(self, capsys) -> None: + """Doctor output includes a 'Checks:' section.""" + run_cli(["doctor"]) + out = capsys.readouterr().out + + assert "Checks:" in out + + def test_doctor_output_has_result_summary(self, capsys) -> None: + """Doctor output ends with a 'Result:' summary line.""" + run_cli(["doctor"]) + out = capsys.readouterr().out + + assert "Result:" in out + assert "PASS" in out + assert "SKIP" in out + + def test_doctor_exit_code_nonzero_on_fail(self, monkeypatch, capsys) -> None: + """Exit code is 1 when any check FAILs.""" + monkeypatch.setenv("STORAGE_BACKEND", "bad_backend") + code = run_cli(["doctor"]) + assert code == 1 + + def test_doctor_exit_code_zero_on_all_pass_skip(self, capsys) -> None: + """Exit code is 0 when no checks FAIL.""" + code = run_cli(["doctor"]) + assert code == 0 diff --git a/tests/unit_tests/test_context_injection.py b/tests/unit_tests/test_context_injection.py index c177bfe..b20060c 100644 --- a/tests/unit_tests/test_context_injection.py +++ b/tests/unit_tests/test_context_injection.py @@ -61,6 +61,73 @@ def test_tags_filter_requires_all_tags(self): assert [hit.item.id for hit in tagged_response] == [kept.id] assert {hit.item.id for hit in unfiltered_response} == {kept.id, other.id} + def test_tags_filter_any_matches_partial(self): + ctx = ContextSeek() + kept = ctx.add( + "database backup runbook", + scope="t/p", + source="cli", + tags=["ops", "database"], + ) + other = ctx.add( + "database onboarding guide", + scope="t/p", + source="cli", + tags=["docs", "database"], + ) + + response = ctx.retrieve( + "database", scope="t/p", tags=["ops", "docs"], tag_match="any" + ) + + assert {hit.item.id for hit in response} == {kept.id, other.id} + + def test_tags_filter_any_with_non_matching_tag(self): + ctx = ContextSeek() + kept = ctx.add( + "database backup runbook", + scope="t/p", + source="cli", + tags=["ops", "database"], + ) + ctx.add( + "database onboarding guide", + scope="t/p", + source="cli", + tags=["docs", "database"], + ) + + response = ctx.retrieve( + "database", scope="t/p", tags=["ops", "nonexistent"], tag_match="any" + ) + + assert [hit.item.id for hit in response] == [kept.id] + + def test_tag_match_defaults_to_all(self): + ctx = ContextSeek() + kept = ctx.add( + "database backup runbook", + scope="t/p", + source="cli", + tags=["ops", "database"], + ) + ctx.add( + "database onboarding guide", + scope="t/p", + source="cli", + tags=["docs", "database"], + ) + + response_default = ctx.retrieve( + "database", scope="t/p", tags=["ops", "docs"] + ) + response_explicit_all = ctx.retrieve( + "database", scope="t/p", tags=["ops", "docs"], tag_match="all" + ) + + assert [hit.item.id for hit in response_default] == [] + assert [hit.item.id for hit in response_explicit_all] == [] + class TestExpand: def test_expand_returns_full_items_without_scope(self): diff --git a/tests/unit_tests/test_obsidian_plug.py b/tests/unit_tests/test_obsidian_plug.py new file mode 100644 index 0000000..9764a5b --- /dev/null +++ b/tests/unit_tests/test_obsidian_plug.py @@ -0,0 +1,355 @@ +"""Unit tests for Obsidian vault DataPlug components.""" + +from __future__ import annotations + +import pathlib + +import pytest + +from contextseek.plugs.obsidian.sync_state import FileSyncState, SyncStateStore +from contextseek.plugs.obsidian.wikilinks import ( + WIKILINK_RE, + extract_wikilinks, + resolve_wikilink_targets, +) +from contextseek.plugs.obsidian.plug import ( + ObsidianVaultPlug, + _extract_frontmatter_tags, + _iter_markdown_files, +) + + +# ─── SyncStateStore ───────────────────────────────────────────────── + + +class TestSyncStateStore: + def test_load_missing_file_returns_empty(self, tmp_path: pathlib.Path) -> None: + store = SyncStateStore(tmp_path / "sync_state.json") + assert store.load() == {} + + def test_save_then_load_roundtrip(self, tmp_path: pathlib.Path) -> None: + path = tmp_path / ".contextseek" / "sync_state.json" + store = SyncStateStore(path) + states = { + "vault/PageA.md": FileSyncState(mtime=1000.0, content_hash="abc", context_item_id="id-a"), + "vault/PageB.md": FileSyncState(mtime=2000.0, content_hash="def", context_item_id="id-b"), + } + store.save(states) + loaded = store.load() + assert loaded == states + + def test_all_paths(self, tmp_path: pathlib.Path) -> None: + store = SyncStateStore(tmp_path / "sync_state.json") + states = { + "a.md": FileSyncState(mtime=1.0, content_hash="x", context_item_id="1"), + "b.md": FileSyncState(mtime=2.0, content_hash="y", context_item_id="2"), + } + store.save(states) + assert store.all_paths() == {"a.md", "b.md"} + + +# ─── Wikilink parsing ─────────────────────────────────────────────── + + +class TestExtractWikilinks: + def test_simple_wikilink(self) -> None: + assert extract_wikilinks("See [[PageA]] for details") == ["PageA"] + + def test_aliased_wikilink(self) -> None: + assert extract_wikilinks("See [[PageA|Custom Alias]]") == ["PageA"] + + def test_heading_anchor(self) -> None: + assert extract_wikilinks("[[PageA#Section]]") == ["PageA"] + + def test_heading_with_alias(self) -> None: + assert extract_wikilinks("[[PageA#Section|Alias]]") == ["PageA"] + + def test_multiple_wikilinks(self) -> None: + content = "[[PageA]] and [[PageB|B]] and [[PageC#Head]]" + assert extract_wikilinks(content) == ["PageA", "PageB", "PageC"] + + def test_dedup_preserves_order(self) -> None: + content = "[[PageA]] [[PageB]] [[PageA]]" + assert extract_wikilinks(content) == ["PageA", "PageB"] + + def test_no_wikilinks(self) -> None: + assert extract_wikilinks("plain text without links") == [] + + def test_regex_pattern(self) -> None: + assert WIKILINK_RE.search("text [[Target]] more") is not None + + +class TestResolveWikilinkTargets: + def test_resolves_known_targets(self) -> None: + mapping = {"PageA": "id-a", "PageB": "id-b"} + content = "[[PageA]] references [[PageB]]" + assert resolve_wikilink_targets(content, mapping) == ["id-a", "id-b"] + + def test_skips_unknown_targets(self) -> None: + mapping = {"PageA": "id-a"} + content = "[[PageA]] and [[UnknownPage]]" + assert resolve_wikilink_targets(content, mapping) == ["id-a"] + + def test_dedup_item_ids(self) -> None: + mapping = {"PageA": "id-a"} + content = "[[PageA]] [[PageA]] [[PageA]]" + assert resolve_wikilink_targets(content, mapping) == ["id-a"] + + +# ─── Front-matter tag extraction ──────────────────────────────────── + + +class TestExtractFrontmatterTags: + def test_bracket_style(self) -> None: + text = "---\ntitle: Test\ntags: [python, testing, ci]\n---\n\nContent" + assert _extract_frontmatter_tags(text) == ["python", "testing", "ci"] + + def test_list_style(self) -> None: + text = "---\ntags:\n - python\n - testing\n---\n\nContent" + assert _extract_frontmatter_tags(text) == ["python", "testing"] + + def test_inline_style(self) -> None: + text = "---\ntags: python testing\n---\n\nContent" + assert _extract_frontmatter_tags(text) == ["python", "testing"] + + def test_no_frontmatter(self) -> None: + assert _extract_frontmatter_tags("plain text") == [] + + def test_empty_tags(self) -> None: + text = "---\ntitle: Test\n---\n\nContent" + assert _extract_frontmatter_tags(text) == [] + + +# ─── File iteration ───────────────────────────────────────────────── + + +class TestIterMarkdownFiles: + def test_finds_md_files(self, tmp_path: pathlib.Path) -> None: + (tmp_path / "PageA.md").write_text("A") + (tmp_path / "PageB.md").write_text("B") + (tmp_path / "readme.txt").write_text("not md") + files = _iter_markdown_files(tmp_path) + assert len(files) == 2 + assert all(f.suffix == ".md" for f in files) + + def test_skips_ignored_dirs(self, tmp_path: pathlib.Path) -> None: + (tmp_path / "PageA.md").write_text("A") + (tmp_path / ".obsidian").mkdir() + (tmp_path / ".obsidian" / "config.md").write_text("config") + (tmp_path / ".contextseek").mkdir() + (tmp_path / ".contextseek" / "sync_state.json").write_text("{}") + files = _iter_markdown_files(tmp_path) + assert len(files) == 1 + assert files[0].name == "PageA.md" + + +# ─── ObsidianVaultPlug ────────────────────────────────────────────── + + +class TestObsidianVaultPlug: + def test_metadata(self, tmp_path: pathlib.Path) -> None: + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + meta = plug.metadata() + assert meta.name == "obsidian_vault" + assert meta.source_type == "document" + assert "Obsidian" in meta.description + + def test_stream_yields_new_files(self, tmp_path: pathlib.Path) -> None: + (tmp_path / "PageA.md").write_text("# Page A\n\nContent of A") + (tmp_path / "PageB.md").write_text("# Page B\n\nContent of B") + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events = list(plug.stream()) + assert len(events) == 2 + assert plug.added == 2 + assert all("obsidian" in (e.tags or []) for e in events) + + def test_stream_skips_unchanged(self, tmp_path: pathlib.Path) -> None: + (tmp_path / "PageA.md").write_text("# Page A\n\nContent") + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + # First sync + events = list(plug.stream()) + assert len(events) == 1 + plug.record_item_id(events[0].source, "fake-id") + plug.persist_state() + + # Second sync — no changes + plug2 = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events2 = list(plug2.stream()) + assert len(events2) == 0 + assert plug2.skipped == 1 + assert plug2.added == 0 + + def test_stream_detects_vanished(self, tmp_path: pathlib.Path) -> None: + (tmp_path / "PageA.md").write_text("A") + (tmp_path / "PageB.md").write_text("B") + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events = list(plug.stream()) + plug.record_item_id(events[0].source, "id-a") + plug.record_item_id(events[1].source, "id-b") + plug.persist_state() + + # Delete PageB + (tmp_path / "PageB.md").unlink() + plug2 = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + list(plug2.stream()) + assert any("PageB" in v for v in plug2.vanished_files) + + def test_frontmatter_tags_in_event(self, tmp_path: pathlib.Path) -> None: + (tmp_path / "PageA.md").write_text( + "---\ntags: [python, web]\n---\n\n# Page A\n\nContent" + ) + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events = list(plug.stream()) + assert len(events) == 1 + tags = events[0].tags or [] + assert "python" in tags + assert "web" in tags + assert "obsidian" in tags + + def test_stream_empty_vault(self, tmp_path: pathlib.Path) -> None: + """An empty vault yields no events.""" + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events = list(plug.stream()) + assert len(events) == 0 + assert plug.added == 0 + assert plug.skipped == 0 + + def test_mtime_changed_content_same_skips(self, tmp_path: pathlib.Path) -> None: + """mtime changed but content hash is the same → file is skipped.""" + import time as _time + + (tmp_path / "PageA.md").write_text("unchanged content") + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + list(plug.stream()) + plug.record_item_id((tmp_path / "PageA.md").as_posix(), "id-a") + plug.persist_state() + + # Touch the file to change mtime without changing content + _time.sleep(0.02) + p = tmp_path / "PageA.md" + with p.open("a") as f: + f.flush() + import os + os.utime(p, None) + + plug2 = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events = list(plug2.stream()) + assert len(events) == 0 + assert plug2.skipped == 1 + assert plug2.added == 0 + + def test_nested_directories_scanned(self, tmp_path: pathlib.Path) -> None: + """Markdown files in nested directories are found.""" + (tmp_path / "subdir").mkdir() + (tmp_path / "PageA.md").write_text("A") + (tmp_path / "subdir" / "PageB.md").write_text("B") + (tmp_path / "subdir" / "nested").mkdir() + (tmp_path / "subdir" / "nested" / "PageC.md").write_text("C") + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events = list(plug.stream()) + assert len(events) == 3 + + def test_new_file_on_second_sync(self, tmp_path: pathlib.Path) -> None: + """Adding a file between syncs produces exactly one new event.""" + (tmp_path / "PageA.md").write_text("A") + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + list(plug.stream()) + plug.record_item_id((tmp_path / "PageA.md").as_posix(), "id-a") + plug.persist_state() + + (tmp_path / "PageB.md").write_text("B") + plug2 = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events = list(plug2.stream()) + assert len(events) == 1 + assert plug2.added == 1 + assert plug2.skipped == 1 + assert "PageB" in events[0].source + + def test_wikilink_to_nonexistent_page(self, tmp_path: pathlib.Path) -> None: + """A [[wikilink]] to a page that doesn't exist in the vault + should not cause an error — it's simply not resolved.""" + (tmp_path / "PageA.md").write_text("[[NonExistent]] and [[PageB]]") + (tmp_path / "PageB.md").write_text("B") + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events = list(plug.stream()) + assert len(events) == 2 + + def test_obsidian_config_dir_skipped(self, tmp_path: pathlib.Path) -> None: + """The .obsidian config directory should not be scanned.""" + (tmp_path / "PageA.md").write_text("A") + (tmp_path / ".obsidian").mkdir() + (tmp_path / ".obsidian" / "workspace.json").write_text("{}") + (tmp_path / ".obsidian" / "graph.json").write_text("{}") + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events = list(plug.stream()) + assert len(events) == 1 + + def test_non_md_files_ignored(self, tmp_path: pathlib.Path) -> None: + """Only .md files are picked up — .txt, .json, images etc. are ignored.""" + (tmp_path / "PageA.md").write_text("A") + (tmp_path / "notes.txt").write_text("text") + (tmp_path / "config.json").write_text("{}") + (tmp_path / "image.png").write_bytes(b"\x89PNG") + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events = list(plug.stream()) + assert len(events) == 1 + assert "PageA" in events[0].source + + def test_resolve_links_idempotent(self, tmp_path: pathlib.Path) -> None: + """Calling resolve_links twice should not create duplicate links.""" + from contextseek import ContextSeek + + (tmp_path / "PageA.md").write_text("[[PageB]]") + (tmp_path / "PageB.md").write_text("[[PageA]]") + ctx = ContextSeek() + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + for event in plug.stream(): + item = ctx.add(event.content, scope="test/vault", source=event.source, + source_type="document", tags=event.tags, check_conflicts=False) + plug.record_item_id(event.source, item.id) + plug.persist_state() + + links1 = plug.resolve_links(ctx) + links2 = plug.resolve_links(ctx) + assert links1 > 0 + assert links2 == 0 # No new links on second call + + def test_handle_deletes_no_vanished(self, tmp_path: pathlib.Path) -> None: + """handle_deletes returns 0 when no files have vanished.""" + from contextseek import ContextSeek + + (tmp_path / "PageA.md").write_text("A") + ctx = ContextSeek() + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + list(plug.stream()) + assert plug.handle_deletes(ctx) == 0 + + def test_handle_deletes_already_deleted(self, tmp_path: pathlib.Path) -> None: + """handle_deletes silently skips items that are already deleted.""" + from contextseek import ContextSeek + + (tmp_path / "PageA.md").write_text("A") + (tmp_path / "PageB.md").write_text("B") + ctx = ContextSeek() + plug = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + events = list(plug.stream()) + for e in events: + item = ctx.add(e.content, scope="test/vault", source=e.source, + source_type="document", tags=e.tags, check_conflicts=False) + plug.record_item_id(e.source, item.id) + plug.persist_state() + + # Delete both files + (tmp_path / "PageA.md").unlink() + (tmp_path / "PageB.md").unlink() + + plug2 = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + list(plug2.stream()) + first_deletes = plug2.handle_deletes(ctx) + assert first_deletes == 2 + + # Second call should return 0 (already deleted) + plug3 = ObsidianVaultPlug(vault_path=tmp_path, scope="test/vault") + list(plug3.stream()) + second_deletes = plug3.handle_deletes(ctx) + assert second_deletes == 0 \ No newline at end of file