Alkemio Virtual Contributor

Unified microkernel engine with pluggable handlers for AI-powered virtual contributors. Consolidates 7 formerly standalone services into a single Python 3.12 codebase using a microkernel + hexagonal (ports and adapters) architecture.

Expert evaluation resolves one frozen deeply immutable v7 composition before adapter wiring and stores invariant plus mode-bound full composition fingerprints. It uses the public evaluation-case-identity/v1 serializer and accepts only finite inclusive [0,1] case and aggregate metrics. Paired Expert comparison is fail-closed: it accepts only a failure-free flat-to-hierarchical v7 pair with matching canonical test-set, BoK, corpus-revision, explicit evaluation-case-identity/v1, and successful case identities. Corpus revision is an audit token, not deployment or re-ingestion proof; privacy approval, evaluation, enablement, and rollout stay human gates.

Query embeddings are bounded to 32768 UTF-8 bytes (rewrites to 4096), with up to three adapter-owned attempts under 20-second attempt and 45-second total deadlines. Compatible Chroma requests reuse only successful exact inputs within one request scope; provider tasks own successful cache publication and terminal eviction. Terminal results follow unpublished → published-unacked → settled, never raw-retrying after publication. Early-ACK tasks retrieve every terminal state with type-only diagnostics. published-unacked forbids application-managed raw republish or same-callback rerun, but an ambiguous ACK may be broker-redelivered and is not an exactly-once promise. Reject settlement faults are contained with type-only diagnostics. Expert identity is validated before provider/scorer/file work; each successful run has exactly four required finite unit metrics. Embedding failures never use flat fallback or RabbitMQ redelivery and receive a generic safe response.

This remediation evidence supports only the pending review sequence: review round 05 may begin after its independent gate, while rounds 06–07 remain contingent human reviews. It does not authorize enablement, deployment, or data mutation.

Features

Multi-provider LLM support — Mistral, OpenAI, and Anthropic via a unified LangChain adapter with per-plugin overrides
Configurable summarization — Optional separate LLM for ingest pipeline summarization
RAG with score filtering — Single-collection and multi-collection retrieval with configurable score thresholds and context budgets
PromptGraph engine — JSON-defined LangGraph workflows with dynamic Pydantic schema generation
Content-hash deduplication — SHA-256 fingerprinting with change detection and orphan cleanup during ingest
Ingest pipelines — Website crawling and Alkemio space tree ingestion with chunking, summarization, and embedding
RAG evaluation — RAGAS-based quality measurement with faithfulness, relevancy, precision, and recall metrics
Plugin architecture — Duck-typed protocol contract; add new plugins with zero core code changes
IoC container — Automatic dependency injection via __init__ type hint introspection
Sequential processing — prefetch=1 per instance; horizontal scaling via Kubernetes replicas
Health monitoring — HTTP health server with liveness (/healthz) and readiness (/readyz) endpoints

Architecture

The system follows two complementary patterns:

Microkernel — core/ is the core system that provides infrastructure (routing, configuration, dependency injection, transport). plugins/ are independently deployable components that implement the PluginContract protocol.

Hexagonal (Ports and Adapters) — External dependencies (LLMs, vector database, message broker, embeddings) are accessed through @runtime_checkable Protocol interfaces in core/ports/. Concrete adapters in core/adapters/ can be swapped without changing plugin code.

Ports (Interfaces)

Port	Methods	Purpose
`LLMPort`	`invoke(messages)`, `stream(messages)`	Chat model invocation and streaming
`EmbeddingsPort`	`embed(texts)`	Batch text embedding
`KnowledgeStorePort`	`query()`, `ingest()`, `get()`, `delete()`, `delete_collection()`	Vector database operations
`TransportPort`	`connect()`, `consume()`, `publish()`, `close()`	Message broker communication

Adapters (Implementations)

Adapter	Implements	Technology
`LangChainLLMAdapter`	`LLMPort`	Any LangChain `BaseChatModel` (Mistral, OpenAI, Anthropic)
`ChromaDBAdapter`	`KnowledgeStorePort`	ChromaDB (HTTP client mode)
`RabbitMQAdapter`	`TransportPort`	RabbitMQ via aio-pika
`OpenAICompatibleEmbeddingsAdapter`	`EmbeddingsPort`	OpenAI-compatible API (Scaleway, vLLM, etc.)
`OpenAIAssistantAdapter`	N/A	OpenAI Assistants API (threads, runs, files)

Message Flow

RabbitMQ message
    |
    v
Router.parse_event(body)          # Dispatches to correct Pydantic event model
    |
    v
Plugin.handle(event)              # Plugin-specific logic (RAG, LLM call, ingest, etc.)
    |
    v
Router.build_response_envelope()  # Wraps response for wire format
    |
    v
RabbitMQ publish (result queue)

Startup Sequence

main.py
  |- Load BaseConfig (env vars + .env)
  |- Setup JSON structured logging
  |- Resolve plugin-specific LLM overrides ({PLUGIN_NAME}_LLM_*)
  |- PluginRegistry.discover(PLUGIN_TYPE)
  |- Container.register() adapters:
  |    |- LLMPort        <- create_llm_adapter(config)
  |    |- EmbeddingsPort <- OpenAICompatibleEmbeddingsAdapter
  |    |- KnowledgeStorePort <- ChromaDBAdapter
  |    '- OpenAIAssistantAdapter
  |- Container.resolve_for_plugin(PluginClass)  # Auto-inject via type hints
  |- Plugin.startup()
  |- RabbitMQAdapter.connect() + consume(queue, handler)
  |- HealthServer.start(:8080)
  '- await SIGTERM/SIGINT -> graceful shutdown

Repository Structure

virtual-contributor/
├── core/                              # Core system
│   ├── ports/                         # Technology-agnostic interfaces
│   │   ├── llm.py                     # LLMPort — chat model invocation
│   │   ├── embeddings.py              # EmbeddingsPort — text embedding
│   │   ├── knowledge_store.py         # KnowledgeStorePort — vector DB
│   │   └── transport.py               # TransportPort — message bus
│   ├── adapters/                      # Concrete implementations
│   │   ├── langchain_llm.py           # Unified LLM (any LangChain provider)
│   │   ├── openai_assistant.py        # OpenAI Assistants API
│   │   ├── chromadb.py                # ChromaDB vector store
│   │   ├── rabbitmq.py                # RabbitMQ transport (aio-pika)
│   │   ├── openai_compatible_embeddings.py  # OpenAI-compatible embeddings
│   │   └── openai_embeddings.py       # OpenAI embeddings
│   ├── domain/                        # Shared domain logic
│   │   ├── prompt_graph.py            # LangGraph workflow engine
│   │   ├── ingest_pipeline.py         # Document, Chunk, IngestResult models
│   │   ├── summarize_graph.py         # Summarize-then-refine pattern
│   │   └── pipeline/                  # Ingest pipeline engine
│   │       ├── README.md              # Pipeline architecture, flow diagrams, step reference
│   │       ├── engine.py              # Ordered step executor (sequential + batched modes)
│   │       └── steps.py              # Chunk, Hash, Detect, Summarize, Embed, Store, Cleanup
│   ├── events/                        # Pydantic wire-format models
│   │   ├── input.py                   # Input, HistoryItem, ExternalConfig
│   │   ├── response.py               # Response, Source
│   │   ├── ingest_website.py          # IngestWebsite
│   │   └── ingest_space.py            # IngestBodyOfKnowledge
│   ├── config.py                      # Pydantic Settings (env var binding)
│   ├── container.py                   # IoC container (port -> adapter)
│   ├── registry.py                    # Plugin registry (import-based discovery)
│   ├── router.py                      # Content-based message router
│   ├── provider_factory.py            # LLM provider switching (Mistral/OpenAI/Anthropic)
│   ├── health.py                      # HTTP health server (/healthz, /readyz)
│   └── logging.py                     # JSON structured logging
│
├── plugins/                           # Plugin implementations (each has its own README.md)
│   ├── expert/                        # PromptGraph + knowledge retrieval RAG
│   ├── generic/                       # Direct LLM with history condensation
│   ├── guidance/                      # Multi-collection RAG with score filtering
│   ├── openai_assistant/              # OpenAI Assistants API (threads/runs/files)
│   ├── ingest_website/                # Web crawler + HTML parser + ingest
│   └── ingest_space/                  # Alkemio space tree reader + file parsers
│
├── evaluation/                        # RAG evaluation framework (RAGAS)
│   ├── cli.py                         # Click CLI (run, compare, generate, list)
│   ├── dataset.py                     # Golden test set I/O (JSONL)
│   ├── metrics.py                     # RAGAS metric configuration
│   ├── pipeline_invoker.py            # Direct plugin invocation for eval
│   ├── runner.py                      # Evaluation orchestration + scoring
│   ├── generator.py                   # Synthetic test case generation
│   └── report.py                      # Result reporting
│
├── tests/                             # Test suite (mirrors source layout)
│   ├── conftest.py                    # Mock ports, event factories
│   ├── core/                          # Core unit + contract tests
│   └── plugins/                       # Plugin unit tests
│
├── specs/                             # Feature specifications (001-025)
├── docs/adr/                          # Architecture Decision Records (0001–0013)
├── main.py                            # Single entry point
├── Dockerfile                         # Distroless build: 3.13-slim-trixie builder -> distroless/python3-debian13
├── docker/
│   ├── native-imports.py              # Derives native-extension modules from poetry.lock
│   └── image-verify.sh                # Persisted regression: structure, interpreter match, imports, 6-plugin matrix
├── docker-compose.yaml                # All services from one image
├── poetry.lock                        # Committed — lock-only installs, `poetry check --lock` gated
├── pyproject.toml                     # Poetry config + test/coverage settings
└── .github/workflows/                 # CI/CD (lint, test, build, deploy) — Python 3.13

Prerequisites

Python 3.13 (matches the distroless runtime interpreter; CI and the image builder both pin 3.13)
Poetry (dependency management)
Docker + Docker Compose (for local infrastructure or full deployment)

Getting Started

Install dependencies

poetry install

Configure environment

cp .env.example .env
# Edit .env with your API keys and service endpoints

Run a single plugin

PLUGIN_TYPE=generic poetry run python main.py

The process loads config from environment variables, discovers the plugin specified by PLUGIN_TYPE, wires adapter dependencies via the IoC container, and starts consuming from the configured RabbitMQ queue.

Run all plugins with Docker Compose

# Build the image
docker build -t alkemio/virtual-contributor .

# Start all plugins + RabbitMQ + ChromaDB
docker-compose up

Each plugin runs in its own container from the same image, differentiated only by the PLUGIN_TYPE environment variable.

Health checks

curl http://localhost:8080/healthz   # Liveness probe
curl http://localhost:8080/readyz    # Readiness probe (RabbitMQ + plugin status)

Configuration

All configuration is via environment variables (or .env file). See .env.example for a complete reference.

Core

Variable	Default	Description
`PLUGIN_TYPE`	`generic`	Plugin to run (`expert`, `generic`, `guidance`, `openai-assistant`, `ingest-website`, `ingest-space`)
`LOG_LEVEL`	`INFO`	Logging level
`HEALTH_PORT`	`8080`	Health server port

RabbitMQ

Variable	Default	Description
`RABBITMQ_HOST`	`localhost`	Broker hostname
`RABBITMQ_PORT`	`5672`	Broker port
`RABBITMQ_USER`	`alkemio-admin`	Username
`RABBITMQ_PASSWORD`	`alkemio!`	Password
`RABBITMQ_QUEUE`	`virtual-contributor-engine-{plugin}`	Input queue name
`RABBITMQ_EVENT_BUS_EXCHANGE`	`event-bus`	Exchange for publishing results
`RABBITMQ_RESULT_ROUTING_KEY`	`invoke-engine-result`	Routing key for results

Vector Database (ChromaDB)

Variable	Default	Description
`VECTOR_DB_HOST`	`localhost`	ChromaDB hostname
`VECTOR_DB_PORT`	`8765`	ChromaDB port
`VECTOR_DB_CREDENTIALS`	(empty)	Optional auth token

LLM

Variable	Default	Description
`LLM_PROVIDER`	`mistral`	Provider: `mistral`, `openai`, or `anthropic`
`LLM_API_KEY`	(required)	API key for the chosen provider
`LLM_MODEL`	Provider default	Model name (e.g. `mistral-large-latest`, `gpt-4o`, `claude-sonnet-4-6`)
`LLM_BASE_URL`	(empty)	Custom endpoint for local/self-hosted models
`LLM_TEMPERATURE`	(provider default)	Sampling temperature (0.0-2.0)
`LLM_MAX_TOKENS`	(provider default)	Maximum output tokens
`LLM_TOP_P`	(provider default)	Nucleus sampling (0.0-1.0)
`LLM_TIMEOUT`	`120`	Request timeout in seconds

Per-Plugin LLM Overrides

Any LLM setting can be overridden per plugin by prefixing with the plugin name in uppercase:

EXPERT_LLM_PROVIDER=anthropic
EXPERT_LLM_MODEL=claude-sonnet-4-6
EXPERT_LLM_API_KEY=sk-ant-...

Summarization LLM

A separate LLM can be configured for ingest pipeline summarization. All three fields are required to enable:

Variable	Description
`SUMMARIZE_LLM_PROVIDER`	Provider for summarization
`SUMMARIZE_LLM_MODEL`	Model name
`SUMMARIZE_LLM_API_KEY`	API key
`SUMMARIZE_LLM_TEMPERATURE`	Sampling temperature (optional)
`SUMMARIZE_LLM_TIMEOUT`	Request timeout (optional)

Retrieval Parameters

Variable	Default	Description
`EXPERT_N_RESULTS`	`5`	Number of chunks to retrieve (expert plugin)
`EXPERT_MIN_SCORE`	`0.3`	Minimum relevance score (expert plugin)
`EXPERT_HIERARCHICAL_RETRIEVAL_ENABLED`	`false`	Opt-in expert overview/summary route then branch-scoped detail retrieval
`EXPERT_HIERARCHY_MAX_BRANCHES`	`3`	Maximum relevant nearest branches selected (must be 2 or 3)
`EXPERT_HIERARCHY_DISPLAY_NAMES_ENABLED`	`false`	Separately opt-in, sanitized hierarchy display names in generation context
`GUIDANCE_N_RESULTS`	`5`	Number of chunks per collection (guidance plugin)
`GUIDANCE_MIN_SCORE`	`0.3`	Minimum relevance score (guidance plugin)
`MAX_CONTEXT_CHARS`	`20000`	Context budget — lowest-scoring chunks dropped first

Hierarchical expert retrieval

This expert-only enhancement is off by default. Each eligible request runs one dense Stage-1 query over overview/summary entries and uses no permanent collection-capability cache. Specific subspace routes dominate any root route; a root-only or unusable route takes the established flat path. A safe route runs the existing hybrid/re-rank/threshold/top-K pipeline over scoped detail.

A nonempty scoped result remains scoped even when short: there is no unscoped backfill. An empty scoped result or hierarchy-stage failure exits the hierarchy handler before exactly one flat attempt; an error from that flat attempt remains visible. Hierarchy labels, when disclosure is enabled, are sanitized and rendered before the context budget, so all model-visible label bytes are charged before eviction. Both hierarchy retrieval and display-name disclosure are independently controlled: setting hierarchy retrieval to false restores flat retrieval, while setting display names to false omits model-visible hierarchy names without disabling scoped retrieval.

Enable only after deployment with both controls off, re-ingestion of selected spaces, and the required human gates. RG-05 is the representative paired flat/on RAGAS evaluation on one reviewed query set; its runs require equal invariant fingerprints and present, recomputable mode-bound full fingerprints that differ between flat and hierarchical modes. Comparison fails closed when those conditions are not met. RG-05P is the separate provider-processing gate required before display names may be enabled. The repository's deterministic proxy is structural evidence only, while SC-009 is the full-suite regression gate. SUMMARIZE_ENABLED remains unchanged.

Re-ranking

Re-orders retrieved chunks before context assembly, blending vector similarity with lexical overlap against the query's own terms. Runs in-process: no network call, no extra dependency, ~0.7 ms for a top-20 re-rank.

Off by default — it changes what every answer is grounded in, so it is opted into rather than inherited. Turning it off restores prior behaviour exactly: the same number of results is requested and no re-ranking code runs.

Variable	Default	Description
`RERANK_ENABLED`	`false`	Enable the re-ranking stage
`RERANK_CANDIDATE_N`	`20`	Candidates fetched when enabled (must be ≥ `RERANK_TOP_K`)
`RERANK_TOP_K`	`5`	Candidates kept after re-ranking
`RERANK_LEXICAL_WEIGHT`	`0.6`	Lexical share of the blend; `0.0` provably reproduces vector order

Two rollbacks, coarse and fine: RERANK_ENABLED=false removes the stage entirely, and RERANK_LEXICAL_WEIGHT=0.0 leaves it wired but reproduces vector ordering exactly.

Keep the weight above 0.5. Scores are normalised across the candidate pool, so the best-vector candidate sits at 1.0 and the worst at 0.0; at exactly 0.5 a chunk that matches the query's wording but has the worst vector distance ties with the incumbent and loses — which is the case re-ranking exists to fix.

Re-ranking is a ranking signal only. Whether a chunk is relevant enough to use is still judged on its vector score, because the blended score is relative to the pool rather than absolute.

Adaptive Routing

Classifies each question and scales retrieval to it: small talk skips retrieval entirely, direct lookups retrieve narrowly, comparative questions retrieve more and get a wider context budget.

Classification is rule-based and in-process — no model, no network call, no new dependency, sub-millisecond. An LLM classifier was rejected on arithmetic: it would cost a round trip on every question including the simple ones it exists to speed up, needing (at 350ms) 39% of all traffic on the cheap path just to break even. That share has never been measured here.

Off by default. Disabled, both plugins take their existing code path with their existing constants.

Variable	Default	Description
`ROUTING_ENABLED`	`false`	Enable adaptive routing
`ROUTING_SIMPLE_N_RESULTS`	`3`	Width for direct lookups (must be ≤ the default width)
`ROUTING_COMPLEX_N_RESULTS`	`10`	Width for comparative / multi-hop questions
`ROUTING_COMPLEX_CONTEXT_CHARS`	`40000`	Context budget for those questions

The complex budget is widened alongside the width on purpose. At the deployed 9000-character chunk size, the default 20000-character budget already admits only two chunks — so widening retrieval without widening the budget would fetch more evidence and then discard it, producing a route that looks implemented and changes nothing.

The rules are deliberately lopsided. Only one mistake is user-visible — routing a real question to skip retrieval answers it ungrounded — so that is the only route with a hard gate: the whole message must match an anchored small-talk list, carry no question mark, and be at most six words. Everything else falls through to retrieval, so every misclassification degrades to "retrieve anyway".

Faithfulness validation

Logs a warning when an answer asserts something after retrieval returned nothing. That is the one case provably unsupportable without a model or citations: there was no evidence, so whatever was said came from elsewhere.

Variable	Default	Description
`FAITHFULNESS_VALIDATION_ENABLED`	`false`	Enable the check

Observation only. The answer a member receives is never changed, delayed, or withheld — enabling this adds a log line and nothing else. Runs in-process: no model, no network call, no dependency.

What it deliberately does not do. Checking whether the answer's words appear in the context was built, measured, and rejected: no threshold separates a fabrication from a faithful paraphrase, and because the platform answers in the member's language, a Dutch answer over English context scores the same as a lie. It would have flagged every non-English answer. So this checks context sufficiency, not content matching — which is why paraphrase and translation are structurally incapable of being flagged.

Honest limit. It does not catch a fabrication built on thin-but-nonempty context. That needs either citation verification (which requires the model to be asked to cite — a separate change) or a judge.

Citation verification is a string check against the context, so it can occupy this port directly. A judge cannot. validate() is synchronous by design and runs on the response path, so an implementation that calls a model would block the event loop for every message this worker is serving — not just its own. A judge belongs out of band: the port is the seam for recording the verdict, not for fetching it.

Embeddings

Variable	Default	Description
`EMBEDDINGS_API_KEY`	(required for ingest)	API key for embedding service
`EMBEDDINGS_ENDPOINT`	`https://api.scaleway.ai/v1`	Embedding API endpoint
`EMBEDDINGS_MODEL_NAME`	`qwen3-embedding-8b`	Embedding model
`EMBEDDINGS_QUERY_MAX_UTF8_BYTES`	`32768`	Logged query-input byte cap
`QUERY_REWRITE_MAX_UTF8_BYTES`	`4096`	Logged rewrite-input byte cap
`EMBEDDINGS_MAX_ATTEMPTS`	`3`	Logged outer query-attempt cap
`EMBEDDINGS_ATTEMPT_TIMEOUT_SECONDS`	`20`	Logged per-attempt query timeout
`EMBEDDINGS_TOTAL_DEADLINE_SECONDS`	`45`	Logged total query deadline

Ingest Pipeline

Variable	Default	Description
`CHUNK_SIZE`	`2000`	Characters per chunk
`CHUNK_OVERLAP`	`400`	Overlap between chunks
`BATCH_SIZE`	`20`	Embedding batch size
`SUMMARY_LENGTH`	`2500`	Max summary length
`SUMMARY_CHUNK_THRESHOLD`	`4`	Minimum chunks to trigger summarization
`PROCESS_PAGES_LIMIT`	`20`	Max pages to crawl (ingest-website)

Backward Compatibility

Legacy Mistral-specific variables (MISTRAL_API_KEY, MISTRAL_SMALL_MODEL_NAME) are still supported and used as fallbacks when the corresponding LLM_* variables are not set.

Plugins

Each plugin has its own README with detailed documentation, flow diagrams, and configuration reference.

Plugin	Type	Queue	README
Expert	`expert`	`virtual-contributor-engine-expert`	plugins/expert/README.md
Generic	`generic`	`virtual-contributor-engine-generic`	plugins/generic/README.md
Guidance	`guidance`	`virtual-contributor-engine-guidance`	plugins/guidance/README.md
OpenAI Assistant	`openai-assistant`	`virtual-contributor-engine-openai-assistant`	plugins/openai_assistant/README.md
Ingest Website	`ingest-website`	`virtual-contributor-ingest-website`	plugins/ingest_website/README.md
Ingest Space	`ingest-space`	`virtual-contributor-ingest-body-of-knowledge`	plugins/ingest_space/README.md

Expert (`PLUGIN_TYPE=expert`)

PromptGraph-based plugin with single-collection RAG. If the incoming event includes a prompt_graph definition, it compiles a LangGraph workflow from JSON — nodes have prompt templates, input variables, and optional output schemas. A special "retrieve" node queries the knowledge store. Falls back to simple RAG if no graph is defined. Results are filtered by score threshold and capped by context budget.

Generic (`PLUGIN_TYPE=generic`)

Direct LLM invocation with optional chat history condensation. When conversation history is present, it first condenses prior exchanges into a standalone question via the LLM, then builds a message array from system prompts and the condensed question. Returns raw LLM output with no sources.

Guidance (`PLUGIN_TYPE=guidance`)

Multi-collection RAG that queries three Alkemio knowledge bases in parallel (alkem.io-knowledge, welcome.alkem.io-knowledge, www.alkemio.org-knowledge). Merges results, deduplicates by source URL (keeping the highest score per page), filters by threshold, and enforces context budget. Context chunks are prefixed with [source:N] citations. Parses the LLM's JSON response to extract the answer.

OpenAI Assistant (`PLUGIN_TYPE=openai-assistant`)

Wraps the OpenAI Assistants API with thread and run management. Requires per-request external_config with api_key and assistant_id. Creates or resumes threads (via thread_id in external_metadata), adds messages, polls runs to completion, and returns the answer with thread ID for conversation continuity.

Ingest Website (`PLUGIN_TYPE=ingest-website`)

Web crawler that fetches pages from a base URL (configurable page limit), extracts text and titles from HTML via BeautifulSoup, then runs the full ingest pipeline. The collection is named {domain}-knowledge.

Ingest Space (`PLUGIN_TYPE=ingest-space`)

Fetches the Alkemio space tree via GraphQL, parses attached files (PDF, DOCX, XLSX), and runs the ingest pipeline with space-specific 2,500-character chunks. The collection is named {body_of_knowledge_id}-{purpose}.

Ingest Pipeline

Both ingest plugins share a common pipeline engine (core/domain/pipeline/) that executes an ordered sequence of steps. The engine supports both sequential and batched execution modes.

flowchart TD
    subgraph "Per-Batch Steps"
        A[Documents] --> B["ChunkStep<br/><i>Split via RecursiveCharacterTextSplitter</i>"]
        B --> C["ContentHashStep<br/><i>SHA-256 fingerprint per chunk</i>"]
        C --> D["ChangeDetectionStep<br/><i>Query store, mark unchanged, find orphans</i>"]
        D --> E["DocumentSummaryStep<br/><i>Per-document refine-pattern summary</i>"]
        E --> F["EmbedStep<br/><i>Batch embed via EmbeddingsPort</i>"]
        F --> G["StoreStep<br/><i>Persist to ChromaDB, skip unchanged</i>"]
        G --> H{More batches?}
    end

    H -->|Yes| A
    H -->|No| I[Finalize Steps]

    subgraph "Finalize Steps"
        I --> J["BodyOfKnowledgeSummaryStep<br/><i>Single overview summary</i>"]
        J --> K["OrphanCleanupStep ⚠️<br/><i>Delete orphaned + removed chunks</i>"]
        K --> L[IngestResult]
    end

Content-hash deduplication: Each chunk gets a SHA-256 fingerprint computed from its content, title, source, type, and document ID. During change detection, existing chunks in the store are compared by hash — unchanged chunks skip re-embedding, saving compute. Orphaned chunks (from deleted or modified documents) are cleaned up after the store step succeeds.

Safety: Steps marked as destructive (e.g., OrphanCleanupStep) are automatically skipped when prior pipeline errors exist, preventing data loss from partial processing.

Each step reports metrics (duration, items in/out, error count) and failures are caught per-step so the pipeline can continue with remaining steps. See core/domain/pipeline/README.md for the full step reference and context field documentation.

Docker Deployment

Image

The project ships a distroless runtime image (workspace#026-distroless-runtime-images), built as a matched Python 3.13 pair:

Builder stage (python:3.13.<patch>-slim-trixie, glibc/Debian, digest-pinned): installs Poetry, verifies the committed poetry.lock is in sync with pyproject.toml (poetry check --lock — the build fails on a stale lock), and installs production dependencies exclusively from the lock into a relocatable /venv (POETRY_VIRTUALENVS_CREATE=false against an already-activated venv). The exact interpreter patch version is recorded into /venv/PYTHON_VERSION for the runtime match check below.
Runtime stage (gcr.io/distroless/python3-debian13:nonroot, digest-pinned): contains only /venv, core/, plugins/, and main.py — no shell, no package manager, no Poetry/pip. Runs as UID 65532 (nonroot). PYTHONPATH=/venv/lib/python3.13/site-packages exposes the relocated dependencies; CMD ["main.py"] runs against the distroless image's own python3 entrypoint.
The builder's and runtime's Python interpreters MUST match at major.minor (3.13 == 3.13) — the compiled-extension wheels installed by the builder must be ABI-compatible with the runtime interpreter. Verified mechanically by docker/image-verify.sh.
Exposes port 8080 for the health server (HEALTH_PORT, /healthz, /readyz — see Startup Sequence).
Default PLUGIN_TYPE=generic, overridable at runtime.

docker build -t alkemio/virtual-contributor .

# Persisted regression (US3 contract `vc-runtime-image`): structure, interpreter
# match, every native-extension import from poetry.lock, and a live six-plugin
# start matrix (expert, generic, guidance, openai-assistant, ingest_website,
# ingest_space) against RabbitMQ + ChromaDB.
docker/image-verify.sh alkemio/virtual-contributor

poetry.lock is committed (not gitignored) — dependencies are installed only from the lock, both locally (poetry install) and in the image build. poetry check --lock is a required gate; run poetry lock and re-commit the lock whenever pyproject.toml changes.

Docker Compose

The docker-compose.yaml runs the full stack from a single image:

Infrastructure:

RabbitMQ (AMQP :5672, management UI :15672)
ChromaDB (:8765)

Engine plugins (each a separate container):

expert, generic, guidance, openai-assistant

Ingest plugins (each a separate container):

ingest-website, ingest-space

Each service sets its own PLUGIN_TYPE and queue configuration. All services share the same RabbitMQ and ChromaDB instances.

docker-compose up        # Start everything
docker-compose up expert # Start just the expert plugin + dependencies

RAG Evaluation Framework

The evaluation/ module provides systematic RAG quality measurement using RAGAS.

Metrics

Metric	What it Measures
Faithfulness	Is the answer grounded in the retrieved context?
Answer Relevancy	Does the answer address the question?
Context Precision	Are the retrieved chunks relevant to the question?
Context Recall	Are all necessary chunks retrieved?

Usage

# Run evaluation against the guidance plugin
poetry run python -m evaluation.cli run --plugin guidance --test-set evaluation/data/test_set.jsonl

# Compare two evaluation runs
poetry run python -m evaluation.cli compare --baseline <run-id-1> --current <run-id-2>

# Generate synthetic test cases from a ChromaDB collection
poetry run python -m evaluation.cli generate --collection alkem.io-knowledge --output evaluation/data/test_set.jsonl

# List previous evaluation runs
poetry run python -m evaluation.cli list

How it Works

Golden test set: JSONL file with question, expected_answer, and relevant_documents fields
Pipeline invoker: Instantiates the plugin directly (bypassing RabbitMQ); for Expert, it shares production composition and captures the exact final rendered generation contexts after retrieval, ranking, filtering, and budget processing
Scorer: Wraps RAGAS metrics, uses the pipeline's own LLM as judge via LangchainLLMWrapper
Runner: Executes the test suite, computes per-metric aggregates, persists results to evaluations/{id}.json

Paired Expert comparisons require equal non-secret invariant-composition fingerprints. Both full fingerprints must be present, recomputable from the invariant identity and their explicit modes, and different for flat versus hierarchical runs; otherwise comparison fails closed rather than treating a configuration change as a hierarchy-only experiment.

Testing

Tests use asyncio_mode = "auto" — async test functions run automatically without @pytest.mark.asyncio.

# Run all tests
poetry run pytest

# With coverage report
poetry run pytest --cov=core --cov=plugins --cov-report=term-missing

# Single file
poetry run pytest tests/plugins/test_expert.py

# Single test
poetry run pytest tests/plugins/test_expert.py::test_handle

For this hierarchy feature, the full suite is the SC-009 regression gate; focused tests and the deterministic precision proxy do not replace the representative paired RAGAS gate (RG-05).

Test Infrastructure

Mock ports (tests/conftest.py): MockLLMPort, MockEmbeddingsPort, MockKnowledgeStorePort — in-memory implementations with call tracking
Event factories: make_input(...), make_ingest_website(...) — build test events with sensible defaults
Coverage: Minimum 80% enforced in CI; adapters, config, health, logging, and main.py are excluded from coverage

Testing Pattern

async def test_expert_handle():
    llm = MockLLMPort(responses=["The answer is 42."])
    store = MockKnowledgeStorePort(results=[...])
    plugin = ExpertPlugin(llm=llm, knowledge_store=store)

    response = await plugin.handle(make_input(message="What is the answer?"))

    assert "42" in response.result
    assert len(store.query_calls) == 1

Linting and Type Checking

# Lint with Ruff
poetry run ruff check core/ plugins/ tests/

# Type check with Pyright
poetry run pyright core/ plugins/

Both checks are enforced in CI.

Adding a New Plugin

Create plugins/{name}/plugin.py with a class implementing the PluginContract:

from core.events.input import Input
from core.events.response import Response
from core.ports.llm import LLMPort


class MyPlugin:
    name = "my-plugin"
    event_type = Input

    def __init__(self, llm: LLMPort):
        self._llm = llm

    async def startup(self) -> None:
        pass

    async def shutdown(self) -> None:
        pass

    async def handle(self, event: Input) -> Response:
        result = await self._llm.invoke([{"role": "user", "content": event.message}])
        return Response(result=result)

Port dependencies declared as __init__ type hints are resolved automatically by the IoC container
Add tests in tests/plugins/test_{name}.py
Run with PLUGIN_TYPE={name} poetry run python main.py

No core code changes required. See docs/adr/0003-plugin-contract-design.md for the full contract specification.

Architecture Decision Records

Detailed design rationale is documented in docs/adr/:

ADR	Decision
0001	Microkernel + hexagonal architecture consolidating 7 repositories
0002	TypeScript-to-Python port for ingest-space plugin
0003	Duck-typed plugin contract via Python Protocol
0004	Sequential message processing with horizontal scaling
0005	Unified LangChain adapter with provider factory
0006	Content-hash deduplication and KnowledgeStorePort extension
0007	RAGAS as RAG evaluation framework
0008	Composable step-based ingest pipeline engine
0009	Configurable summarization LLM with per-plugin retrieval parameters
0010	Pipeline destructive step safety via duck-typed gating
0011	Pipeline reliability — thread pool sizing, partial failure resilience
0012	Module-local OpenTelemetry pipeline tracing

Tracing & Observability

Pipeline tracing ships dark. It emits OTLP/HTTP traces only when an explicit internal endpoint is configured; ambient OTEL_* exporter and sampler settings are never used.

Variable	Default	Purpose
`TRACING_ENABLED`	`false`	Enable tracing only with an endpoint
`TRACING_OTLP_ENDPOINT`	unset	Full internal OTLP/HTTP trace URL
`TRACING_OTLP_HEADERS`	unset	Comma-separated `key=value` authentication headers
`TRACING_SERVICE_NAME`	plugin-derived	Service resource name
`TRACING_SAMPLE_RATIO`	`1.0`	Trace sampling ratio, from 0 through 1
`TRACING_CAPTURE_CONTENT`	`false`	Off by default. When `true`, exports bounded member messages, prompts, and model completions to the trace collector as span attributes — enable only as a recorded, deliberate decision.
`TRACING_CONTENT_MAX_CHARS`	`1000`	Maximum characters per content attribute

For local self-hosted Langfuse, point the service at its OTLP endpoint:

export TRACING_ENABLED=true
export TRACING_OTLP_ENDPOINT="http://localhost:3000/api/public/otel/v1/traces"
export TRACING_OTLP_HEADERS="Authorization=Basic ${AUTH}"
PLUGIN_TYPE=guidance poetry run python main.py

The same explicit endpoint/header configuration works with self-hosted Grafana Tempo or Elastic APM OTLP receivers. Never set an endpoint you do not control: trace content can include user messages and retrieved context. | 0013 | Grounded, citable generation prompts |

Feature Specifications

Detailed design specifications for all features are in specs/:

#	Spec	Status
001	Unified Microkernel Engine	Draft
002	Multi-Provider LLM Support	Draft
003	Async Performance Optimizations	In Progress
004	Composable Ingest Pipeline Engine	Implemented
005	Document Processing Reliability	Implemented
006	Content-Hash Deduplication	Draft
007	Configurable Summarization LLM	Draft
008	Early ACK with Async Processing (PRD)	Draft
009	Configurable Vector DB Distance Function	Implemented
010	BoK LLM & Factory Hardening	Implemented
011	Pipeline Destructive Step Safety	Implemented
012	Empty Corpus Re-Ingestion	Implemented
013	Summary Lifecycle Management	Implemented
014	Concurrent Document Summarization	Implemented
015	Early ACK with Async Processing	In Progress
016	Skip Unchanged Upsert	Implemented
017	Incremental Embedding	Implemented
018	Consistent Summarization	Implemented
019	Batched Ingest Pipeline	Draft
020	Pipeline Reliability & BoK Resilience	Implemented
021	Website Content Quality	Implemented
022	Space Ingest Context & URI Tracking	Implemented
023	PromptGraph Robustness	Implemented
024	Retry Error Reporting	Implemented
025	RAG Evaluation Framework	Draft

CI/CD

Workflow	Trigger	What it Does
`ci.yml`	Push/PR	Ruff lint, Pyright type check, pytest with 80% coverage minimum
`build.yml`	Push to `develop`/`main`, tags	Docker build and push with branch/semver/SHA tags
`deploy-dev.yml`	After successful build on `develop`	Deploys all 6 plugins to alkemio-dev Kubernetes namespace

License

EUPL-1.2 (European Union Public Licence)

Name		Name	Last commit message	Last commit date
.claude		.claude
.github/workflows		.github/workflows
.specify		.specify
core		core
docker		docker
docs		docs
evaluation		evaluation
evaluations		evaluations
plugins		plugins
scripts		scripts
specs		specs
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.flake8		.flake8
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
README.md		README.md
docker-compose.yaml		docker-compose.yaml
main.py		main.py
poetry.lock		poetry.lock
pyproject.toml		pyproject.toml
pyrightconfig.json		pyrightconfig.json
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

Alkemio Virtual Contributor

Table of Contents

Features

Architecture

Ports (Interfaces)

Adapters (Implementations)

Message Flow

Startup Sequence

Repository Structure

Prerequisites

Getting Started

Install dependencies

Configure environment

Run a single plugin

Run all plugins with Docker Compose

Health checks

Configuration

Core

RabbitMQ

Vector Database (ChromaDB)

LLM

Per-Plugin LLM Overrides

Summarization LLM

Retrieval Parameters

Hierarchical expert retrieval

Re-ranking

Adaptive Routing

Faithfulness validation

Embeddings

Ingest Pipeline

Backward Compatibility

Plugins

Expert (PLUGIN_TYPE=expert)

Generic (PLUGIN_TYPE=generic)

Guidance (PLUGIN_TYPE=guidance)

OpenAI Assistant (PLUGIN_TYPE=openai-assistant)

Ingest Website (PLUGIN_TYPE=ingest-website)

Ingest Space (PLUGIN_TYPE=ingest-space)

Ingest Pipeline

Docker Deployment

Image

Docker Compose

RAG Evaluation Framework

Metrics

Usage

How it Works

Testing

Test Infrastructure

Testing Pattern

Linting and Type Checking

Adding a New Plugin

Architecture Decision Records

Tracing & Observability

Feature Specifications

CI/CD

License

About

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Expert (`PLUGIN_TYPE=expert`)

Generic (`PLUGIN_TYPE=generic`)

Guidance (`PLUGIN_TYPE=guidance`)

OpenAI Assistant (`PLUGIN_TYPE=openai-assistant`)

Ingest Website (`PLUGIN_TYPE=ingest-website`)

Ingest Space (`PLUGIN_TYPE=ingest-space`)