Skip to content

multi-agent.md

Latest commit

 

History

History
414 lines (317 loc) · 14.4 KB

multi-agent.md

File metadata and controls

414 lines (317 loc) · 14.4 KB
Raw

Multi-agent orchestration

Durable mission workflows — scope document in, delegator decomposes a DAG, parallel Temporal child workflows execute scoped tasks, Postgres tracks progress, shared RAG holds cross-agent knowledge.

Related: agent-roadmap.md · agent-skills.md · agent-run-logs.md · memory/README.md

North star

Feed a well-defined scope document. A delegator (orchestrator) draws the dependency graph and spins subagents as durable Temporal workflows. Graph + state track information flow between subagents. Every subagent reads and writes shared memory for mission context.

Design first, then execute: only create an edge when step B actually reads step A's output. Independent work runs in parallel. Consolidation fans in only where dependencies exist.

Two graph layers (do not conflate)

flowchart TB
  subgraph mission["Mission layer — Temporal orchestrator"]
    SCOPE[Scope document]
    MW[MissionWorkflow delegator]
    DEC[decompose → DAG]
    C1[Child workflow t1]
    C2[Child workflow t2]
    C3[Child workflow t3]
    FAN[consolidate fan-in]
    SCOPE --> MW --> DEC
    DEC --> C1 & C2 & C3
    C1 & C2 & C3 --> FAN
  end

  subgraph child["Child layer — LangGraph per subagent"]
    R[retrieve_context] --> PR[propose_edits]
    PR --> AP[apply_edits]
    AP --> WO[write_observation]
  end

  C1 --> child
  PG[(Postgres board)]
  MEM[(Shared RAG)]
  MW --> PG
  C1 & C2 & C3 --> PG
  C1 & C2 & C3 --> MEM
  FAN --> MEM
Loading
Layer Runtime Owns Target shape
Mission Temporal MissionWorkflow Decompose, dispatch, wait, fan-in, synthesis Orchestrator draws graph, does not execute work
Child LangGraph inside AgentRunWorkflow Plan/edit/observe for one scoped task Short role-specific graph (builder / reviewer)
Board Postgres Task status, DAG edges, outputs Source of truth for progress + inter-agent data
Memory sqlite-vec / pgvector Seed knowledge + observations Cross-agent context (slow path)

Graph engineering rules

From the dependency-first model (nodes = work units, edges = real data dependencies):

The edge test

For every "and then" in a workflow, ask:

Does the next step actually read the previous step's output?

Answer Action
Yes Real edge — keep sequential order (depends_on / blocked_by)
No No edge — run in parallel

Orchestrator responsibilities

  1. Input: scope document (not a one-liner).
  2. Output: { nodes, edges } validated and stored in Postgres.
  3. Execute: fan-out independent nodes; start dependent nodes only when deps complete.
  4. Fan-in: consolidate layers (batch → summarize → final report if N is large).
  5. Never: edit repo files directly — children own execution.

Failure modes to design for

Failure Mitigation
Context collapse Layered fan-in (batch 20–50 outputs, summarize, then consolidate summaries)
False independence Audit shared resources (same file, same API) — disjoint paths[] today; rate limits later
Silent node failure Fan-in checks completed_count == expected_count; flag gaps in synthesis

Inter-agent information flow (two channels)

Channel Speed Use
Postgres task_outputs (F2+) Fast, explicit Parent passes prior child JSON to dependents
Shared RAG Eventually consistent Mission-scoped retrieve; observations tagged [mission:id][task:id]

Target scope document

F1 accepts a string; target input:

{
  "title": "Audit route files and update docs",
  "scope": "Check auth middleware on all API routes; update local-dev if commands drift.",
  "acceptance_criteria": [
    "Every route file audited",
    "Consolidated report by severity",
    "docs/local-dev.md matches worker commands"
  ],
  "constraints": ["no new dependencies", "paths under docs/ and policies/ only"],
  "resources": ["docs/multi-agent.md", "temporal/graph/graph.py"]
}

Decompose reads this and returns a DAG:

{
  "nodes": [
    {
      "task_id": "audit_routes",
      "role": "builder",
      "goal": "List and audit API route files for auth middleware",
      "paths": ["docs/audit-routes.md"],
      "depends_on": []
    },
    {
      "task_id": "update_local_dev",
      "role": "builder",
      "goal": "Align docs/local-dev.md with current worker commands",
      "paths": ["docs/local-dev.md"],
      "depends_on": []
    },
    {
      "task_id": "consolidate",
      "role": "synthesizer",
      "goal": "Merge audit findings into one severity-grouped report",
      "paths": ["docs/audit-report.md"],
      "depends_on": ["audit_routes", "update_local_dev"]
    }
  ]
}

audit_routes and update_local_dev have no edge between them — parallel. consolidate has real edges — waits for both.

Target runtime shape

                    Scope document
                          │
                          ▼
              MissionWorkflow (delegator)
                          │
         decompose → DAG in Postgres (nodes + depends_on)
                          │
        ┌─────────────────┼─────────────────┐
        ▼                 ▼                 ▼
   Child workflow    Child workflow    Child workflow   ← parallel layer
   (builder graph)   (builder graph)   (builder graph)
        │                 │                 │
        └──── task_outputs + shared memory ─┘
                          │
              fan-in (deps satisfied only)
                          │
              consolidate (LLM, layered if N > 50)
                          │
              final observation + mission synthesis

Temporal = orchestrator runtime (parallel dispatch, durability, retries).
Postgres = graph structure + progress + explicit task outputs.
LangGraph = reasoning inside each child (short graph per role).
RAG = shared knowledge — not a substitute for explicit edges.

Current implementation vs target

Mission layer (F1 shipped)

Capability Status Notes
Delegator workflow ✅ MissionWorkflow
Decompose to tasks ✅ LLM → flat task list
Parallel child workflows ✅ Up to MAX_PARALLEL_AGENTS
Postgres board ✅ agent_missions, agent_mission_tasks
Path-scoped edits ✅ Disjoint paths[] per task
Mission-tagged memory ✅ [mission:id][task:id] observations
Scope document input ❌ String only today
DAG / depends_on execution ❌ Schema has blocked_by; F1 runs all parallel
task_outputs between children ❌ F2
LLM consolidation fan-in ❌ Text rollup only
Expected-count failure check ❌ F2

Child layer (repo-agent graph)

Current chain:

draft_plan → retrieve_context → refine_plan → prepare_edits
→ propose_edits → apply_edits → write_observation
Transition Real edge? Target
draft_plan → retrieve_context Weak — retrieve uses goal, not draft Run in parallel or drop draft for scoped tasks
retrieve_context → refine_plan Yes Keep
refine_plan → edits Yes Keep
Full chain on every subagent Wasteful when mission already decomposed doc-agent: retrieve → propose → apply → observe

State durability

Store Holds Shared across subagents?
AgentState + InMemorySaver Per-child LangGraph ❌ Per run only
Postgres mission board Status, paths, artifacts ✅ Progress
RAG memory Observations + seed ⚠️ Read at child start; not guaranteed mid-mission

Principles

Layer Owns
Temporal Orchestration — fan-out, retries, durability, failure policy
LangGraph Reasoning inside each child
Postgres Mission DAG, task status, task outputs (F2+)
RAG Long-lived knowledge + mission observations
Guardrails Path allowlist, content checks, task path scope

Postgres mission board

Use existing Postgres in namespace lunar (deploy/postgres/postgres.yaml, pgvector/pgvector:pg16). Same DATABASE_URL as pgvector memory when MEMORY_BACKEND=postgres.

kubectl port-forward svc/postgres -n lunar 5432:5432
export DATABASE_URL=postgresql://lunar:PASSWORD@localhost:5432/lunar

Local dev: worker (Terminal B) needs DATABASE_URL and OPENAI_API_KEY — exporting in the start-mission terminal does not affect the worker process. See local-dev.md.

Run logs stay in gitignored local/agent-runs/ for chat review; Postgres is source of truth for mission state.

Schema (F1 + planned F2 columns)

Apply: kubectl apply -f deploy/postgres/mission-schema-init.yaml

-- F1 (shipped)
CREATE TABLE IF NOT EXISTS agent_missions (
    mission_id     TEXT PRIMARY KEY,
    goal           TEXT NOT NULL,
    status         TEXT NOT NULL DEFAULT 'pending',
    plan           JSONB NOT NULL DEFAULT '[]',
    synthesis      TEXT,
    error          TEXT,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE IF NOT EXISTS agent_mission_tasks (
    task_id        TEXT NOT NULL,
    mission_id     TEXT NOT NULL REFERENCES agent_missions(mission_id) ON DELETE CASCADE,
    role           TEXT NOT NULL DEFAULT 'builder',
    goal           TEXT NOT NULL,
    status         TEXT NOT NULL DEFAULT 'pending',
    workflow_id    TEXT,
    paths          JSONB NOT NULL DEFAULT '[]',
    blocked_by     JSONB NOT NULL DEFAULT '[]',  -- depends_on edges
    artifacts      JSONB NOT NULL DEFAULT '[]',
    error          TEXT,
    sort_order     INT NOT NULL DEFAULT 0,
    started_at     TIMESTAMPTZ,
    completed_at   TIMESTAMPTZ,
    updated_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (mission_id, task_id)
);

F2 planned: scope JSONB on agent_missions; task_outputs JSONB on agent_mission_tasks.

Agent roles & skills

Role When Graph Skills
Decomposer Once per mission Activity (LLM) caveman
Builder Scoped tasks repo-agent → doc-agent (F3) ponytail
Reviewer Before synthesis (F3) review-agent ponytail-review pattern
Synthesizer Fan-in node Activity (LLM) caveman

See agent-skills.md.

Conflict prevention

  1. Each builder task gets explicit paths[] (subset of allowlist).
  2. No path in two tasks for the same mission.
  3. Single-file scope → one builder task.
  4. Reviewer tasks: read-only.

Phased rollout

F1 — Mission shell + Postgres + parallel children ✅

Deliverable Status
MissionWorkflow, decompose_mission, coordination activities ✅
Parallel child AgentRunWorkflows with mission context ✅
Path-scoped guardrails ✅
start-mission.sh, schema init Job ✅
Tests (test_mission_tasks.py, etc.) ✅

Run locally

# A — Temporal
temporal server start-dev --db-filename /tmp/temporal-training.db

# B — worker (env must be HERE, not in C)
kubectl port-forward svc/postgres -n lunar 5432:5432
export DATABASE_URL=postgresql://lunar:PASSWORD@localhost:5432/lunar
export OPENAI_API_KEY=$(kubectl get secret openai-api-key -n lunar \
  -o jsonpath='{.data.OPENAI_API_KEY}' | base64 -d)
kubectl apply -f deploy/postgres/mission-schema-init.yaml
cd temporal && uv sync --extra local && uv run worker.py

# C — mission
./scripts/temporal/start-mission.sh --wait --scope-file docs/examples/mission-scope.json
./scripts/temporal/show-mission.sh mission-<id>

# D — dashboard
./scripts/visibility/start-dashboard.sh   # http://localhost:8765

Keep K8s temporal-worker at 0 replicas while iterating locally.

F2 — DAG execution + information flow ✅

Deliverable Status
Scope document input (--scope-file, JSON dict) ✅
Execute blocked_by / wave scheduler ✅
task_outputs JSONB + upstream pass to children ✅
Mission-scoped RAG retrieve ✅
LLM consolidate + expected-count check ✅
show-mission.sh ✅
Agent visibility dashboard ✅

Example scope: docs/examples/mission-scope.json

F3 — Role-specific child graphs

Deliverable Purpose
doc-agent retrieve → propose → apply (no double-plan)
review-agent Read-only critique before synthesis
Role drives graph + env task.role in Postgres

F4 — K8s worker always-on (deferred)

Wire DATABASE_URL + secrets on pod; rebuild + rollout restart as update loop. See agent-roadmap.md. Deferred while iterating on graph design locally.

Implementation priority (next)

# Change Why
1 F3: doc-agent slim graph Stop double-planning inside scoped doc tasks
2 F3: review-agent read-only pass Validate against acceptance criteria before fan-in
3 F3: Role → graph routing in child workflow task.role picks graph, not always repo-agent
4 F4: K8s worker always-on When local iteration slows down

Configuration

Variable Default Purpose
DATABASE_URL — Required for missions
MEMORY_BACKEND sqlite Use postgres for concurrent multi-agent
MAX_PARALLEL_AGENTS 3 Parent concurrency cap
MAX_MISSION_TASKS 3 Decompose cap (F1)
AGENT_PONYTAIL / AGENT_CAVEMAN full / lite Per agent-skills.md

Revision log

Date Change
2026-08-23 F1 implemented — MissionWorkflow, Postgres board, start-mission.sh
2026-08-24 Graph engineering target — two layers, edge test, scope doc, fan-in, F2+ priorities
2026-08-24 F2 — scope doc, DAG scheduler, task_outputs, mission RAG, LLM consolidate (live-tested)
2026-08-24 Agent visibility dashboard — live step timeline at :8765