Skip to content

api.md

Latest commit

 

History

History
913 lines (727 loc) · 48.3 KB

api.md

File metadata and controls

913 lines (727 loc) · 48.3 KB
Raw

Mantis CUA HTTP API

Reference for callers who want to use the Mantis CUA service directly — without going through a host wrapper. For library-shaped integrations where you drive MicroPlanRunner in your own process, see Embedding MicroPlanRunner and the any-agent integration playbook.

Endpoints

Path Auth Purpose
POST /v1/predict X-Mantis-Token (run scope) Run a plan, poll status, fetch result. The high-level orchestrator.
POST /predict X-Mantis-Token (run scope) Backwards-compat alias for /v1/predict. Identical behavior.
POST /v1/cua X-Mantis-Token (run scope) Pure CUA pass-through — Mantis as a thin Holo3 driver. No decomposition; Claude assist off by default (opt-in ground_clicks, loop-only director). See Pure CUA mode.
POST /v1/chat/completions X-Mantis-Token (run scope) OpenAI-compat reverse proxy to in-pod Holo3 (raw inference).
GET /v1/models open OpenAI-compat model list. Returns holo3.
GET /v1/health, GET /health open Liveness/readiness probe.
GET /v1/version open Runtime version snapshot — version, model, ready, git_sha, build_time. Useful for pinning client behavior to a specific build.
GET /metrics open Prometheus scrape endpoint. Returns 503 if prometheus_client not installed.
GET /v1/runs/{run_id} X-Mantis-Token Cheap-poll lifecycle (#806) — phase + adaptive polling_backoff_ms_hint. Use instead of action=status for active polling loops.
GET /v1/runs/{run_id}/status X-Mantis-Token Full-detail status (alias for action=status).
GET /v1/runs/{run_id}/result X-Mantis-Token Result payload once terminal (alias for action=result).
POST /v1/runs/{run_id}/cancel X-Mantis-Token Cancel a run (alias for action=cancel).
GET /v1/runs/{run_id}/events X-Mantis-Token SSE event stream (#808) with ?sse=true. JSON parity with action=reasoning_trace otherwise.
GET /v1/queue X-Mantis-Token Per-tenant queue snapshot — counts of queued / running / recovering runs.
POST /v1/recipes X-Mantis-Token Runtime recipe registration (#809) — {name, schema: ExtractionSchema}. Tenant-scoped.
GET /v1/recipes X-Mantis-Token List runtime recipes registered under the caller's tenant.
GET /v1/recipes/{name} X-Mantis-Token Fetch a runtime recipe by name.
DELETE /v1/recipes/{name} X-Mantis-Token Delete a runtime recipe (idempotent).
GET /v1/runs/{run_id}/video X-Mantis-Token Download the screencast captured during a run. Returns 404 if record_video was not requested.
GET /v1/runs/{run_id}/artifacts/{name} X-Mantis-Token Download a run artifact (#508). Allowlisted names: leads.csv, extracted_rows.csv, extracted_rows.json, result.json. Returns 404 when the artifact wasn't produced (no leads, no structured rows).
GET /docs, GET /redoc open Interactive Swagger UI / Redoc viewer over /openapi.json. Disable on production tenant fleets with MANTIS_ENABLE_DOCS_UI=0.
GET /openapi.json open Machine-readable OpenAPI spec. Always served, even when the interactive UIs are disabled — this is what client SDKs and IDE plugins consume.

When deployed behind Baseten, all requests must also carry Authorization: Api-Key <BASETEN_API_KEY> (gateway auth, separate from container auth).

Authentication

The service uses two layers of auth when deployed on Baseten:

Header Layer Purpose
Authorization: Api-Key <BASETEN_API_KEY> Baseten gateway Authenticates the platform request. Required for any call.
X-Mantis-Token: <tenant_token> Container Authenticates the tenant. Required for /v1/predict and /v1/chat/completions.

X-Mantis-Token is split into a custom header (rather than another Authorization: Bearer) because the Baseten gateway's Authorization: Api-Key header is forwarded to the container; using the same header for both auth layers would clash.

If MANTIS_TENANT_KEYS_PATH is configured on the deployment, each tenant has its own token. Otherwise a single MANTIS_API_TOKEN works for all callers (single-tenant mode).

Rate / scale caps

Per-request server-side caps that the caller cannot exceed:

Env var Default Effect
MANTIS_MAX_STEPS_PER_PLAN 200 Plans larger than this are rejected with 400.
MANTIS_MAX_LOOP_ITERATIONS 50 loop_count in any loop step is silently clamped to this.
MANTIS_MAX_RUNTIME_MINUTES 60 max_time_minutes in the request body is clamped.
MANTIS_MAX_COST_USD 25.0 max_cost in the request body is clamped.

Plus per-tenant caps when multi-tenant is enabled (max_concurrent_runs, max_cost_per_run, max_time_minutes_per_run).

POST /v1/predict

Run a plan, poll an existing run, fetch the result, or fetch live logs. The mode is determined by the action field (or its absence).

Run a new plan

The request body must contain exactly one of these plan-shape fields, in priority order:

Field Type Description
task_suite object Inline task-suite dict. Use this for arbitrary plans where you don't want to bake them into the container image.
task_file_contents string JSON-as-string. Same shape as task_suite but pre-serialized.
task_file string Path inside the container image (e.g. tasks/crm/crm_tasks.json).
micro string Path to a micro-plan JSON or plain-text plan inside the image (e.g. plans/example/extract_listings.json).
plan_text string Inline plain-English plan. Decomposed via Claude on the server side.

Plus the run options:

Field Default Description
detached true Return a run_id immediately and continue work in the background. Set false to block until done (only useful for short plans — 5–10s).
profile_id "default" (#341) Chrome user-data-dir identity. Server prefixes with tenant_id. Sticky across plan revisions — same id ⇒ same cookies / logged-in sessions (reuse a login by passing the same id; no resume_state needed). Two concurrent runs on one profile_id → 409 (Chrome user-data-dir lock); distinct ids run in parallel. Recommended convention <user>:<platform>. See Profiles & login reuse.
workflow_id plan_signature[:12] (#341) Checkpoint identity. Server prefixes with tenant_id. Rotate when the plan definition changes; pair with resume_state to pick up where the last run with this id stopped.
state_key "" Legacy single-field identity. When set alone, the server routes it to both profile_id and workflow_id (back-compat). Prefer the split fields above in new code; see #341.
resume_state false Reconstruct browser state from the latest checkpoint at workflow_id before starting.
max_cost 25.0 Cap in USD; clamped against the tenant cap.
max_time_minutes 60 Wall-clock cap; clamped against the tenant cap.
proxy_city, proxy_state unset Optional IPRoyal geo overrides. Subject to allowlist.
record_video false If true, captures the Xvfb display while the run executes and saves a screencast under the per-tenant run dir. Fetch via GET /v1/runs/{run_id}/video.
video_format "mp4" One of mp4, webm, gif.
video_fps 5 Capture rate; clamped to [1, 30]. Higher fps = larger file + more CPU.
live_viewer false (#416) Stand up an MJPEG tunnel onto the Xvfb display and surface its URL as viewer_url on action=status. Open the URL in a browser to watch the run live. Currently only the holo3 executor wires this through.

The following go inside task_suite (not top-level), alongside the plan:

Suite field Description
_challenger_model (#918) Serve this full merged-GGUF model in place of the base (the promotion-gate challenger via a -m swap, base --mmproj reused). This is the working path for the holo3 (qwen3_5_moe) base, whose LoRA adapter can't be GGUF-converted but whose merged model can. A "<volume>:/path/merged.gguf" ref. llama.cpp bases only; mutually exclusive with _lora_adapter.
_lora_adapter (#911) Serve base + this LoRA adapter (an overlay, not a full swap). A ref "<volume>:/checkpoints/<algo>" or a mounted path. For llama.cpp bases a pre-converted .gguf adapter (does not work for the holo3 MoE arch — use _challenger_model); vLLM bases (fara) serve the PEFT dir directly. Omit (with _challenger_model) to serve the base. Modal only — on Baseten the challenger is a deployment-level env (MANTIS_LORA_ADAPTER / MANTIS_HOLO3_GGUF), see Baseten hosting.
_lora_name vLLM only — served-model-name for the adapter (default challenger).
_lora_scale llama.cpp only — adapter scale (default 1.0; emits --lora-scaled).

Detached response

{
  "status": "queued",
  "created_at": "2026-04-28T01:57:08.316Z",
  "model": "holo3",
  "mode": "detached",
  "run_id": "20260428_021432_076255ef",
  "payload": { ... echoed input ... },
  "updated_at": "2026-04-28T01:57:08.317Z",
  "status_path":  "/workspace/mantis-data/runs/<run_id>/status.json",
  "result_path":  "/workspace/mantis-data/runs/<run_id>/result.json",
  "csv_path":     "/workspace/mantis-data/runs/<run_id>/leads.csv",
  "events_path":  "/workspace/mantis-data/runs/<run_id>/events.log"
}

The *_path fields are server-internal; you fetch them through the polling actions (next section).

Poll / fetch / cancel an existing run

Set action and run_id in the body:

{ "action": "status", "run_id": "20260428_021432_076255ef" }
{ "action": "result", "run_id": "..." }
{ "action": "logs",   "run_id": "...", "tail": 200 }
{ "action": "cancel", "run_id": "..." }

status returns the current state plus a summary block when the run is in a terminal state:

{
  "status": "succeeded",          // or running | failed | cancelled
  "run_id": "...",
  "started_at": "...",
  "finished_at": "...",
  // Present only when the run was started with ``live_viewer: true``
  // and the executor has stood up the MJPEG tunnel. Hot-link in any
  // browser while the run is still running.
  "viewer_url": "https://ta-...-7860-....w.modal.host?token=...",
  "summary": {
    "total_time_s": 569,
    "steps_executed": 17,
    "viable": 3,
    "leads_with_phone": 1,
    "result_path": "...",
    "csv_path": "...",
    "dynamic_verification_summary": { ... },
    "cost_total": 0.42,
    "cost_breakdown": {
      "gpu":    0.12,
      "claude": 0.12,
      "proxy":  0.18
    },
    "wall_time_breakdown": {
      "perceive":       12.4,
      "think":          88.1,
      "act":             6.7,
      "settle":         32.0,
      "claude_ground":  18.9,
      "claude_extract": 71.2,
      "claude_verify":   4.3,
      "load":           12.1,
      "overhead":        1.3
    }
  }
}

result returns the full lead list and per-step trace. logs returns the last tail events written by the runner (default 200, max 10000).

Structured extraction artifacts (#508)

In addition to the legacy leads string list, every result carries an artifacts array describing structured extracted data and downloadable files. The legacy fields (leads, csv_path, result_path) are kept for back-compat — artifacts is the new contract for callers that want schema-keyed rows or want to fetch files over HTTP rather than read server-local paths.

{
  "artifacts": [
    {
      "name": "extracted_rows",
      "kind": "structured_data",
      "mime_type": "application/json",
      "schema": { "fields": ["title", "url", "department"] },
      "row_count": 12,
      "data": [
        { "title": "ML Engineer", "url": "https://...", "department": "Eng" }
      ]
    },
    {
      "name": "leads.csv",
      "kind": "file",
      "mime_type": "text/csv",
      "row_count": 12,
      "download_url": "/v1/runs/<run_id>/artifacts/leads.csv"
    },
    {
      "name": "extracted_rows.csv",
      "kind": "file",
      "mime_type": "text/csv",
      "schema": { "fields": ["title", "url", "department"] },
      "row_count": 12,
      "download_url": "/v1/runs/<run_id>/artifacts/extracted_rows.csv"
    },
    {
      "name": "extracted_rows.json",
      "kind": "file",
      "mime_type": "application/json",
      "schema": { "fields": ["title", "url", "department"] },
      "row_count": 12,
      "download_url": "/v1/runs/<run_id>/artifacts/extracted_rows.json"
    }
  ]
}
Field Type Description
name string Stable identifier (extracted_rows, leads.csv, extracted_rows.csv, extracted_rows.json).
kind string structured_data (inline rows) or file (downloadable via download_url).
mime_type string Content type — for structured_data, always application/json; for file, the on-the-wire MIME of the served file.
schema.fields string[] (where applicable) Column order matching the ExtractionSchema field names. extracted_rows.csv uses these names as the CSV header; leads.csv keeps the legacy fixed columns.
row_count int Number of rows / lines the artifact contains.
data object[] (structured_data only) The rows themselves, keyed by schema field name.
download_url string (file only) Path to fetch the file from the artifact endpoint. Auth follows the standard X-Mantis-Token rules.

The extracted_rows structured artifact is the canonical form — it includes every schema field on every row even when a value is missing (empty string). leads.csv is preserved as a legacy file artifact with the historic fixed columns (status, year, make, ...) for callers that depend on that shape; new code should prefer extracted_rows.csv which uses the schema's actual columns.

The artifacts array is empty when a run produces no leads and no schema-driven extraction rows, so consumers can always iterate it without a KeyError guard.

Custom extraction schemas (#508)

/v1/predict accepts an optional top-level extraction_schema field describing the columns the run should extract. When set, this schema takes precedence over any plan-derived ObjectiveSpec schema and is the dict that drives ClaudeExtractor's tool-use JSON schema.

{
  "task_suite": { ... },
  "extraction_schema": {
    "entity_name": "job",
    "fields": [
      { "name": "title",      "type": "str", "required": true,  "example": "ML Engineer" },
      { "name": "url",        "type": "str", "required": true,  "example": "https://..." },
      { "name": "department", "type": "str", "required": false },
      { "name": "location",   "type": "str", "required": false }
    ],
    "required_fields": ["title", "url"]
  }
}

Fields not listed here are not extracted. The schema's fields order becomes the column order in extracted_rows.csv; missing values show up as empty strings rather than absent keys.

Wall-time breakdown

summary.wall_time_breakdown (epic #362) reports where wall-clock seconds went, alongside the existing cost_breakdown for dollars. Both come from the runner — cost_breakdown is owned by CostMeter, wall_time_breakdown by TimeMeter.

Each terminal summary carries the same nine buckets. step_details[i].time_breakdown (same shape, scoped to one step) lets you pinpoint which step dominated a bucket — e.g. one extract step consuming 65 s of claude_extract.

Bucket What lands here
perceive env.screenshot() capture, viewport sync.
think Brain inference — Holo3 / Gemma4 / Claude action emission.
act env.step(action) — xdotool keystroke / mouse / scroll, CDP click.
settle Post-action wait (fixed or adaptive); page-render quiescence.
claude_ground ClaudeGrounding.refine_* coordinate refinement on grounded steps.
claude_extract ClaudeExtractor.find_* extraction calls.
claude_verify Gate verification, DynamicPlanVerifier checks.
load env.reset(url) page-load + Cloudflare wait + proxy CONNECT.
overhead Residual — runner orchestration, retry loops, dispatch, anything not above.

Sum of the buckets tracks total_time_s within ±5 %; the residual lives in overhead. The mantis Python client exposes a typed accessor:

status = client.status(run_id)
bd = status.wall_time_breakdown()  # {} on pre-terminal runs
if bd:
    largest = max(bd, key=bd.get)
    print(f"largest bucket: {largest} ({bd[largest]:.1f}s)")

Pre-Phase-B summaries omit the wall_time_breakdown key entirely; the accessor returns {} so existing client code keeps working.

Errors

Status Meaning
400 Bad request. Common causes: no plan-shape provided, malformed JSON, plan exceeds MANTIS_MAX_STEPS_PER_PLAN, micro-step missing intent/type.
401 Missing or invalid X-Mantis-Token.
403 Token valid but tenant lacks run scope (read-only key).
404 action=status|result|logs referenced an unknown run_id.
429 (Tier 2) Tenant exceeded concurrent-run cap.
500 Unhandled exception — check events_path for traceback.
502 Upstream Holo3 (/v1/chat/completions) or Anthropic API unreachable.
503 Server auth not configured (MANTIS_API_TOKEN unset and no keys file).

POST /v1/chat/completions

OpenAI-compatible reverse proxy to the in-pod Holo3 server. For raw inference only — no plan orchestration, no Claude grounding, no checkpointing. Designed for clients that want to run their own perception-action loop and use Holo3 as the brain.

curl -X POST "https://model-qvvgkneq.api.baseten.co/production/sync/v1/chat/completions" \
  -H "Authorization: Api-Key $BASETEN_API_KEY" \
  -H "X-Mantis-Token: $MANTIS_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "holo3",
    "messages": [
      {"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
        {"type": "text", "text": "Click the boat listing title."}
      ]}
    ],
    "max_tokens": 256
  }'

Auth headers and Mantis-side cookies are stripped before the request is forwarded to llama.cpp; the upstream never sees your tenant credentials.

For the orchestrated/reliable path that handles the full plan, use /v1/predict instead.

GET /v1/models

OpenAI-compatible model listing.

{
  "object": "list",
  "data": [
    { "id": "holo3", "object": "model", "owned_by": "mantis" },
    { "id": "fara",  "object": "model", "owned_by": "mantis" }
  ]
}

See CUA models for the full list of cua_model values the dispatcher accepts and the action-space differences between brains.

End-to-end example: 3-listing extraction

TOKEN=$(read -srp "MANTIS_API_TOKEN: " v && echo "$v")
BTKEY="$BASETEN_API_KEY"
# Baseten gateway forwards /sync/<any path> to the container. /predict is
# the legacy default route (equivalent to /sync/predict).
ENDPOINT="https://your-model.api.baseten.co/production/sync"

# 1. Launch detached run — supply your own plan_text or a micro-plan.
RESP=$(curl -fsS -X POST "$ENDPOINT/v1/predict" \
  -H "Authorization: Api-Key $BTKEY" \
  -H "X-Mantis-Token: $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "detached": true,
    "plan_text": "Extract the first 3 listings from <your URL>: year, make, model, price, phone, url.",
    "profile_id":  "smoke",
    "workflow_id": "smoke-test-v1",
    "resume_state": false,
    "max_cost": 2,
    "max_time_minutes": 20
  }')
RUN_ID=$(echo "$RESP" | jq -r .run_id)
echo "run_id: $RUN_ID"

# 2. Poll status until terminal
while true; do
  STATUS=$(curl -fsS -X POST "$ENDPOINT/v1/predict" \
    -H "Authorization: Api-Key $BTKEY" \
    -H "X-Mantis-Token: $TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"action\":\"status\",\"run_id\":\"$RUN_ID\"}" | jq -r .status)
  echo "$(date '+%H:%M:%S') $STATUS"
  case "$STATUS" in succeeded|failed|cancelled) break ;; esac
  sleep 30
done

# 3. Fetch leads
curl -fsS -X POST "$ENDPOINT/v1/predict" \
  -H "Authorization: Api-Key $BTKEY" \
  -H "X-Mantis-Token: $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"action\":\"result\",\"run_id\":\"$RUN_ID\"}" \
  | jq .result.leads

Result shape (one row per successfully extracted listing):

<year> <make> <model>  — <price> — phone <phone or 'none'>
<year> <make> <model>  — <price>
<year> <make> <model>  — <price>

Plan shapes — when to use which

Use case Recommended shape
Recurring high-volume workflow with predictable steps Hand-author a micro-plan JSON, ship it in the image at plans/<domain>/<workflow>.json, reference via micro
Arbitrary plain-English request plan_text — server decomposes it via Claude (cached after first run)
Ad-hoc plan you don't want baked into the image task_suite (inline JSON dict)
Multi-task suite with task_id + verify clauses task_suite or task_file

Plan formats

micro — micro-plan JSON

A flat list of step objects executed by MicroPlanRunner:

[
  {"intent": "Navigate to https://...", "type": "navigate",
   "section": "setup", "required": true},
  {"intent": "Verify filters applied",  "type": "extract_data",
   "claude_only": true, "section": "setup", "gate": true,
   "verify": "Page shows boat listings ..."},
  {"intent": "Click listing title",     "type": "click",
   "grounding": true, "section": "extraction"},
  {"intent": "Read URL",                "type": "extract_url",
   "claude_only": true, "section": "extraction"},
  {"intent": "Scroll to description",   "type": "scroll",
   "budget": 10, "section": "extraction"},
  {"intent": "Extract data",            "type": "extract_data",
   "claude_only": true, "section": "extraction"},
  {"intent": "Go back",                 "type": "navigate_back",
   "section": "extraction"},
  {"intent": "Loop",                    "type": "loop",
   "loop_target": 2, "loop_count": 3, "section": "extraction"}
]

Step types: navigate, filter, click, scroll, extract_url, extract_data, navigate_back, paginate, loop.

Key fields:

Field Effect
section One of setup, extraction, pagination. Used by retry/halt logic.
required If true, retry on fail then halt the whole run.
gate Claude verifies a condition; halt on fail.
verify Free-text condition Claude checks.
claude_only Skip Holo3; Claude does the perception. Use for extract / gate steps.
grounding Refine click coordinates with ClaudeGrounding.
budget Max actions Holo3 can take in this step (default 8).
loop_target Step index to jump back to.
loop_count Max loop iterations (clamped to MANTIS_MAX_LOOP_ITERATIONS).

Plan-level runtime defaults

Plans can wrap the step list in {steps, runtime} so they declare their own proxy / cost / time defaults without every caller remembering the right submission flags:

{
  "runtime": {
    "proxy_disabled": false,
    "proxy_provider": "privateproxy",
    "proxy_city": "miami",
    "max_cost": 3.0,
    "max_time_minutes": 10
  },
  "steps": [ /* … */ ]
}

Submission overrides win — an explicit proxy_disabled: true in the HTTP body beats proxy_disabled: false in the plan, but omitting the body field falls back to the plan default. Schema and field reference: Plan formats → Declaring runtime defaults.

task_suite — multi-task JSON

For Claude-CUA-style autonomous-per-task workflows (the existing tasks/crm/crm_tasks.json is this shape):

{
  "session_name": "crm_demo",
  "base_url": "https://crm.example.com",
  "auth": { "user_id": "...", "password": "..." },
  "tasks": [
    {
      "task_id": "login",
      "intent": "Go to https://... and log in with user X and password Y",
      "save_session": true,
      "start_url": "https://...",
      "verify": { "type": "url_not_contains", "value": "login" }
    },
    {
      "task_id": "update_lead_industry",
      "intent": "Go to the Leads Page. Update industry of qualified lead to 'Space Exploration'.",
      "require_session": true,
      "start_url": "https://...",
      "verify": { "type": "page_contains_text", "value": "Space Exploration" }
    }
  ]
}

Each task runs with its own max_steps budget; Claude decides what to do per task. The runner verifies the verify clause after each.

plan_text — plain-English

{
  "plan_text": "Go to a marketplace listings site, filter to private sellers above $35,000 in Florida, extract listing details for the first 3 listings, save year/make/model/price/phone."
}

PlanDecomposer (Claude-backed, cached by signature) converts this into a micro-plan and proceeds. Decomposition costs ~$0.10 the first time per unique plan text; subsequent runs hit the cache.

Pricing (verified end-to-end)

Real numbers from a 3-listing marketplace-extraction run on Baseten:

Item Cost
GPU (Holo3 on H100, ~10 min) ~$0.12
Claude (gates + extract + grounding) ~$0.12
Proxy (IPRoyal residential) ~$0.18
Total per 3-listing run ~$0.42
Per-listing ~$0.14

For comparison, equivalent Claude-only CUA flow ~$0.50–$1.50 per listing.

Security model

Concern Guarantee
Tenant token confidentiality Stored in Baseten secrets; constant-time compare on validation; never echoed in logs
Per-tenant Anthropic key Resolved from the tenant's anthropic_secret_name — keys are not shared across tenants
Per-tenant browser profile Mounted at /workspace/mantis-data/tenants/<tenant_id>/chrome-profile/<profile_id>/ — cookies cannot bleed across tenants
Per-tenant run state Same volume layout — profile_id / workflow_id / legacy state_key are all server-prefixed so callers cannot read another tenant's checkpoint
Plan injection (e.g., loop_count: 999_999) Server-side hard caps clamp the values; oversized plans are rejected with 400
Upstream credential leak /v1/chat/completions strips X-Mantis-Token, Authorization, Cookie before forwarding to in-pod llama.cpp

Limits / caveats

  • Detached runs survive replica restart (state on the data volume) but only on the same Baseten model. Cross-region failover not supported.
  • /v1/chat/completions is unstreamed in v1. Streaming SSE is a Tier 2 follow-up.
  • Single Anthropic-key per tenant at request time (re-resolved on every call).

Pause / resume

OTP / 2FA / human-in-the-loop confirmation — #344.

When a plan hits an auth wall the agent can't get past on its own — an OTP code, a 2FA push, an explicit "yes, refund this" confirmation — a registered host tool raises PauseRequested. The runner snapshots its state, writes pause_state.json to the run dir, and flips the run's status to paused. The caller polls status, surfaces the prompt to a human (or fetches the code from a side channel), then resumes with the answer.

A default request_user_input host tool is registered on every detached /v1/predict run. Brains that emit Action(TOOL_CALL, name="request_user_input", params={"prompt": "..."}) will pause the run on the first call and receive the caller's user_input on the second (after resume).

The plan_text hand-over pattern (ask the user mid-run)

You don't drive steps yourself — submit a single plan_text (detached) and the decomposer breaks it into a MicroPlan and runs it step by step. To get a human hand-over, write the hand-over into the plan in plain English — phrase it as "ask the user … and wait for the answer, then use that answer to …". The decomposer emits a request_user_input step followed by a step that references the answer through the {{user_input}} token.

Example plan:

"Go to news.ycombinator.com, read the top 3 story titles, ask the user which story title to open and wait for the answer, then open that story and report its title."

Decomposes to a 5-step plan:

[0] navigate           → news.ycombinator.com
[1] extract_data       → top 3 story titles
[2] request_user_input → pauses; prompt shown to you
[3] click {{user_input}}  → the answer is substituted in verbatim
[4] extract_data       → the opened story title

Step [2] is the hand-over: the run flips to status=paused and surfaces a prompt. You resume with action=resume + user_input, and the value is substituted verbatim into every {{user_input}} token on the remaining steps (intent + string params) before they execute. End to end:

# 1. submit (detached) — note the answer is used as a CLICK TARGET, so the
#    prompt should ask for something usable as one (a title/label, not "1/2/3").
RUN_ID=$(curl -s "$ENDPOINT/v1/predict" -H "X-Mantis-Token: $TOKEN" \
  -d '{"plan_text":"Go to news.ycombinator.com, read the top 3 story titles, ask the user which story title to open and wait for the answer, then open that story and report its title.","detached":true}' \
  | jq -r .run_id)

# 2. poll until paused, read the prompt
curl -s "$ENDPOINT/v1/predict" -H "X-Mantis-Token: $TOKEN" \
  -d "{\"action\":\"status\",\"run_id\":\"$RUN_ID\"}" | jq '{status, prompt, reason}'
# → { "status": "paused", "prompt": "Which story title should I open?", "reason": "user_input" }

# 3. resume with the human's answer → substituted into {{user_input}} on step [3]
curl -s "$ENDPOINT/v1/predict" -H "X-Mantis-Token: $TOKEN" \
  -d "{\"action\":\"resume\",\"run_id\":\"$RUN_ID\",\"user_input\":\"Show HN: my side project\"}"
# → { "status": "running", ... }

# 4. keep polling until terminal (succeeded / failed)

Use the answer verbatim. Whatever string you send as user_input is what replaces {{user_input}} — so phrase the plan's question so the answer is directly usable by the next step (a concrete title or on-screen label for a click, a code for an OTP field, etc.). An ordinal like "2" will be clicked literally as the text "2".

Troubleshooting — run never pauses, logs say request_user_input step N: no host tool registered; downgrade to skip. This means the deployment is running code older than #883 — the request_user_input step is downgraded to a skip, {{user_input}} is never filled, the run finishes completed_with_failures, and action=resume is rejected with requires a paused run. The log line is byte-identical between the old and fixed handlers, so it does not tell you which code is live. Fix: redeploy the app from current main (modal app stop <app> then modal deploy deploy/modal/modal_mantis_server.py for the mantis-server endpoint). Pause/resume + {{user_input}} substitution require #883 (wiring), #885 (resume staging), and #887 (initial-path staging) — all on main as of 2026-06-13.

Status poll on a paused run

POST /v1/predict
{"action": "status", "run_id": "20260513_180527_abc"}

→ {
  "status":       "paused",
  "run_id":       "20260513_180527_abc",
  "prompt":       "Enter the 6-digit code from your authenticator",
  "reason":       "user_input",
  "pause_state":  { /* opaque PauseState blob — hand back on resume */ }
}

pause_state is opaque — the server is the only thing that interprets it. Treat it as a token: store it if you want, but you don't need to send it back yourself. The server already has the canonical copy on disk under the run_id; it round-trips automatically on resume.

What's captured at pause time

Captured Restored on resume Notes
Step index + plan signature ✓ — runner picks up at the next un-run step Round-trips via pause_state.step_index + plan_signature.
Step results so far ✓ — replayed into the runner state Lets _handle_success / dedup logic see prior outputs.
Pending tool call (pending_tool + pending_arguments + prompt) ✓ — the resumed runner re-invokes the tool with user_input set The mechanism that lets a paused tool finish its call_tool round-trip.
URL + scroll + viewport (browser_state, epic #358 Phase A) ✓ — agent re-lands on the exact pixel CDP-captured (location.href, window.scrollX/Y, window.innerWidth/Height) just before pause raises. Empty when the env doesn't expose CDP (legacy adapters).
Cookies / localStorage / IndexedDB ✓ — but via the profile_id Chrome user-data-dir, not pause_state Persists across runs on its own; profile_id is the identity that scopes the dir.
Unsubmitted form input (browser_state.form_state, Phase B (#360)) ✓ — half-filled inputs / selects / checkboxes / radios / contenteditable repopulate on resume Keyed by stable selector (data-* > id > short CSS path). Passwords masked: the selector is kept so the caller knows which field to re-prompt, but the value is dropped before serialization (opt in via MANTIS_PAUSE_CAPTURE_PASSWORDS=1 for test/debug only). Missing selectors on the resumed page are silently skipped.
In-memory JS state (React/Redux store, in-flight network) ✗ — fresh page load Container-level snapshots would be the right answer; out of scope.

Resume

POST /v1/predict
{"action": "resume", "run_id": "20260513_180527_abc", "user_input": "123456"}

→ {"status": "running", "run_id": "20260513_180527_abc", "resumed_at": "2026-05-13T18:09:12Z"}

The server rehydrates the stored PauseState, calls runner.resume(state, user_input=...) against the same profile_id / workflow_id the original run used, and continues from the paused step. Subsequent action=status polls return running until the run reaches a terminal status (succeeded / failed / cancelled) — or pauses again, in which case the cycle repeats with a fresh prompt.

Error cases

Status Cause
400 action='resume' requires user_input Missing the user_input field
400 action='resume' requires a paused run Run isn't currently in paused status (succeeded, running, cancelled, ...)
400 plan signature mismatch on resume Disk-stored pause_state.plan_signature doesn't match the current plan derived from the stored payload — usually means someone edited the on-disk state
404 unknown run_id No status.json for that run_id on this tenant

Deployment coverage

Deployment Pause / resume
Baseten (/v1/predict via BasetenCUARuntime) ✅ All brains — Holo3, Claude, EvoCUA, OpenCUA, Gemma4-CUA. Default request_user_input tool registered on every detached run (#344).
Modal mantis-server (<workspace>--mantis-server-api.modal.run, deploy/modal/modal_mantis_server.py) ✅ Serves the same baseten_server FastAPI app as Baseten, so the plan_text / micro / task_suite paths all register the default request_user_input tool and pause/resume identically. This is the endpoint to use for the plan_text hand-over pattern above. Must be deployed from main ≥ #883 — older deploys skip the step (see the troubleshooting note above).
Modal mantis-cua-server (<workspace>--mantis-cua-server-api.modal.run, deploy/modal/modal_cua_server.py) ✅ Holo3 micro path — constructs MicroPlanRunner directly. Claude / EvoCUA / OpenCUA / Gemma4-CUA on this app go through task_loop.run_executor_lifecycle and don't currently surface paused state (#347).
Modal local_entrypoint (CLI: modal run ...) ❌ Not wired. Use an HTTP endpoint or embed the library.
Library-embedded (MicroPlanRunner / GymRunner direct) ✅ Always — pause/resume is a property of the runner. The HTTP surfaces above are wrappers on top.

Library-embedded integrations

If you embed MicroPlanRunner / GymRunner directly (no HTTP), the same primitives are available in-process via runner.run_with_status(plan) returning RunnerResult(paused=True, pause_state=...), plus runner.resume(state, user_input=..., plan=plan). See Embedding MicroPlanRunner for the canonical walkthrough — the HTTP surface is just a wrapper on top of the same library API.

Screencast / video recording

Send a plan with record_video: true and the runtime produces a feature-walkthrough video — title card → captioned run footage → outro card with the result summary. Fetch with GET /v1/runs/{run_id}/video. The raw screencast is preserved alongside; pass ?raw=1 to fetch it instead.

The walkthrough has three segments plus animated click ripples on top of the run footage:

┌─────────────────┐  ┌─────────────────────────┐  ┌─────────────────┐
│  Title card     │→ │  Run footage (captions  │→ │  Outro card     │
│  (3s)           │  │   + click ripples)      │  │  (5s)           │
│                 │  │  per-step intent shown  │  │                 │
│  Mantis CUA     │  │  with [OK] / [FAIL]     │  │  Run complete   │
│  ───            │  │  in the bottom strip    │  │  ───            │
│  <plan name>    │  │  while the action plays │  │  3 viable leads │
│  tenant: …      │  │  + expanding sky-blue   │  │  1 with phone   │
│  run: …         │  │  ripple at every click  │  │  17 steps · 9m  │
│                 │  │                         │  │  cost: $0.42    │
└─────────────────┘  └─────────────────────────┘  └─────────────────┘

Title and outro are rendered with PIL. Captions are SRT cues burned in by ffmpeg's subtitles= filter (libass). Click ripples are PNG-sequence overlay frames composited via ffmpeg's overlay filter. Polish is best-effort — if anything fails (PIL, ffmpeg, libass not built in the image), the raw recording is still saved and the endpoint serves it.

Action overlays — universal computer use

Every kind of agent action gets a visual cue, regardless of what application is in focus (browser, file manager, terminal, dialogs, anything visible on the Xvfb display). The agent emits actions with pixel coordinates / key chords / text, and the overlay renderer composites the matching visual onto the recording.

Agent action Overlay
CLICK (single) Sky-blue expanding ripple at (x, y), 0.6 s, fades out
DOUBLE_CLICK Same as click + a second offset ring 0.1 s later
KEY_PRESS (e.g. Ctrl+S, Tab, Enter) Slate badge in the bottom-right with the chord text, 1.5 s, slide-in then fade
TYPE (typed text) "⌨ Typing: "…"" caption near the top, 1.8 s, fades after text appears on screen
SCROLL (up / down / left / right) Sky-blue arrow at the matching screen edge, slides in the scroll direction, 0.8 s
DRAG Animated trail line from start to end with a moving head dot, 0.9 s
WAIT, NAVIGATE, DONE No overlay (no useful visual locus)

All overlays are deliberately minimal — visible without being disruptive. Sky-blue accent color across the set so they read as a single visual language.

You'll see counts in the result metadata under video.actions:

{
  "video": {
    "path": ".../recording.mp4",
    "polished_path": ".../recording_polished.mp4",
    "actions": {
      "clicks": 17,
      "keys":   3,
      "types":  2,
      "scrolls": 8,
      "drags":  0
    },
    "clicks": 17,    // backwards-compat field
    ...
  }
}
# 1. Submit a recorded run
RESP=$(curl -fsS -X POST "$ENDPOINT/v1/predict" \
  -H "Authorization: Api-Key $BTKEY" \
  -H "X-Mantis-Token: $TOK" \
  -H "Content-Type: application/json" \
  -d '{
    "detached": true,
    "micro": "plans/example/extract_listings.json",
    "profile_id":  "demo-recording",
    "workflow_id": "demo-recording-v1",
    "max_cost": 2,
    "max_time_minutes": 20,
    "record_video": true,
    "video_format": "mp4",
    "video_fps": 8
  }')
RUN_ID=$(echo "$RESP" | jq -r .run_id)

# 2. Poll status until succeeded ... (same as the regular flow)

# 3. Download the screencast
curl -fsS -o demo.mp4 \
  -H "X-Mantis-Token: $TOK" \
  "$ENDPOINT/v1/runs/$RUN_ID/video"

Result-side metadata (in the summary block):

{
  "video": {
    "path": "/workspace/mantis-data/tenants/<tenant>/runs/<run_id>/recording.mp4",
    "polished_path": "/workspace/mantis-data/tenants/<tenant>/runs/<run_id>/recording_polished.mp4",
    "format": "mp4",
    "duration_seconds": 567.3,
    "bytes": 31457280,
    "error": null
  }
}

polished_path is set only when the post-process compose step succeeded; on failure it's omitted and the endpoint falls back to the raw recording.

Endpoint behavior

Request Returns
GET /v1/runs/{run_id}/video Polished mp4 (preferred) → raw mp4 (fallback) → 404
GET /v1/runs/{run_id}/video?raw=1 Raw mp4 only → 404

Format tradeoffs

Format Container Encode cost Output size (typical 10-min run) Best for
mp4 H.264 (libx264, ultrafast preset, CRF 28) low ~30–80 MB sharing, downloads
webm VP9 (libvpx-vp9, cpu-used 5, CRF 32) medium ~25–60 MB embedding in web pages
gif palettegen + paletteuse high ~50–200 MB docs, Slack, animated thumbnails (lossy)

For long recordings or tight bandwidth, prefer mp4 at 5 fps. The gif path uses a palette-aware filtergraph but file size grows fast — use only for short demos (< 60 s).

Operational caveats

  • The container image must have ffmpeg installed. Both docker/server.Dockerfile and deploy/baseten/holo3/config.yaml ship it; if you're rolling your own image, add ffmpeg to the apt deps. Without ffmpeg, record_video: true is a soft-fail — the run completes normally, and the response carries video.error: "ffmpeg-not-installed".
  • video.error envelopes you may see (soft-fail in every case — the run itself never fails because recording couldn't start):
    • ffmpeg-not-installed — ffmpeg isn't on PATH inside the container.
    • x-display-not-ready:<display> — the Xvfb display ffmpeg targets wasn't up when the recorder fired. The runtime now calls env.ensure_display_ready() before spawning the recorder, so this error only appears when a custom image hasn't installed xvfb / xdpyinfo or when a third-party caller spawns ScreenRecorder directly outside the standard runtime. Fix: install xvfb + xdpyinfo (apt: xvfb x11-utils) in your image.
    • ffmpeg-startup-failed:<stderr> — ffmpeg exited within ~300 ms of spawn. The trailing stderr blob is the actionable part; common cause was historically Cannot open display :99, error 1 (fixed by the display-ready probe above).
    • empty-output — ffmpeg started and stopped cleanly but wrote a zero-byte file. Usually means the run completed before any frames captured.
    • spawn-failed:<oserror> — the Popen itself raised (e.g., process budget exhausted).
  • Recordings live at $MANTIS_DATA_DIR/tenants/<tenant_id>/runs/<run_id>/recording.<fmt> so tenants cannot read each other's files. The download endpoint uses the authenticated tenant's dir; even if you guess another tenant's run_id, the file lookup is scoped.
  • video_fps is clamped to [1, 30]. Higher fps doesn't help much (UI rarely changes faster than 5–10 fps) and bloats the file.
  • Each second of recording is ~50 KB at 5 fps mp4. Multiply by your target run duration + tenant count to size the EFS / Filestore.

Tier 2 features (rate limits, idempotency, webhooks, allowlist, metrics)

Rate limits

Two dimensions, both enforced per-tenant:

Dimension Source Behavior on exceed
Concurrent runs tenant.max_concurrent_runs (default 5) 429 Too Many Requests with Retry-After: 5
Rate (token bucket) tenant.rate_limit_per_minute (default 30) 429 with Retry-After: <seconds-until-token>

State is in-process per replica. Behind a load balancer with N replicas, the effective per-tenant cap is roughly N × configured_cap. For strict cluster-wide limits, deploy a single replica or swap to a Redis-backed limiter (planned Tier 2.5).

Idempotency keys

Send Idempotency-Key: <unique-string> on POST /v1/predict. The server caches (tenant_id, key) → run_id with a 24-hour TTL. Subsequent retries with the same key return the original run_id without starting a new run.

curl -X POST "$ENDPOINT/v1/predict" \
  -H "Authorization: Api-Key $BTKEY" \
  -H "X-Mantis-Token: $TOK" \
  -H "Idempotency-Key: order-7afc3b91" \
  -H "Content-Type: application/json" \
  -d '{...}'

The cache is sidecar-backed ($MANTIS_DATA_DIR/idempotency/<tenant_id>/<key_hash>.json) so a replica restart preserves it.

Webhook callbacks

Two ways to receive run-completion notifications:

  1. Per-tenant default — set webhook_url and webhook_secret_name in the tenant keys file.
  2. Per-request override — pass callback_url in the /v1/predict body.

When the run reaches a terminal state (succeeded, failed, cancelled), the server POSTs:

{
  "run_id": "20260428_021432_076255ef",
  "tenant_id": "tenant_a",
  "status": "succeeded",
  "summary": { ... same shape as /v1/predict status response ... },
  "delivered_at": "2026-04-28T02:24:01.648Z"
}

With an HMAC-SHA256 signature in X-Mantis-Signature: sha256=<hex> (signed with the tenant's webhook secret). 3 retries with exponential backoff (1s, 5s, 30s) if the receiver returns non-2xx or fails to connect.

Verify the signature on receipt:

import hmac, hashlib
def verify(body: bytes, header_sig: str, secret: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header_sig)

URL allowlist enforcement

If a tenant has allowed_domains set in the keys file, every plan submitted via /v1/predict is scanned for navigate-type URLs and task_suite.base_url / task.start_url. Off-list hosts return 403 Forbidden before any run starts:

{
  "detail": "plan references host(s) not in tenant allowlist: evil.com"
}

Wildcards: *.example.com matches any subdomain but not example.com.evil.com. Empty allowed_domains (the default) skips this check.

Prometheus metrics

GET /metrics returns Prometheus text format. Metric names + labels:

Metric Type Labels Notes
mantis_predict_requests_total counter tenant_id, mode, outcome mode = run|status|result|logs|cancel; outcome = ok|bad_request|rate_limited|denied_allowlist|idempotent_hit|error
mantis_chat_completions_total counter tenant_id, outcome outcome = ok|status_4xx|status_5xx|upstream_error
mantis_run_duration_seconds histogram tenant_id, model, status Buckets: 10s … 3600s
mantis_run_cost_usd histogram tenant_id, model, status Buckets: $0.01 … $25
mantis_concurrent_runs gauge tenant_id Currently in-flight runs
mantis_rate_limit_rejections_total counter tenant_id, kind kind = rate|concurrent

If prometheus_client isn't installed in the container (e.g., orchestrator-only install), /metrics returns 503 and all metric calls become no-ops — the rest of the API is unaffected.

Tier roadmap

This API is at Tier 2 — production-quality multi-tenant. Upcoming:

  • Tier 3: billing records, admin API, multi-region.

See Architecture for the bigger architectural picture.