A simplified personal AI gateway with browser control — built to understand how production AI agent systems actually work from the inside.
Browser UI ──WS──► Gateway :18789 ──asyncio.Task──► Claude API
│ │
│◄──── streaming tokens ────────────┘
│
└──HTTP──► Browser Server :18790 ──► Playwright Chromium
| Chat with streaming | Browser control in action |
|---|---|
![]() |
![]() |
Type a message and press Enter (or Shift+Enter for a newline). Responses stream in token-by-token and are rendered as markdown when complete.
The assistant label turns green when a response is in progress. Click Stop to abort the current run — the pipeline cancels mid-stream.
Each browser tab gets its own conversation — opening a new tab starts fresh. Within a tab, the full conversation history is in scope so you can ask follow-up questions naturally.
Claude has five browser tools it can use autonomously:
| Tool | What it does |
|---|---|
navigate |
Go to a URL |
screenshot |
Take a viewport PNG — Claude sees it via vision |
click |
Click an element by text or CSS selector |
type_text |
Type into an input field |
get_text |
Extract text from the page or a specific element |
scroll |
Scroll the page in any direction |
You don't invoke these manually — just describe what you want and Claude decides which tools to call and in what order.
Example prompts that work well:
Show me the top headlines on news.ycombinator.com, with a screenshot
Go to github.com/trending and summarise what's trending in Python today
Search for "asyncio tutorial" on Google and open the first result
Go to weather.com, find the forecast for New York, and tell me if I need an umbrella this week
While Claude is browsing you'll see ⚙ indicators in the bubble — one per tool call — showing the tool name and a preview of its input. Screenshots are rendered inline so you can see exactly what Claude saw when it made decisions.
By default headless: false in config.yaml, so the Playwright Chromium window is visible on screen. You can watch Claude navigate in real time. Set headless: true to run silently in the background.
Edit config.yaml:
agent:
model: "claude-opus-4-6" # more capable, slower
# model: "claude-sonnet-4-6" # default — fast and capable
max_tokens: 8096Or use an environment variable without restarting (requires a new session):
MINIAGENT_AGENT__MODEL=claude-opus-4-6 uv run python -m miniagentConversations are stored as JSONL files under ~/.miniagent/sessions/. Each session key maps to one file. To keep history across page reloads, change the session key in miniagent/ui/index.html:
// line ~373 in ui/index.html
const SESSION_KEY = "my-persistent-session"; // fixed key = persistent history
// const SESSION_KEY = "session-" + crypto.randomUUID(); // random = fresh each loadTo clear a session, delete the corresponding file:
rm ~/.miniagent/sessions/my-persistent-session.jsonl| Symptom | Likely cause | Fix |
|---|---|---|
| Blinking cursor, no response | API key not set | Check .env has MINIAGENT_ANTHROPIC_API_KEY |
| "Browser connection refused" | Browser server not running | Restart — __main__.py now starts both servers |
| Click times out | JS overlay intercepting click | Already handled — dispatch_event fallback is automatic |
| Image too large error | Full-page screenshot > 8000px | Already fixed — viewport-only screenshots |
| Response includes old context | Using same session key across conversations | Each tab gets its own key by default |
I wanted to deeply understand how a production AI agent system is actually structured — not from tutorials, but by building a real working system from scratch.
The reference I used was OpenClaw, a production-grade AI gateway. OpenClaw is well-architected but large (TypeScript, multi-provider, multi-channel). I built miniagent as a Python translation of its core ideas, stripped down to the minimum needed to understand the patterns:
- One AI provider (Anthropic Claude)
- One channel (browser WebSocket)
- One browser backend (Playwright Chromium)
The goal wasn't to build something novel — it was to understand why each piece exists by having to build it myself. You can't really understand a broadcaster, a connection lifecycle, or a double-response RPC pattern by reading about them. You have to wire them up and watch them break.
A naive implementation would forward messages directly from the WebSocket to the API. The real architecture is more careful:
- The WebSocket connection lifecycle is a state machine (pending → connected → closed), with a handshake timeout and a
connect.challengeto confirm the client is real before anything is registered - The broadcaster fan-out is separate from the connection — events flow from the agent pipeline through an in-process event emitter, then to the broadcaster, then to every connected client. This means multiple tabs see the same run
- The double-response pattern for
agentrequests matters: the handler sends an immediateacceptedack, then fires anasyncio.Task. The WebSocket is unblocked immediately; the background task sends a second response when the run finishes. Without this, the UI would hang waiting for a response that could take minutes
The browser server runs on its own port (:18790) and is called over HTTP, never imported directly. This looks like over-engineering at first. The reason: if Playwright crashes (and it does, on complex JS-heavy pages), it takes down its own process only. The gateway and active sessions survive. You restart the browser server and continue.
This is the same pattern OpenClaw uses — browser isolation isn't a deployment detail, it's a resilience decision baked into the architecture.
Sending a screenshot to Claude as str({"image_b64": "..."}) produces ~800,000 tokens. The correct approach is Anthropic's vision image block format — the same PNG sent as a properly typed base64 block costs ~2,000–3,000 tokens and Claude can actually see it. This isn't obvious from the docs until you hit the 200k token limit error in production.
Similarly, Playwright's full_page=True captures the entire scroll height of a page — news sites can be 15,000px tall. Anthropic's vision API rejects images over 8,000px in either dimension. Viewport-only screenshots stay within limits.
Every message in the same session shares history. That's by design — it's how multi-turn conversations work. The subtlety is that "session" needs to be scoped deliberately: the same hardcoded key across all browser tabs means tab A's Google News search bleeds into tab B's Yahoo News question. Generating a unique key per page load gives each conversation a clean scope while still supporting multi-turn within a session.
Modern sites like Google News use overlapping <div> elements with custom jsaction handlers that intercept pointer events before they reach the actual link. Playwright's locator.click() times out because it correctly detects that another element would intercept the click. The fix is dispatch_event('click') as a fallback — this fires the event directly on the target DOM node, bypassing pointer interception entirely.
The entire project was built using Claude Code with a parallel sub-agent workflow. Rather than writing code sequentially, multiple agents worked concurrently on independent modules.
The key insight that made this work: the modules were designed to have clean interfaces before any code was written. config/ and protocol/ have no internal dependencies, so they were built in parallel. sessions/ and browser/ only depend on config/, so they ran in parallel next. agent/ depended on config/ and protocol/. gateway/ wired everything together last.
Each agent received:
- The exact files it owned
- The relevant DESIGN.md section
- A clear interface contract (what it imports, what imports it)
- Instructions to update a shared
MEMORY.mdwhen done
The CLAUDE.md file at the project root was injected into every agent automatically, so conventions (async everywhere, Pydantic v2, no real API calls in tests) were enforced without repeating them in every prompt.
The whole thing — 197 tests, 6 modules, a working browser agent — was built in a single session.
- Streaming chat — tokens stream in real time over WebSocket, rendered as markdown when complete
- Browser control — Claude can navigate, screenshot, click, type, and scroll
- Session memory — conversation history persisted as JSONL, scoped per session key
- Single-file UI — zero build step, vanilla JS, dark theme, inline markdown renderer
- Process isolation — browser server runs independently; gateway survives browser crashes
- 197 tests — fully mocked with
AsyncMock, no real API calls needed
Prerequisites: Python 3.12+, uv
# 1. Clone and install
git clone https://github.com/ridhamdave/miniagent
cd miniagent
uv sync
# 2. Install Playwright browser
uv run playwright install chromium
# 3. Set your Anthropic API key
cp .env.example .env
# edit .env → set MINIAGENT_ANTHROPIC_API_KEY=sk-ant-...
# 4. Start
uv run python -m miniagentOpen http://localhost:18789 in your browser.
config.yaml (committed with safe defaults):
gateway:
port: 18789
host: "127.0.0.1"
browser:
control_port: 18790
headless: false # set true to hide the browser window
timeout_ms: 8000
agent:
model: "claude-sonnet-4-6"
max_tokens: 8096Environment variables always win (prefix MINIAGENT_):
MINIAGENT_ANTHROPIC_API_KEY=sk-ant-...
MINIAGENT_PORT=18789
MINIAGENT_BROWSER_PORT=18790miniagent/
├── config/ Pydantic config — types, env settings, singleton loader
├── protocol/ WebSocket frames — RequestFrame, ResponseFrame, EventFrame, error codes
├── gateway/ FastAPI WebSocket server, connection lifecycle, RPC dispatch
│ └── handlers/ agent (double-response), chat (history/abort), browser (HTTP proxy)
├── agent/ Anthropic streaming pipeline, recursive tool loop, event emitter
├── browser/ Standalone FastAPI HTTP server — Playwright Chromium on port 18790
├── sessions/ JSONL append/load/clear — one file per session key
└── ui/ index.html — single-file vanilla JS chat, streaming cursor, markdown
tests/
├── test_config.py 30 tests — config merging, env override, singleton
├── test_protocol.py 43 tests — Pydantic round-trips, literal discriminators
├── test_sessions.py 15 tests — JSONL append/load/clear, tmp_path isolation
├── test_browser.py 35 tests — route handlers, lifespan, mock Playwright
├── test_agent_pipeline.py 38 tests — mock Anthropic streaming, tool call loop
└── test_gateway.py 36 tests — WebSocket handshake, RPC dispatch, broadcaster
uv run pytest # all 197 tests (~0.6s)
uv run pytest -v # verbose with test names
uv run pytest tests/test_agent_pipeline.py # single module| Decision | Why |
|---|---|
| Browser on a separate port | Crash isolation — Playwright can fail without taking down the gateway |
Double-response for agent RPC |
Unblocks the WebSocket immediately; background task streams tokens |
| Event emitter → broadcaster → WebSocket | Decouples pipeline from transport; multiple tabs see the same run |
| JSONL for session storage | Append-only, human-readable, no database dependency |
| Screenshots as vision blocks | str(base64) = 800k tokens; typed image block = ~2k tokens |
| Per-page-load session key | Each browser tab gets a clean conversation scope |
dispatch_event('click') fallback |
Bypasses pointer interception on JS-heavy sites (Google News, etc.) |
| Layer | Library |
|---|---|
| Web framework | FastAPI + uvicorn |
| WebSocket | Starlette native |
| AI provider | anthropic SDK (async streaming) |
| Browser | Playwright (async Chromium) |
| HTTP client | aiohttp |
| Config | Pydantic v2 + pydantic-settings |
| Tests | pytest + pytest-asyncio |
| Package manager | uv |

