taproot

Open-source harness that drives a real Android device or emulator from natural language, using an off-the-shelf vision model plus accessibility-tree grounding. It runs on Google's AndroidWorld benchmark today and is built to target MobileWorld next.

The name is "tap" (mobile) plus "root" (a taproot is the deep anchoring root): grounding, on device.

Why this exists

The intelligence to operate a phone is now a commodity you rent (Gemini computer-use, Claude computer-use). The hard, unsolved part is reliability: turning a model's rough intent into taps that actually land, verifying they worked, and recovering when they did not. taproot is a grounding-first harness focused on that reliability layer, not on training a new model.

What it does

Given a goal like "add a contact named Alex, number 555 0142", taproot:

1. Screenshots the current screen and sends it to the model.
2. Gets a proposed action (click, type, scroll, open app).
3. Grounds it: snaps the model's coordinate to the nearest real UI element from the accessibility tree, then clicks by element index, which is the most reliable action.
4. Executes over ADB, verifies the result, and repeats.

Architecture

File	Role
agent/model.py	Gemini computer-use client; returns the next action
agent/device.py	ADB wrapper: screenshot, UI dump, tap/type/swipe, open app
agent/grounding.py	Accessibility-tree parsing and coordinate snapping (the reliability layer)
agent/loop.py	The observe, act, verify orchestrator
agent/verify.py	Success predicates
agent/androidworld_agent.py	Adapter to run inside AndroidWorld with strict scoring
agent/mobileworld_agent.py	Adapter for MobileWorld (untested; needs a Linux/KVM host)
bench/run_androidworld.py	AndroidWorld runner
eval/	Small local task suite for fast iteration
scripts/	Emulator setup (macOS) and one-command Linux setup

Quickstart (AndroidWorld)

python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env          # add GEMINI_API_KEY

# Install AndroidWorld and set up the AndroidWorldAvd emulator.
# scripts/setup_linux.sh has the exact, tested steps (Python 3.11, proto compile, etc).
# Boot the emulator with the gRPC port it needs:
#   emulator -avd AndroidWorldAvd -no-snapshot -grpc 8554

python -m bench.run_androidworld --setup ContactsAddContact   # one-time app install
python -m bench.run_androidworld --sample                     # scored sample run

For a quick local smoke test without AndroidWorld, point eval/run.py at any booted emulator or device:

python -m agent.loop "add a contact named Alex, number 5550142"
python -m eval.run

Status

Early and honest. The harness runs end to end on real AndroidWorld and passes real tasks under strict verifiers. On a 15-task sample on a macOS emulator it scores in the 40 percent range with an off-the-shelf model and zero tuning: above raw single-agent frontier baselines (roughly 25 to 35 percent) and below the tuned state of the art (MobileUse at 62.9 percent, DroidRun at 91.4 percent). Numbers on a laptop emulator are high variance; a stable multi-seed number needs a Linux host.

AndroidWorld is effectively solved at the top of the leaderboard, so here it is a calibration target, not the goal. The goal is MobileWorld (longer, cross-app, agent-user and tool-use tasks), where there is real headroom and no runaway leader yet.

Roadmap

• Grounding directly on the benchmark's own UI elements
• Reliable form filling and multi-step recovery
• Trajectory and UI-map memory: skills learned once and replayed deterministically
• MobileWorld on a Linux/KVM host, including MCP tool-use and agent-user turns

License

Apache License 2.0. See LICENSE.