arkai

For troubleshooting, add the global --debug option before a command. It enables maximum log output and prints a full exception traceback when a command fails:

arkai --debug inference start

arkai is a CLI for running AI agents with local models on your own hardware. It orchestrates three services — an OpenAI-compatible inference server (llama.cpp or optional MLX-LM), an MCP plugin server (wtmcp), and optionally an arapuca sandbox — so you can launch a fully configured agent session with a single command. Configuration is layered: user-level defaults, per-project overrides, and CLI flags.

How It Works

When you run arkai agent start, it starts llama-server (loading your GGUF model), starts wtmcp (which exposes MCP tools to the agent over HTTP), then launches the agent binary configured to talk to both over localhost. Optionally it wraps the agent in an arapuca sandbox that restricts filesystem and network access. On exit, arkai tears down wtmcp — and optionally the inference server — unless --keep-mcp or --keep-inference are passed.

┌─────────────────────────────────────────────────────┐
│  arkai agent start                                  │
│                                                     │
│  ┌──────────────┐   HTTP/v1   ┌─────────────────┐   │
│  │  llama-server│◄────────────│                 │   │
│  │  (inference) │             │   agent binary  │   │
│  └──────────────┘             │ (opencode/crush │   │
│                               │   /claude)      │   │
│  ┌──────────────┐   MCP/HTTP  │                 │   │
│  │    wtmcp     │◄────────────│                 │   │
│  │  (plugins)   │             └─────────────────┘   │
│  └──────────────┘                    │              │
│                               ┌──────┴──────┐       │
│                               │   arapuca   │       │
│                               │  (sandbox)  │       │
│                               └─────────────┘       │
└─────────────────────────────────────────────────────┘

Requirements

Common development and CLI tools

• Python
• pip
• git
• shasum
• grep, sed, bash (modern version)

External tools:

• git — for downloading and updating Hugging Face model repositories
• llama.cpp (llama-server and llama-quantize)
• mlx-lm (pip install -e '.[mlx]', optional; Apple Silicon)
• wtmcp — for MCP tool integration with agents (optional)
• arapuca — for sandboxed agent execution (optional)
• Agent binary: opencode, crush, claude, or pi — at least one required

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/arkai.git
   cd arkai

2. Create and activate a Python virtual environment:

   python3 -m venv venv
   source venv/bin/activate

3. Install the package:

   pip install -e .

   For development tools (pytest, ruff, etc.):

   pip install -e ".[dev]"

On Linux, arapuca is available as a Fedora package. On macOS, llama.cpp and arapuca are available through Homebrew.

Quick Start

Download a model repository:

arkai model download hf:ibm-granite/granite-4.1-8b-GGUF

Start agent:

arkai agent start -m hf:ibm-granite/granite-4.1-8b-GGUF -a opencode

Install pi.dev and its standard MCP, web-access, and subagent packages:

arkai agent install pi

Run arkai agent install without a name to list installable agents. The command warns that installation writes files to disk and requires confirmation. It installs @earendil-works/pi-coding-agent, pi-mcp-adapter, pi-web-access, and pi-subagents.

Pi is installed only for arkai under ~/.local/state/arkai/pi/; it is not added to your global npm installation or shell PATH. Node.js and npm must be installed.

If you think you only live once (or don't have wtmcp or arapuca installed):

arkai agent start \
    -m hf:ibm-granite/granite-4.1-8b-GGUF \
    -a opencode \
    --no-sandbox \
    --no-mcp

Managing Models

Models are stored as GGUF files in ~/.local/share/arkai/models/. The inference.model config key refers to the filename within that directory. Named model profiles can be kept in the same configuration file. Set inference.model to the profile name; its model value is started and its sampling values are passed to the selected backend:

inference:
  model: balanced
  profiles:
    fast:
      model: hf:mlx-community/Llama-3.2-3B-Instruct-4bit
      temperature: 0.2
      top_k: 40
    balanced:
      model: granite-4.1-8b-instruct-Q6_K.gguf
      temperature: 0.7
      top_p: 0.9
      min_p: 0.05
      presence_penalty: 0.0
      frequency_penalty: 0.0
      repetition_penalty: 1.1

Download from HuggingFace

arkai model download <provider>:<model-reference>

Downloads the model into Arkai's local provider cache. Every download must use an explicit provider, such as hf:owner/model or ollama:model:tag. Some repositories contain files that llama.cpp can use directly; others must be converted to GGUF before they can be loaded locally. Large model repositories may require the repository's configured Git large-file support (git lfs) to materialize the weight files.

arkai model download hf:ibm-granite/granite-4.1-8b-instruct-GGUF
arkai model download ollama:llama3.2:latest

After downloading model, use the same provider-qualified reference for inference, for example arkai agent start -m ollama:llama3.2:latest.

Convert to GGUF

arkai model convert <provider>:<model-reference> [-q QUANTIZATION] [-o OUTPUT]

Converts a cached provider model to GGUF format and places it in the models directory. Use this only when the downloaded repository is not already supported directly by llama.cpp.

Flag	Default	Description
-q/--quantization	Q6_K	Quantization level
-o/--output	~/.local/share/arkai/models/MODEL-QUANTIZATION.gguf	Output path

Common quantization levels: Q4_K_M (smaller, faster), Q5_K_M, Q6_K (near original inference quality).

arkai model convert hf:ibm-granite/granite-4.1-8b-instruct-GGUF -q Q6_K

List models

arkai model list

Shows locally converted and provider models in Arkai's model cache.

Remove a model

arkai model remove <model-name>

Deletes a local GGUF file by filename (not full path).

arkai model remove granite-4.1-8b-instruct-Q6_K.gguf

Running an Inference Server

arkai inference manages the configured OpenAI-compatible inference server. The server runs in the background and stays running between agent sessions.

Start

arkai inference start [--model NAME] [--gpu-layers N] [--context N] [--port N]

Starts llama-server on localhost:8081 (default). CLI flags override the config values for that run only.

Flag	Description
--model NAME	Override model from config
--gpu-layers N	Override number of GPU layers (-1 = all)
--context N	Override context window size in tokens
--port N	Override port from config

Note: use hf:org/model as the model value to load a HuggingFace model.

GPU detection: arkai auto-detects Metal (Apple Silicon), CUDA (NVIDIA), ROCm (AMD), or falls back to CPU. Setting gpu_layers: -1 in config offloads all layers to GPU.

The inference server exposes an OpenAI-compatible API at http://127.0.0.1:<port>/v1.

MLX-LM backend

Install the optional backend and select an MLX model from Hugging Face:

pip install -e '.[mlx]'
arkai model download hf:mlx-community/Llama-3.2-3B-Instruct-4bit
arkai inference start --model hf:mlx-community/Llama-3.2-3B-Instruct-4bit

inference:
  backend: mlx
  model: hf:mlx-community/Llama-3.2-3B-Instruct-4bit
  port: 8081

Arkai resolves hf: models from its local provider cache before starting MLX-LM, so the server does not download an already cached model. gpu_layers and context_size are llama.cpp-specific settings and are not passed to MLX-LM.

Status

arkai inference status

Shows: running/stopped, PID, model, port, GPU layers, context size, health, and detected GPU type.

Stop

arkai inference stop

Terminates the server and cleans up PID and state files.

Running an Agent

arkai agent start is the primary command. It auto-starts the inference server if not already running, starts wtmcp (unless --no-mcp), launches the agent, and tears down services on exit.

arkai agent start [-a AGENT] [-m MODEL] [options]

Supported agents: opencode, crush, claude

Flags

Flag	Description
-a/--agent AGENT	Override agent from config
-m/--model MODEL	Override model from config
--no-mcp	Skip wtmcp entirely
--no-sandbox	Skip sandbox for this run
-s/--sandbox PROFILE	Use a specific sandbox profile for this run
-I/--keep-inference	Leave inference server running after exit
-M/--keep-mcp	Leave wtmcp running after exit
-v/--volume PATH	Mount extra volume in sandbox (repeatable)
-e/--env KEY=VALUE	Set env var in sandbox (repeatable)
--cwd PATH	Override directory mounted as cwd in sandbox

Note: use hf:org/model as the model value to load a HuggingFace model.

Lifecycle

On exit: wtmcp stops automatically. The inference server also stops unless --keep-inference is passed. Use --keep-inference when you want the server warm for the next session.

Managing Plugins (wtmcp)

wtmcp is an MCP server that exposes tools (plugins) to the agent over HTTP. arkai agent start manages wtmcp automatically, but you can also control it directly.

Enabling and disabling plugins

These commands edit .arkai.yaml in the current directory (must exist — run arkai config init first):

arkai wtmcp enable <plugin>    # add plugin to .arkai.yaml
arkai wtmcp disable <plugin>   # remove plugin from .arkai.yaml

Plugin list semantics

wtmcp.plugins in .arkai.yaml is a list of plugin names:

• Key absent: inherit plugins from user config
• plugins: []: explicitly disable all plugins
• plugins: [workspace, terminal]: use only these plugins

List plugins

arkai wtmcp list [--port N]

Shows all discovered plugins with status:

• 🟢 enabled (in config and discovered by wtmcp)
• ⚪ discovered but not enabled
• 🔴 enabled but not discovered by wtmcp

Requires a running wtmcp instance. If multiple instances are running, specify --port.

Server lifecycle

arkai wtmcp start [--port N] [--enable PLUGIN] [--disable PLUGIN]
arkai wtmcp stop  [--port N]
arkai wtmcp status [--port N]

--enable/--disable on start override the project config plugins for that run only. --port is required for stop and list when multiple instances are running.

Sandboxing

arkai can wrap agent execution in an arapuca sandbox. The sandbox restricts what the agent can access:

• Filesystem: only explicitly mounted paths are visible
• Network (Linux): only the inference server and wtmcp ports are reachable; all other network access is denied
• Network (macOS): baseline seccomp profile is applied

The current working directory is mounted read-write by default (override path with --cwd).

Profiles

Named profiles are presets stored in your user config under sandbox.profiles.<name>. When arkai agent start runs, it uses the active profile (or the root sandbox defaults if no profile is active).

Profile names must be alphanumeric + underscores. The names default and active are reserved.

Managing profiles

# List all profiles and which is active
arkai sandbox list

# Show full details of a profile
arkai sandbox show <profile-name>
arkai sandbox show default   # root sandbox defaults
arkai sandbox show active    # currently active profile

# Create a profile
arkai sandbox create <name> [--from PROFILE] [--memory MB] [--cpus N] \
    [--pids N] [--timeout S] [-v /path[:ro]] [-e KEY=VALUE]

# Delete a profile
arkai sandbox delete <name>

# Promote a profile's settings to root defaults
arkai sandbox set-default <name>

# Set the active profile (used by arkai agent)
arkai sandbox active <name>

# Clear the active profile (use root defaults)
arkai sandbox active

Example workflow

# Create a restricted profile for untrusted code
arkai sandbox create restricted --memory 1024 --cpus 1 --pids 128

# Make it the active profile
arkai sandbox active restricted

# Run the agent (uses restricted profile automatically)
arkai agent start

# Override to a different profile for one run
arkai agent start -s dev

Per-run overrides

These flags on arkai agent start override the active profile for that run only:

Flag	Effect
-s/--sandbox PROFILE	Use a specific profile
--no-sandbox	Disable sandbox entirely
-v/--volume PATH	Add an extra volume mount
-e/--env KEY=VALUE	Add an env var inside the sandbox
--cwd PATH	Override cwd mounted in sandbox

Configuration Reference

Configuration is YAML. Precedence (lowest to highest):

1. Built-in defaults
2. User config: ~/.config/arkai/arkai.yaml
3. Project config: .arkai.yaml in current directory
4. CLI flags

arkai config init              # scaffold .arkai.yaml in current directory
arkai config validate          # validate .arkai.yaml
arkai config validate --file PATH  # validate a specific file

agent

Key	Type	Default	Description
agent.name	string	opencode	Agent to use: opencode, crush, or claude
agent.path	string	(agent name)	Custom path to agent binary
agent.mcp	bool	true	Whether to start wtmcp for agent sessions

inference

Key	Type	Default	Description
inference.model	string	—	Local model filename, or hf:<repo> for a HuggingFace model
inference.profiles	mapping	—	Named model profiles; each profile has model and optional sampling settings
inference.backend	string	llama-cpp	llama-cpp or optional mlx
inference.path	string	backend-dependent	Path to llama-server or mlx_lm.server
inference.port	int	8081	Port for inference server (1024–65535)
inference.gpu_layers	int	-1	GPU layers to offload (-1 = all)
inference.context_size	int	65536	Context window size in tokens
inference.startup_timeout	int	600	Maximum seconds to wait for the server to become ready

The sampling settings can also be placed directly under inference to provide defaults for the selected model: temperature, top_p, top_k, min_p, presence_penalty, frequency_penalty, and repetition_penalty. Profile values override these defaults.

Model sampling settings are temperature, top_p, top_k, min_p, presence_penalty, frequency_penalty, and repetition_penalty. They are translated to the corresponding llama-server or mlx_lm.server CLI options.

Inference backends are registered command builders. The manager supplies the common model, port, GPU-layer, and context settings, then handles the process lifecycle and OpenAI-compatible REST API. Add a backend by implementing InferenceBackend.build_command() and registering it with register_backend().

wtmcp

Key	Type	Default	Description
wtmcp.path	string	wtmcp	Path to wtmcp binary
wtmcp.port	int	8080	Port for wtmcp server (1024–65535)
wtmcp.workdir	string	—	Working directory for wtmcp
wtmcp.plugins	list	(inherit)	Plugin list; [] = disable all; absent = inherit from user config

sandbox

Key	Type	Default	Description
sandbox.enabled	bool	true	Enable sandboxing
sandbox.path	string	arapuca	Path to arapuca binary
sandbox.memory_mb	int	2048	Memory limit in MB
sandbox.cpus	int	2	CPU count
sandbox.pids	int	256	PID limit
sandbox.timeout	int	0	Timeout in seconds (0 = no limit)
sandbox.volume	list	[]	Extra volume mounts (/path or /path:ro)
sandbox.environment	map	—	Extra environment variables passed into sandbox
sandbox.active_profile	string	—	Name of the active sandbox profile
sandbox.profiles.<name>	map	—	Named sandbox profile (same keys as above)

Example user config

agent:
  name: opencode

inference:
  backend: llama-cpp
  path: /usr/local/bin/llama-server
  port: 8081
  gpu_layers: -1
  context_size: 65536

wtmcp:
  path: /usr/local/bin/wtmcp
  port: 8080

sandbox:
  enabled: true
  path: /usr/local/bin/arapuca
  memory_mb: 2048
  cpus: 2
  pids: 256
  timeout: 0

Example project config

inference:
  model: granite-4.1-8b-instruct-Q6_K.gguf

wtmcp:
  plugins: [workspace, terminal, github]

License

Apache 2.0 — see LICENSE.