Skip to content
139 changes: 139 additions & 0 deletions .pr/design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
# Per-call `llm_profile` on the task tool

## Problem

The task tool can delegate work to a subagent, but an inherited subagent always
uses the parent conversation's LLM. A parent that wants a mixed-model workflow
must either switch its own model or rely on a static `model:` value in an agent
definition. Neither supports choosing a worker profile for an individual task
while the parent keeps its current model.

## User-facing behavior

`TaskAction` gains an optional `llm_profile` field naming a saved LLM profile.

An inherited agent uses the task call's `llm_profile` when supplied and otherwise
uses the parent model. Supplying `llm_profile` for an agent definition that already
selects a model is a configuration conflict and fails explicitly; neither choice
is silently discarded.

For resumable tasks, a bare resume retains the task's effective per-call profile;
an explicit profile on a resume replaces it. Resume remains scoped to the current
in-memory `TaskManager` lifecycle.

Unknown profiles fail loudly and become an error `TaskObservation`. The task is
not partially registered and the worker never silently falls back to the parent
model.

## Shared profile-loading boundary

`LocalConversation.load_profile_llm()` is the common cipher-aware loading
primitive for conversation-owned profile operations. It selects the
conversation's default store or a caller-supplied profile directory, then calls
`LLMProfileStore.load(profile_name, cipher=self._cipher)`.

The operation only loads. It does not activate the returned LLM, add it to the
parent conversation's registry, or bind parent conversation context.

The existing higher-level methods add their own behavior:

- `switch_profile()` loads through the primitive, assigns the canonical
`profile:<name>` usage ID, and activates the LLM;
- `get_or_create_profile_llm()` returns a registry hit or loads through the
primitive, assigns the caller's usage ID, registers the LLM, and binds the
conversation context;
- `TaskManager` loads through the primitive, resets metrics, and passes the LLM
into the worker factory. The worker `LocalConversation` then owns registration
and context binding.

Using `get_or_create_profile_llm()` directly for a worker would register an
otherwise unused template LLM in the parent conversation and give it the wrong
metrics/context owner. Adding a mode flag to that method would also make its
registry-oriented contract ambiguous.

`LLMProfileStore.load()` already performs persisted subscription restoration
through `LLM.from_persisted()`. The loading primitive therefore returns the
runtime subscription LLM while still avoiding parent registration and
activation.

The store's native error contract is preserved: a missing profile raises
`FileNotFoundError` with the available profile files, while invalid or corrupted
profiles raise `ValueError`. There is no separate list-before-load operation, so
validation and reading happen through the store's locked load path.

## Worker construction and metrics

The selected LLM is injected before `factory_func` runs. This matters because a
factory may derive a default condenser from its input LLM; swapping afterward
would leave the condenser on the parent model.

Subscription-backed LLMs retain the factory-created condenser. Subscription
completion dispatch supports condenser calls, matching top-level agent creation
and profile switching.

The loaded LLM gets a fresh metrics object before factory construction. It is
never registered in the parent's `llm_registry`. The worker conversation tracks
its own agent and condenser usage, and the existing task completion path copies
the worker's combined metrics into the parent under `task:<task-id>` exactly
once. The parent's active model is unchanged.

## Profile discovery and confidentiality

The task tool description lists saved profile names using the same public
`get_llm_profile_names()` and `format_llm_profiles()` helpers as `switch_llm`.
It exposes names only: profile model IDs, provider URLs, API keys, and persisted
JSON are not included. When no profiles exist, the section is omitted.

The list is a creation-time snapshot of the default profile store. When any
registered agent specifies a custom `profile_store_dir`, the task tool omits the
list rather than advertising names from a store that the selected agent may not
use.

## Persistence and compatibility

`Task.llm_profile` stores the effective per-call profile for resume. Existing task
calls omit the optional field and continue inheriting the parent model.

The additive action field has the repository's normal version-skew caveat: an old
SDK that forbids unknown action fields cannot deserialize a new event containing
`llm_profile`.

## Verification matrix

Focused tests cover:

- two sequential tasks choosing different saved profiles;
- parent model unchanged and no worker profile entry in the parent registry;
- independent worker metrics and `task:<id>` merge-back;
- encrypted-secret loading through the conversation cipher;
- persisted subscription restoration;
- subscription workers retain their supported LLM-backed condenser;
- custom profile directories;
- native missing-profile errors with no partial task state;
- explicit conflicts between definition-owned models and per-call profiles;
- bare-resume retention and explicit resume replacement;
- a clear missing-profile failure when a stored profile is removed before resume;
- inherited behavior when no override is supplied;
- pre-factory injection and `stream=False` worker behavior;
- profile names present while model IDs, provider URLs, and API keys remain absent
from the tool description;
- omission of the profile section when no profiles exist.
- omission of default-store profile names when a registered agent uses a custom
profile store.

Current verification on the rebased branch:

- 235 task, subagent, profile-switch, and conversation-switch tests pass;
- pre-commit passes on every revised file, including Pyright;
- a live encrypted-profile delegation completed with a MiniMax M3 parent and a
MiMo v2.5 Pro worker. The parent remained on MiniMax, its registry contained no
worker profile entry, its only metrics key was `task:task_00000001`, and a bare
resume retained the encrypted worker profile and MiMo model.

## Out of scope

- changing the `switch_llm` or subagent defaults;
- enabling parallel task execution by default;
- adding frontend worker-model reporting;
- changing definition-level profile loading;
- changing the conversation-spawn workflow.
Original file line number Diff line number Diff line change
Expand Up @@ -1685,7 +1685,7 @@ def switch_profile(self, profile_name: str) -> None:
try:
cached = self.llm_registry.get(usage_id)
except KeyError:
loaded = self._profile_store.load(profile_name, cipher=self._cipher)
loaded = self.load_profile_llm(profile_name)
cached = loaded.model_copy(update={"usage_id": usage_id})
self.switch_llm(cached)

Expand All @@ -1700,13 +1700,40 @@ def get_or_create_profile_llm(self, profile_name: str, usage_id: str) -> LLM:
try:
return self.llm_registry.get(usage_id)
except KeyError:
loaded = self._profile_store.load(profile_name, cipher=self._cipher)
loaded = self.load_profile_llm(profile_name)
llm = loaded.model_copy(update={"usage_id": usage_id})
llm = create_subscription_llm_from_config(llm)
self.llm_registry.add(llm)
self._bind_conversation_context(llm)
return llm

def load_profile_llm(
Comment thread
georgeglarson marked this conversation as resolved.
self, profile_name: str, profile_store_dir: str | None = None
) -> LLM:
"""Load a saved profile LLM without registering or activating it.

The returned LLM is not added to ``llm_registry`` or made active, so the
caller owns its lifecycle and metrics attribution. Persisted subscription
profiles are restored by :class:`LLMProfileStore` during the load.
Secrets are decrypted with the conversation's cipher.

Args:
profile_name: Name of a profile previously saved via
LLMProfileStore (a ``.json`` suffix is tolerated).
profile_store_dir: Optional override for the store directory. When
None, the conversation's default store is used.

Raises:
FileNotFoundError: If the profile does not exist.
ValueError: If the profile is corrupted or invalid.
"""
store = (
self._profile_store
if profile_store_dir is None
else LLMProfileStore(profile_store_dir)
)
return store.load(profile_name, cipher=self._cipher)

def switch_acp_model(self, model: str) -> None:
"""Switch the model on an ACP conversation.

Expand Down
2 changes: 2 additions & 0 deletions openhands-sdk/openhands/sdk/subagent/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,8 @@ Supported YAML frontmatter keys (see `AgentDefinition.load` in `schema.py`):
- `skills` (default: `[]`): a comma-separated string or a list of skill names
- `model` (default: `inherit`): `inherit` reuses the parent LLM; another value is
loaded as an LLM profile name from `profile_store_dir` or the default profile store
- the task tool's per-call `llm_profile` applies only to `model: inherit`
definitions; supplying it for a definition that selects a model is an error
- `color` (optional)
- `max_iteration_per_run` (optional, positive integer)
- `max_budget_per_run` (optional, positive number in USD)
Expand Down
5 changes: 3 additions & 2 deletions openhands-sdk/openhands/sdk/tool/builtins/switch_llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,8 @@ def get_llm_profile_names() -> list[str]:
return [summary["name"] for summary in LLMProfileStore().list_summaries()]


def _format_profiles(profile_names: Sequence[str]) -> str:
def format_llm_profiles(profile_names: Sequence[str]) -> str:
"""Format saved LLM profile names as a bullet list for tool descriptions."""
if not profile_names:
return "- No saved LLM profiles are currently available."
return "\n".join(f"- {name}" for name in sorted(profile_names))
Expand Down Expand Up @@ -164,7 +165,7 @@ def create(
return [
cls(
description=_DESCRIPTION_TEMPLATE.format(
profiles=_format_profiles(profile_names)
profiles=format_llm_profiles(profile_names)
),
action_type=SwitchLLMAction,
observation_type=SwitchLLMObservation,
Expand Down
32 changes: 30 additions & 2 deletions openhands-tools/openhands/tools/task/definition.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@
ToolDefinition,
register_tool,
)
from openhands.sdk.tool.builtins.switch_llm import (
format_llm_profiles,
get_llm_profile_names,
)


if TYPE_CHECKING:
Expand All @@ -48,6 +52,14 @@ class TaskAction(Action):
default="general-purpose",
description="The type of specialized agent to use for this task.",
)
llm_profile: str | None = Field(
default=None,
description=(
"Saved LLM profile for this subagent. If omitted, the subagent "
"inherits the parent model. Do not supply this when the selected "
"agent definition already specifies a model profile."
),
)
resume: str | None = Field(
default=None,
description="Task ID of the task to resume from.",
Expand Down Expand Up @@ -117,7 +129,7 @@ def to_llm_content(self) -> Sequence[TextContent | ImageContent]:
Available agent types and the tools they have access to:
{agent_types_info}

When NOT to use the task tool:
{llm_profiles_section}When NOT to use the task tool:
Comment thread
georgeglarson marked this conversation as resolved.
- A single grep, find, or cat command would answer your question — just run it yourself
- You are making a file edit (use file_editor directly)
- You already have the context needed
Expand Down Expand Up @@ -235,13 +247,29 @@ def create(

agent_types_info = get_factory_info()

registered = {d.name for d in get_registered_agent_definitions()}
agent_definitions = get_registered_agent_definitions()
has_custom_profile_store = any(
definition.profile_store_dir is not None for definition in agent_definitions
)
profile_names = [] if has_custom_profile_store else get_llm_profile_names()
llm_profiles_section = ""
if profile_names:
llm_profiles_section = (
"To run a subagent on a different model than your own, pass "
"llm_profile with one of these saved LLM profiles:\n"
f"{format_llm_profiles(profile_names)}\n"
"This affects only the delegated subagent; "
"your own model is unchanged.\n\n"
)

registered = {definition.name for definition in agent_definitions}
task_tool_examples = "\n".join(
ex for name, ex in TASK_TOOL_EXAMPLES.items() if name in registered
)

task_description = TASK_TOOL_DESCRIPTION.format(
agent_types_info=agent_types_info,
llm_profiles_section=llm_profiles_section,
task_tool_examples=task_tool_examples,
)

Expand Down
1 change: 1 addition & 0 deletions openhands-tools/openhands/tools/task/impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ def __call__(
description=action.description,
resume=action.resume,
conversation=conversation,
llm_profile=action.llm_profile,
)
match task.status:
case TaskStatus.COMPLETED:
Expand Down
Loading
Loading