[Design] Non-text MCP tool result artifacts: modality-aware routing of arbitrary-MIME content to models and users

[Aaronontheweb]

Summary

Design capability: define how NetClaw handles non-text content returned by MCP tool calls. An MCP server may return arbitrary MIME-typed byte artifacts (images, PDFs, archives, audio, video, vendor-specific types), not just text. Today NetClaw projects every MCP tool result to a single string for the text model, and it has no defined behavior for anything that is not plain text.

Issue #2051 is a specific instance of this gap manifesting today (a successful multi-content image+text result is projected to its collection type name). It is the narrow, shippable slice; this issue is the broader design that any durable fix should fit inside.

Problem space

1. Arbitrary MIME types. MCP content blocks can carry any media type via DataContent (image/png, application/pdf, application/zip, audio/*, vendor types). NetClaw must decide how to treat each — not just PNGs.
2. Text-only default vs. multimodal. The netclaw-model-capabilities spec establishes a text-only default with a ModelModality type system. Whether an artifact is readable by the model is a real, per-artifact decision.
3. The deliverable is the user. When the model cannot see an artifact, the sensible behavior is to pass it through to the end user in the session, the same way pasted files and attach_file output are delivered today.
4. Existing machinery. NetClaw already classifies arbitrary inbound media via netclaw-media (MimeTypeCatalog, AttachmentCategory, MediaKind, SupportsModelInput) and delivers attachments through the session-output pipeline (Outputs.AddFileAttachment). MCP-returned artifacts are not meaningfully different from inbound attachments (a byte blob + a MIME type, no filename). The design should route MCP artifacts through this same surface rather than inventing a parallel one.

Scope questions to settle in the spec

• Projection for the model. How is a non-text block rendered in the string the text model sees? (Marker naming the MIME type, e.g. [image: image/png]; not raw base64 into the text window.)
• Routing by modality. Does an artifact go to the model (when it supports that modality), to the user, or both? What decides?
• Delivery seam. Materialize MCP-returned byte artifacts and deliver them through the existing attachment pipeline so end users can open them.
• Unknown / binary safety. Unknown or binary MIME types must never be inlined into the text window; they attach or are marked.
• Size and safety bounds. Bounded egress / tool-output caps; no unbounded artifact pulled into context.

Required output

• An OpenSpec change (this is a design/capability task, not a point bug). Likely home: the netclaw-mcp spec, drawing on netclaw-model-capabilities and netclaw-media.
• Acceptance criteria and regression coverage, including a deterministic smoke MCP server that returns a static image+text result and an end-to-end test proving it propagates through the session to the user.

Relationship to #2051

• MCP multi-content tool result renders as literal type name (AIContent[]) instead of its content #2051 = the formatter projection bug (the "model sees a collection type name" failure). It is the first, independently shippable slice and stays narrow.
• This issue = the parent design that defines the complete, correct behavior for non-text MCP artifacts.

Labels

Parent design issue for non-text MCP artifact handling. Tracks #2051 and the OpenSpec work.