Skip to content

fix(provider): make fm usable by rewriting tool schemas to Apple's subset - #478

Merged
dimetron merged 1 commit into
mainfrom
fix/fm-tool-schemas
Nov 16, 2026
Merged

dimetron merged 1 commit into
mainfrom
fix/fm-tool-schemas

Conversation

@dimetron

Copy link
Copy Markdown
Owner

Summary

pi --model fm/system failed on every tool-using turn with nothing but:

OpenAI API error: 400 Bad Request

Two independent defects, both measured against a running fm serve.

1. Apple rejects the tool schemas

fm serve does not take a JSON Schema — it compiles one into a Swift GenerationSchema and rejects the whole request (400, naming no field) if any single tool uses a keyword outside that model. Bisecting all 40 of pi's tool declarations:

keyword response
type: ["null","boolean"] (git-overview) 400
oneOf 400
anyOf (all five tavily tools) 200 — but see below
additionalProperties: {} 400
$ref / $defs (agent_run) 400 "contains undefined references"
array with no items 400 "because it is missing"
pattern 500 "unsupported pattern"

Because the rejection covers the request rather than the tool, four of pi's own tools being affected made every tool-using turn fail.

fmSanitizeTools rewrites each schema into the accepted subset. It only removes what Apple rejects and fills in what Apple requires — a schema fm already accepted round-trips byte-identical, and no tool is ever dropped.

Direct proof — same five tools, same prompt that fits the window:

result
original schemas 400 couldn't be read…
sanitized schemas 200, correct tool call emitted

2. The 8K window is smaller than the default prompt

The budget is 8192 tokens (the server names it: "Provided 8,910 tokens, but the maximum allowed is 8,192"). Sanitized, the 40 declarations fit — 7,749 tokens with a short prompt, and the model answers.

What does not fit is pi-go's own default system prompt at 13,376 tokens on its own. So the coding agent's turn can never be sent here, however the schemas are rewritten. That request is now refused before the round trip, naming the limit, the per-part breakdown, and the largest part:

the request is roughly 22740 tokens, over the 8192-token context window of `fm serve`
(14625 for the system prompt, 7 for the transcript, 8108 for tool declarations).
Foundation Models is an on-device model with an 8K window: the system prompt is the
largest part, so pass a short one with --system, or use a different model for this session

Calibration

fmCharsPerToken is 4 because it was calibrated, not assumed. Divisors of 3 and 5 were both tried against the server and both misclassified real requests:

  • 3 refused a 40-tool request the server accepts (predicted 10,840 tokens against an actual 7,749) — it would have made fm unusable for work it can do.
  • 5 passed a request the server rejects (6,910 against an actual 8,249).

The one remaining disagreement errs toward refusing, which is the safe direction. TestFMGuardAgreesWithTheServer compares the guard's verdict with the server's on three real requests, and fails at both 3 and 5.

Retry classification

Context-window errors are now terminal. Waiting cannot change a byte count, and the token breakdown in fm's own message contains the substring "500", which matched a transient pattern — so a permanent overflow spent the entire five-retry backoff (~35s) before reporting a failure the user had to fix by changing the model. It now reports in ~9s.

Tests

Fixtures are the 40 declarations captured from a real request (testdata/fm-tools.json), not hand-written schemas. The defect was that simple schemas passed while real ones failed, so a fixture written to illustrate the bug would have gone green either way.

Each guard was verified to fail when its fix is reverted:

revert guard that fails
remove fmSanitizeTools call TestFMPrepareRequestSanitizesEveryTool — 4 keyword leaks
disable the window check TestFMGenerateContentRefusesOversizedRequest
remove terminal patterns TestIsTransient/fm_window_refusal
divisor 3 or 5 TestFMGuardAgreesWithTheServer

Verification

  • go test ./... — 70 packages, 0 failures
  • golangci-lint run ./internal/... ./hack/... — 0 issues
  • go build ./... && go vet — clean
  • End to end: pi ping --model fm/system still works (tool-less path unaffected)

One note: internal/cli's TestExecute hangs — reproduced on untouched main (EXIT=124), so pre-existing and unrelated; excluded to run the rest of that package.

…bset

Selecting fm/system failed on every tool-using turn with

  OpenAI API error: 400 Bad Request

and nothing else. Two independent defects, both measured against a running
`fm serve` rather than inferred from documentation.

The schema wall. fm does not accept a JSON Schema: it compiles one into a
Swift GenerationSchema and rejects the whole request — 400, naming no field —
if any single tool uses a keyword outside that model. Bisecting pi's 40 tool
declarations found six such keywords: type unions (["null","boolean"] in
git-overview), oneOf, anyOf (all five tavily tools), additionalProperties: {},
$ref/$defs (agent_run), array without items, and pattern (which is a 500, not
a 400). Because the rejection covers the request rather than the tool, four of
pi's own tools being affected made every tool-using turn fail.

fmSanitizeTools rewrites each schema into the accepted subset. The
transformation only removes what Apple rejects and fills in what Apple
requires: a schema fm already accepted round-trips byte-identical, and no tool
is ever dropped — a tool the model is never offered is a capability the user
loses silently. Verified directly: the same five tools with the same prompt
that fits the window are 400 with the original schemas and 200 sanitized.

The window wall. The budget is 8192 tokens, which the server names when it
rejects ("Provided 8,910 tokens, but the maximum allowed is 8,192"). Sanitized,
the 40 declarations fit — 7,749 tokens with a short prompt, and the model
answers with a correct tool call. What does not fit is pi-go's own default
system prompt at 13,376 tokens, so the coding agent's turn can never be sent
here however the schemas are rewritten. A request that cannot fit is now
refused before the round trip, naming the limit, the per-part breakdown, and
the largest part, instead of returning a bare 400.

fmCharsPerToken is 4 because it was calibrated, not assumed. Divisors of 3 and
5 were both tried against the server and both misclassified real requests: 3
refused a 40-tool request the server accepts (predicting 10,840 tokens against
an actual 7,749), and 5 passed one it rejects. The one remaining disagreement
errs toward refusing, which is the safe direction. TestFMGuardAgreesWithTheServer
compares the guard's verdict with the server's on three real requests; it fails
at 3 and at 5.

Context-window errors are also now terminal in internal/retry. Waiting cannot
change a byte count, and the token breakdown in fm's own message contains the
substring "500", which matched a transient pattern — so a permanent overflow
spent the entire five-retry backoff before reporting a failure the user had to
fix by changing the model.

Tests use the 40 tool declarations captured from a real request
(testdata/fm-tools.json) rather than hand-written schemas. The defect was that
simple schemas passed while real ones failed, so a fixture written to
illustrate the bug would have gone green either way. Each guard was verified to
fail when its fix is reverted.

Signed-off-by: pi <dimetron@me.com>
@dimetron
dimetron force-pushed the fix/fm-tool-schemas branch from 36831bd to eb9307c Compare November 16, 2026 03:16
@codecov

codecov Bot commented Nov 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.36364% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.50%. Comparing base (0623d8f) to head (eb9307c).

Files with missing lines Patch % Lines
internal/provider/fm.go 82.27% 14 Missing ⚠️
internal/provider/fm_tools.go 89.69% 10 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #478      +/-   ##
==========================================
- Coverage   91.50%   91.50%   -0.01%     
==========================================
  Files         393      394       +1     
  Lines       50434    50610     +176     
==========================================
+ Hits        46148    46309     +161     
- Misses       4280     4295      +15     
  Partials        6        6              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dimetron
dimetron merged commit 8db35a1 into main Nov 16, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant