Repository navigation
fix(provider): make fm usable by rewriting tool schemas to Apple's subset - #478
Merged
Merged
Conversation
…bset
Selecting fm/system failed on every tool-using turn with
OpenAI API error: 400 Bad Request
and nothing else. Two independent defects, both measured against a running
`fm serve` rather than inferred from documentation.
The schema wall. fm does not accept a JSON Schema: it compiles one into a
Swift GenerationSchema and rejects the whole request — 400, naming no field —
if any single tool uses a keyword outside that model. Bisecting pi's 40 tool
declarations found six such keywords: type unions (["null","boolean"] in
git-overview), oneOf, anyOf (all five tavily tools), additionalProperties: {},
$ref/$defs (agent_run), array without items, and pattern (which is a 500, not
a 400). Because the rejection covers the request rather than the tool, four of
pi's own tools being affected made every tool-using turn fail.
fmSanitizeTools rewrites each schema into the accepted subset. The
transformation only removes what Apple rejects and fills in what Apple
requires: a schema fm already accepted round-trips byte-identical, and no tool
is ever dropped — a tool the model is never offered is a capability the user
loses silently. Verified directly: the same five tools with the same prompt
that fits the window are 400 with the original schemas and 200 sanitized.
The window wall. The budget is 8192 tokens, which the server names when it
rejects ("Provided 8,910 tokens, but the maximum allowed is 8,192"). Sanitized,
the 40 declarations fit — 7,749 tokens with a short prompt, and the model
answers with a correct tool call. What does not fit is pi-go's own default
system prompt at 13,376 tokens, so the coding agent's turn can never be sent
here however the schemas are rewritten. A request that cannot fit is now
refused before the round trip, naming the limit, the per-part breakdown, and
the largest part, instead of returning a bare 400.
fmCharsPerToken is 4 because it was calibrated, not assumed. Divisors of 3 and
5 were both tried against the server and both misclassified real requests: 3
refused a 40-tool request the server accepts (predicting 10,840 tokens against
an actual 7,749), and 5 passed one it rejects. The one remaining disagreement
errs toward refusing, which is the safe direction. TestFMGuardAgreesWithTheServer
compares the guard's verdict with the server's on three real requests; it fails
at 3 and at 5.
Context-window errors are also now terminal in internal/retry. Waiting cannot
change a byte count, and the token breakdown in fm's own message contains the
substring "500", which matched a transient pattern — so a permanent overflow
spent the entire five-retry backoff before reporting a failure the user had to
fix by changing the model.
Tests use the 40 tool declarations captured from a real request
(testdata/fm-tools.json) rather than hand-written schemas. The defect was that
simple schemas passed while real ones failed, so a fixture written to
illustrate the bug would have gone green either way. Each guard was verified to
fail when its fix is reverted.
Signed-off-by: pi <dimetron@me.com>
dimetron
force-pushed
the
fix/fm-tool-schemas
branch
from
November 16, 2026 03:16
36831bd to
eb9307c
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #478 +/- ##
==========================================
- Coverage 91.50% 91.50% -0.01%
==========================================
Files 393 394 +1
Lines 50434 50610 +176
==========================================
+ Hits 46148 46309 +161
- Misses 4280 4295 +15
Partials 6 6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pi --model fm/systemfailed on every tool-using turn with nothing but:Two independent defects, both measured against a running
fm serve.1. Apple rejects the tool schemas
fm servedoes not take a JSON Schema — it compiles one into a SwiftGenerationSchemaand rejects the whole request (400, naming no field) if any single tool uses a keyword outside that model. Bisecting all 40 of pi's tool declarations:type: ["null","boolean"](git-overview)oneOfanyOf(all five tavily tools)additionalProperties: {}$ref/$defs(agent_run)arraywith noitemspatternBecause the rejection covers the request rather than the tool, four of pi's own tools being affected made every tool-using turn fail.
fmSanitizeToolsrewrites each schema into the accepted subset. It only removes what Apple rejects and fills in what Apple requires — a schema fm already accepted round-trips byte-identical, and no tool is ever dropped.Direct proof — same five tools, same prompt that fits the window:
400 couldn't be read…200, correct tool call emitted2. The 8K window is smaller than the default prompt
The budget is 8192 tokens (the server names it: "Provided 8,910 tokens, but the maximum allowed is 8,192"). Sanitized, the 40 declarations fit — 7,749 tokens with a short prompt, and the model answers.
What does not fit is pi-go's own default system prompt at 13,376 tokens on its own. So the coding agent's turn can never be sent here, however the schemas are rewritten. That request is now refused before the round trip, naming the limit, the per-part breakdown, and the largest part:
Calibration
fmCharsPerTokenis 4 because it was calibrated, not assumed. Divisors of 3 and 5 were both tried against the server and both misclassified real requests:The one remaining disagreement errs toward refusing, which is the safe direction.
TestFMGuardAgreesWithTheServercompares the guard's verdict with the server's on three real requests, and fails at both 3 and 5.Retry classification
Context-window errors are now terminal. Waiting cannot change a byte count, and the token breakdown in fm's own message contains the substring
"500", which matched a transient pattern — so a permanent overflow spent the entire five-retry backoff (~35s) before reporting a failure the user had to fix by changing the model. It now reports in ~9s.Tests
Fixtures are the 40 declarations captured from a real request (
testdata/fm-tools.json), not hand-written schemas. The defect was that simple schemas passed while real ones failed, so a fixture written to illustrate the bug would have gone green either way.Each guard was verified to fail when its fix is reverted:
fmSanitizeToolscallTestFMPrepareRequestSanitizesEveryTool— 4 keyword leaksTestFMGenerateContentRefusesOversizedRequestTestIsTransient/fm_window_refusalTestFMGuardAgreesWithTheServerVerification
go test ./...— 70 packages, 0 failuresgolangci-lint run ./internal/... ./hack/...— 0 issuesgo build ./... && go vet— cleanpi ping --model fm/systemstill works (tool-less path unaffected)One note:
internal/cli'sTestExecutehangs — reproduced on untouchedmain(EXIT=124), so pre-existing and unrelated; excluded to run the rest of that package.