fix(ci): repair Forge CI without changing contracts, tests or deps - #1416
Merged
Conversation
Three independent problems kept the four Forge jobs red. All three fixes are confined to CI configuration. 1. `.gitmodules` declared `branch = v1.5.0` for lib/forge-std and `branch = v0.0.84` for lib/solady. Both values are tags, not branches. With no foundry.lock committed, `forge install` treats every submodule as out of sync and checks it out at its recorded identifier; a `branch` line makes that identifier the literal string, so forge runs `git checkout v1.5.0` against the depth-1 submodule clone actions/checkout produces, which carries no tags. Dropping both lines leaves the recorded commits as the only pin. All ten resolved gitlinks are unchanged. 2. Every job installed `version: nightly`, so results moved with the toolchain rather than with the code. On this commit the suite is 482/482 on v1.3.6 and v1.4.4, 481/482 on v0.3.0 and v1.4.0, and 475/482 on v1.5.1. Pin v1.4.4, the newest release that is fully green. 3. forge-coverage was the only Forge job missing the two precompile steps, so tests that `vm.getCode` from optimized-out could not find it. It also set no FOUNDRY_PROFILE, and `_getStorageSlotForOrderHash` falls back to "optimized" in that case, asserting one packed storage read while coverage runs unoptimized source that does four. Adding the precompiles and an explicit profile takes that job from 22 failures to 482/482. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_nudgeAddressIfProblematic` exists to move a fuzzed recipient off any address that rejects a bare native transfer. It probed with a 1 wei call and, on failure, returned `_address + 1` without probing again. The two addresses that reject are adjacent: 0x09 (blake2f) and, because evm_version is cancun, 0x0a (point evaluation). An input of 0x09 therefore nudged onto 0x0a and stayed broken, and the native transfer to it reverted with a PrecompileError, surfacing as DifferentialTestAssertionFailed. That is the source of the intermittent red on the forge and forge-ref jobs going back to 2025: run 32660603782 hit it at fuzz run 279 of 1000 with considerationRecipient 0x09 and shouldIncludeNativeConsideration true. Step clear of the precompile range before probing rather than looping on the probe. A failing precompile call consumes all the gas forwarded to it, and `call(gas(), ...)` forwards all but a 64th, so a second failed probe would leave the test with almost no gas. Guarding first keeps the single probe and its existing gas profile. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both helpers that coerce a fuzzed address into a usable recipient probed it with all remaining gas: `isErc1155Receiver` via `to.call(...)` and `_nudgeAddressIfProblematic` via `call(gas(), ...)`. A fuzzed address can be any contract already in state, including the canonical CREATE2 deployer at 0x4e59b44847b379578588920cA78FbF26c0B4956C that foundry pre-deploys. It has 69 bytes of code, so it gets probed, and it reads whatever calldata it receives as a salt plus initcode and attempts a CREATE2. The first probe deploys an empty contract; a second probe with the same calldata collides, and a failed CREATE2 consumes every bit of gas forwarded to it. Measured on this configuration: the first call costs 32834 gas and the second costs 1040389983, the entire budget, so the test dies with OutOfGas. That is why raising gas_limit did not help. A larger budget is simply a larger amount to burn; at 2^34 the failure rate did not improve. Capping the probe at 100000 gas bounds the damage without changing any legitimate path, since a real onERC1155Received or native transfer costs far less. Across the conduit and zone suites with the fuzz cache cleared between runs, this moves 15 trials from roughly a fifth to a half failing to 15 of 15 passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jgriffos
approved these changes
Aug 24, 2026
BCLeFevre
approved these changes
Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Every Forge job in Seaport Test CI now fails on the "Install forge dependencies" step, before any contract compiles:
.gitmodulesdeclaredbranch = v1.5.0forlib/forge-stdandbranch = v0.0.84forlib/solady. Neither value is a branch. Both are tags, and both still exist upstream, so no ref was deleted.forge installreads.gitmodulesto build afoundry.lock. No lockfile is committed here, so every submodule is treated as out of sync and forge checks each one out at its recorded identifier. A submodule carrying abranchline becomesDepIdentifier::Branch { name }, andcheckout_id()returns that literal string, so forge runsgit checkout v1.5.0.actions/checkoutclones submodules at depth 1 and fetches no tags, so the pathspec does not resolve and the step exits 1. Without abranchline the submodule becomesDepIdentifier::Rev { rev }, and forge checks out the recorded commit, which is present in the shallow clone.Which of the two pins gets reported varies, because the out-of-sync set is a hash map with no fixed iteration order. CI has surfaced
v1.5.0; reproducing locally surfacedv0.0.84first. Both had to go.The forge-std pin was also stale in content, not only wrong in kind. The recorded gitlink is
cffb562, a May 2023 commit that sits between v1.5.x and v1.6.0, and not v1.5.0's commit (c223685). Nothing was resolving that label; it was only breaking the install. The solady gitlink does match its v0.0.84 tag.This is newer than it looks, and it is not the only thing wrong
The install break is recent. Reading job-level history for
test.ymlback to the oldest run GitHub still retains shows three separate stories rather than one:forge-offererspassed as recently as 2026-04-14 and only started failing once the install step broke, somewhere between then and 2026-06-20. So thebranchpins are what took the last reliably green Forge job down.The other three were already red before that, for reasons that have nothing to do with submodules.
forge-coveragehas never passed in the entire retained window.forgeandforge-refwere intermittent through late 2025 and went consistently red around January 2026. On commit08013390, which is still the head ofmain, run 18134722144 from 2025-09-30 recordedforgeas passing andforge-refas failing, so the code has not changed underneath these jobs; the toolchain has.Both jobs install
version: nightly, so every run picks up whatever Foundry shipped that morning.Solution
Three CI changes plus two test-helper fixes. No contract source, dependency, or
foundry.tomlchange.Drop the two
branchlines so the recorded submodule commits are the only pin. That is what the repo was already building against, so this changes which ref forge asks git for, not which code lands inlib/. All ten resolved commits are byte for byte identical to the gitlinks onmain.Pin the Foundry toolchain to v1.4.4 instead of tracking
nightly, so the toolchain stops drifting and a green run stays green.Give
forge-coveragethe two precompile steps the other three Forge jobs already run, and name the Foundry profile on its command. Both are gaps in that job's own configuration, not changes to what it asserts.Fix two test helpers that mishandle a fuzzed address. Between them they account for the long-standing intermittency on
forgeandforge-ref, and they are the only changes here that touch test files.Verification
Reproduced against forge 1.8.0-nightly (
e469863), the exact build CI installed on 2026-08-22, using a clone that replicatesactions/checkout(--depth=1plusgit submodule update --init --force --depth=1 --recursive).forge installpathspec 'v0.0.84' did not matchFOUNDRY_PROFILE=reference forge buildFOUNDRY_PROFILE=optimized forge buildDependency resolution is unchanged. All ten recorded gitlinks resolve to the same commits before and after the edit:
Choosing the pin
Each row below is a full run of all three test jobs on this commit, with
FOUNDRY_PROFILE=reference forge buildandFOUNDRY_PROFILE=optimized forge buildre-run under that same version first. That precompile matters: the tests load bytecode fromoptimized-out, so reusing another version's artifacts gives meaningless numbers.Green is not monotonic, so this is a sweep rather than a bisect. v1.4.4 is the newest release that passes everything, which is why it is the pin.
The failing sets are disjoint, which is what shows these are toolchain behavior changes rather than contract bugs. v1.5.1's seven failures are three conduit
OutOfGastests and four zone differential tests, andtestFulfillOrderRevertCounterIncrementedpasses there. v0.3.0 inverts it exactly: those seven pass and onlytestFulfillOrderRevertCounterIncrementedfails. No repository content differs between those two runs.forge-coverage
forge coverageruns a different path from the other three jobs:SEAPORT_COVERAGE=truemakes the test bases deploy from source with the optimizer off instead of loading precompiled optimized bytecode. Under v1.4.4 it started at 22 failures in three groups.mainFOUNDRY_PROFILE=defaultFive of the original failures were
vm.getCode: failed to read .../optimized-out/TransferHelper.sol/TransferHelper.json. This job is the only Forge job that never ran the precompile steps, sooptimized-outdid not exist for the tests that read from it.The remaining sixteen were
Expected a different number of read accessesinFuzzCoverage,FuzzInscribersandFuzzMain._getStorageSlotForOrderHashexpects one storage read under theoptimized,test,liteandreferenceprofiles and four otherwise, and reads the profile withvm.envOr("FOUNDRY_PROFILE", "optimized"). The job set no profile, so the fallback asserted optimized-bytecode behavior (one packed read) while the run executed unoptimized source (four reads). Naming the profile explicitly lets the existing branch resolve correctly.The intermittent failures on
forgeandforge-refThese two jobs did not fail outright, they alternated. That is why the version sweep above cannot be read as proof on its own: for the affected suites a single run samples a coin flip, and some of those rows were lucky draws. Two separate defects were behind it, both in helpers that coerce a fuzzed address into a usable recipient.
A single nudge can land on another bad address.
_nudgeAddressIfProblematicprobes an address with a 1 wei call and, on failure, returns_address + 1without probing again. Measuring which low addresses actually reject such a transfer here:The two that reject are adjacent, so 0x09 nudges onto 0x0a and stays broken; the native transfer then reverts with a
PrecompileError, surfacing asDifferentialTestAssertionFailed. CI run 32660603782 hit exactly this, withconsiderationRecipient: 0x09andshouldIncludeNativeConsideration: true, at fuzz run 279 of 1000. The fix steps clear of the precompile range before probing.Both helpers forwarded all remaining gas to an arbitrary address.
isErc1155Receiverprobes withto.call(...)and_nudgeAddressIfProblematicwithcall(gas(), ...). A fuzzed address can be any contract already in state, including the canonical CREATE2 deployer at0x4e59b44847b379578588920cA78FbF26c0B4956Cthat foundry pre-deploys. It has 69 bytes of code, so it gets probed, and it reads whatever calldata it receives as a salt plus initcode and attempts a CREATE2. The first probe deploys an empty contract; a second probe with the same calldata collides, and a failed CREATE2 consumes every bit of gas forwarded to it:CI run 32662099214 died this way in
ConduitExecuteWithBatch1155, and the trace shows the collision frame consuming 1,020,748,435 gas before the run fell over. Raisinggas_limitis not a fix; a larger budget is simply a larger amount to burn, and at 2^34 the failure rate did not improve. Capping the probe at 100000 gas bounds it without touching any legitimate path, since a realonERC1155Receivedor native transfer costs far less.Both conduit suites are affected because they share
BaseConduitTest, and the deployer address appears in every counterexample collected, local and CI.Measured over the conduit and zone suites with the fuzz cache cleared between runs: before, 4 of 8 and then 2 of 10 full runs failed; after, 15 of 15 passed. The guarded nudge helper was additionally checked to return an address above
0x0athat accepts a transfer for all 256 sequential inputs from0x01and for 256 fuzzed inputs.What this does not fix
Nothing here addresses why the tests are sensitive to the Foundry version in the first place. The conduit and zone tests that fail on v1.5.1 and the
vm.expectRevert/vm.rollbehavior behindtestFulfillOrderRevertCounterIncrementedon v0.3.0 and v1.4.0 are still latent; the pin sidesteps them rather than resolving them. Bumping past v1.4.4 will surface them again and should be done deliberately, with the suite run against the candidate version first.The pin also freezes out newer solc and analysis improvements. That seemed the right trade for a repo whose contracts are deployed and immutable, but it is a choice worth revisiting if Seaport picks up active development again.