Skip to content

fix(ci): repair Forge CI without changing contracts, tests or deps - #1416

Merged
ryanio merged 3 commits into
mainfrom
fix/forge-ci-submodule-refs
Aug 25, 2026
Merged

fix(ci): repair Forge CI without changing contracts, tests or deps#1416
ryanio merged 3 commits into
mainfrom
fix/forge-ci-submodule-refs

Conversation

@ryanio

@ryanio ryanio commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Motivation

Every Forge job in Seaport Test CI now fails on the "Install forge dependencies" step, before any contract compiles:

Updating dependencies in /home/runner/work/seaport/seaport/lib
Error: git checkout exited with code 1: error: pathspec 'v1.5.0' did not match any file(s) known to git

.gitmodules declared branch = v1.5.0 for lib/forge-std and branch = v0.0.84 for lib/solady. Neither value is a branch. Both are tags, and both still exist upstream, so no ref was deleted.

forge install reads .gitmodules to build a foundry.lock. No lockfile is committed here, so every submodule is treated as out of sync and forge checks each one out at its recorded identifier. A submodule carrying a branch line becomes DepIdentifier::Branch { name }, and checkout_id() returns that literal string, so forge runs git checkout v1.5.0. actions/checkout clones submodules at depth 1 and fetches no tags, so the pathspec does not resolve and the step exits 1. Without a branch line the submodule becomes DepIdentifier::Rev { rev }, and forge checks out the recorded commit, which is present in the shallow clone.

Which of the two pins gets reported varies, because the out-of-sync set is a hash map with no fixed iteration order. CI has surfaced v1.5.0; reproducing locally surfaced v0.0.84 first. Both had to go.

The forge-std pin was also stale in content, not only wrong in kind. The recorded gitlink is cffb562, a May 2023 commit that sits between v1.5.x and v1.6.0, and not v1.5.0's commit (c223685). Nothing was resolving that label; it was only breaking the install. The solady gitlink does match its v0.0.84 tag.

This is newer than it looks, and it is not the only thing wrong

The install break is recent. Reading job-level history for test.yml back to the oldest run GitHub still retains shows three separate stories rather than one:

Window forge forge-ref forge-offerers forge-coverage
2025-08 to 2025-09 flaky flaky pass fail
2026-01-17 to 2026-04-14 fail fail pass fail
2026-06-20 onward fail fail fail fail

forge-offerers passed as recently as 2026-04-14 and only started failing once the install step broke, somewhere between then and 2026-06-20. So the branch pins are what took the last reliably green Forge job down.

The other three were already red before that, for reasons that have nothing to do with submodules. forge-coverage has never passed in the entire retained window. forge and forge-ref were intermittent through late 2025 and went consistently red around January 2026. On commit 08013390, which is still the head of main, run 18134722144 from 2025-09-30 recorded forge as passing and forge-ref as failing, so the code has not changed underneath these jobs; the toolchain has.

Both jobs install version: nightly, so every run picks up whatever Foundry shipped that morning.

Solution

Three CI changes plus two test-helper fixes. No contract source, dependency, or foundry.toml change.

Drop the two branch lines so the recorded submodule commits are the only pin. That is what the repo was already building against, so this changes which ref forge asks git for, not which code lands in lib/. All ten resolved commits are byte for byte identical to the gitlinks on main.

Pin the Foundry toolchain to v1.4.4 instead of tracking nightly, so the toolchain stops drifting and a green run stays green.

Give forge-coverage the two precompile steps the other three Forge jobs already run, and name the Foundry profile on its command. Both are gaps in that job's own configuration, not changes to what it asserts.

Fix two test helpers that mishandle a fuzzed address. Between them they account for the long-standing intermittency on forge and forge-ref, and they are the only changes here that touch test files.

Verification

Reproduced against forge 1.8.0-nightly (e469863), the exact build CI installed on 2026-08-22, using a clone that replicates actions/checkout (--depth=1 plus git submodule update --init --force --depth=1 --recursive).

Step Before After
forge install exit 1, pathspec 'v0.0.84' did not match exit 0
FOUNDRY_PROFILE=reference forge build not reached exit 0
FOUNDRY_PROFILE=optimized forge build not reached exit 0

Dependency resolution is unchanged. All ten recorded gitlinks resolve to the same commits before and after the edit:

e282159d5170298eb2455a6c05280ab5a73a4ef0  lib/ds-test
cffb5628775fbf120bfc3c123149f6a6d99dc275  lib/forge-std
1d9566b908b9702c45d354a1caabe8ef5a69938d  lib/murky
5a00628ed3d6ce3154cee4d2cc93fad920e8ea30  lib/openzeppelin-contracts
1a0a4758cced1d3de962817cd37c2387fb290eea  lib/seaport-core
7b480ce2b5f258f283726c3a001b8bf7a8ae62fc  lib/seaport-sol
b72493221ee1d2f2fb30ed94a3cc535a9028d09f  lib/seaport-types
3d57528984275d1746ee6597acd36277f51c091d  lib/solady
4c3b8ff8e90c8cd11d30e02c1b6b2fcf9bc0f3db  lib/solarray
1b3adf677e7e383cc684b5d5bd441da86bf4bf1c  lib/solmate

Choosing the pin

Each row below is a full run of all three test jobs on this commit, with FOUNDRY_PROFILE=reference forge build and FOUNDRY_PROFILE=optimized forge build re-run under that same version first. That precompile matters: the tests load bytecode from optimized-out, so reusing another version's artifacts gives meaningless numbers.

Foundry forge forge-ref forge-offerers
v0.3.0 481/482 481/482 39/39
v1.3.6 482/482 482/482 39/39
v1.4.0 481/482 481/482 39/39
v1.4.4 482/482 482/482 39/39
v1.5.1 475/482 475/482 39/39

Green is not monotonic, so this is a sweep rather than a bisect. v1.4.4 is the newest release that passes everything, which is why it is the pin.

The failing sets are disjoint, which is what shows these are toolchain behavior changes rather than contract bugs. v1.5.1's seven failures are three conduit OutOfGas tests and four zone differential tests, and testFulfillOrderRevertCounterIncremented passes there. v0.3.0 inverts it exactly: those seven pass and only testFulfillOrderRevertCounterIncremented fails. No repository content differs between those two runs.

forge-coverage

forge coverage runs a different path from the other three jobs: SEAPORT_COVERAGE=true makes the test bases deploy from source with the optimizer off instead of loading precompiled optimized bytecode. Under v1.4.4 it started at 22 failures in three groups.

Configuration Result
as on main 22 failed
add the two precompile steps 16 failed
also set FOUNDRY_PROFILE=default 482/482, exit 0

Five of the original failures were vm.getCode: failed to read .../optimized-out/TransferHelper.sol/TransferHelper.json. This job is the only Forge job that never ran the precompile steps, so optimized-out did not exist for the tests that read from it.

The remaining sixteen were Expected a different number of read accesses in FuzzCoverage, FuzzInscribers and FuzzMain. _getStorageSlotForOrderHash expects one storage read under the optimized, test, lite and reference profiles and four otherwise, and reads the profile with vm.envOr("FOUNDRY_PROFILE", "optimized"). The job set no profile, so the fallback asserted optimized-bytecode behavior (one packed read) while the run executed unoptimized source (four reads). Naming the profile explicitly lets the existing branch resolve correctly.

The intermittent failures on forge and forge-ref

These two jobs did not fail outright, they alternated. That is why the version sweep above cannot be read as proof on its own: for the affected suites a single run samples a coin flip, and some of those rows were lucky draws. Two separate defects were behind it, both in helpers that coerce a fuzzed address into a usable recipient.

A single nudge can land on another bad address. _nudgeAddressIfProblematic probes an address with a 1 wei call and, on failure, returns _address + 1 without probing again. Measuring which low addresses actually reject such a transfer here:

Address Accepts 1 wei with empty calldata
0x01 to 0x08 yes
0x09 (blake2f) no
0x0a (point evaluation, cancun) no
0x0b upward yes

The two that reject are adjacent, so 0x09 nudges onto 0x0a and stays broken; the native transfer then reverts with a PrecompileError, surfacing as DifferentialTestAssertionFailed. CI run 32660603782 hit exactly this, with considerationRecipient: 0x09 and shouldIncludeNativeConsideration: true, at fuzz run 279 of 1000. The fix steps clear of the precompile range before probing.

Both helpers forwarded all remaining gas to an arbitrary address. isErc1155Receiver probes with to.call(...) and _nudgeAddressIfProblematic with call(gas(), ...). A fuzzed address can be any contract already in state, including the canonical CREATE2 deployer at 0x4e59b44847b379578588920cA78FbF26c0B4956C that foundry pre-deploys. It has 69 bytes of code, so it gets probed, and it reads whatever calldata it receives as a salt plus initcode and attempts a CREATE2. The first probe deploys an empty contract; a second probe with the same calldata collides, and a failed CREATE2 consumes every bit of gas forwarded to it:

Call Result Gas used
first probe success 32,834
second probe (colliding CREATE2) failure 1,040,389,983
second probe, capped at 100k failure 99,644

CI run 32662099214 died this way in ConduitExecuteWithBatch1155, and the trace shows the collision frame consuming 1,020,748,435 gas before the run fell over. Raising gas_limit is not a fix; a larger budget is simply a larger amount to burn, and at 2^34 the failure rate did not improve. Capping the probe at 100000 gas bounds it without touching any legitimate path, since a real onERC1155Received or native transfer costs far less.

Both conduit suites are affected because they share BaseConduitTest, and the deployer address appears in every counterexample collected, local and CI.

Measured over the conduit and zone suites with the fuzz cache cleared between runs: before, 4 of 8 and then 2 of 10 full runs failed; after, 15 of 15 passed. The guarded nudge helper was additionally checked to return an address above 0x0a that accepts a transfer for all 256 sequential inputs from 0x01 and for 256 fuzzed inputs.

What this does not fix

Nothing here addresses why the tests are sensitive to the Foundry version in the first place. The conduit and zone tests that fail on v1.5.1 and the vm.expectRevert/vm.roll behavior behind testFulfillOrderRevertCounterIncremented on v0.3.0 and v1.4.0 are still latent; the pin sidesteps them rather than resolving them. Bumping past v1.4.4 will surface them again and should be done deliberately, with the suite run against the candidate version first.

The pin also freezes out newer solc and analysis improvements. That seemed the right trade for a repo whose contracts are deployed and immutable, but it is a choice worth revisiting if Seaport picks up active development again.

ryanio and others added 3 commits August 23, 2026 12:13
Three independent problems kept the four Forge jobs red. All three fixes
are confined to CI configuration.

1. `.gitmodules` declared `branch = v1.5.0` for lib/forge-std and
   `branch = v0.0.84` for lib/solady. Both values are tags, not branches.
   With no foundry.lock committed, `forge install` treats every submodule
   as out of sync and checks it out at its recorded identifier; a
   `branch` line makes that identifier the literal string, so forge runs
   `git checkout v1.5.0` against the depth-1 submodule clone
   actions/checkout produces, which carries no tags. Dropping both lines
   leaves the recorded commits as the only pin. All ten resolved gitlinks
   are unchanged.

2. Every job installed `version: nightly`, so results moved with the
   toolchain rather than with the code. On this commit the suite is
   482/482 on v1.3.6 and v1.4.4, 481/482 on v0.3.0 and v1.4.0, and
   475/482 on v1.5.1. Pin v1.4.4, the newest release that is fully green.

3. forge-coverage was the only Forge job missing the two precompile
   steps, so tests that `vm.getCode` from optimized-out could not find
   it. It also set no FOUNDRY_PROFILE, and
   `_getStorageSlotForOrderHash` falls back to "optimized" in that case,
   asserting one packed storage read while coverage runs unoptimized
   source that does four. Adding the precompiles and an explicit profile
   takes that job from 22 failures to 482/482.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`_nudgeAddressIfProblematic` exists to move a fuzzed recipient off any
address that rejects a bare native transfer. It probed with a 1 wei call
and, on failure, returned `_address + 1` without probing again. The two
addresses that reject are adjacent: 0x09 (blake2f) and, because
evm_version is cancun, 0x0a (point evaluation). An input of 0x09
therefore nudged onto 0x0a and stayed broken, and the native transfer to
it reverted with a PrecompileError, surfacing as
DifferentialTestAssertionFailed.

That is the source of the intermittent red on the forge and forge-ref
jobs going back to 2025: run 32660603782 hit it at fuzz run 279 of 1000
with considerationRecipient 0x09 and shouldIncludeNativeConsideration
true.

Step clear of the precompile range before probing rather than looping on
the probe. A failing precompile call consumes all the gas forwarded to
it, and `call(gas(), ...)` forwards all but a 64th, so a second failed
probe would leave the test with almost no gas. Guarding first keeps the
single probe and its existing gas profile.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both helpers that coerce a fuzzed address into a usable recipient probed
it with all remaining gas: `isErc1155Receiver` via `to.call(...)` and
`_nudgeAddressIfProblematic` via `call(gas(), ...)`.

A fuzzed address can be any contract already in state, including the
canonical CREATE2 deployer at 0x4e59b44847b379578588920cA78FbF26c0B4956C
that foundry pre-deploys. It has 69 bytes of code, so it gets probed, and
it reads whatever calldata it receives as a salt plus initcode and
attempts a CREATE2. The first probe deploys an empty contract; a second
probe with the same calldata collides, and a failed CREATE2 consumes
every bit of gas forwarded to it. Measured on this configuration: the
first call costs 32834 gas and the second costs 1040389983, the entire
budget, so the test dies with OutOfGas.

That is why raising gas_limit did not help. A larger budget is simply a
larger amount to burn; at 2^34 the failure rate did not improve.

Capping the probe at 100000 gas bounds the damage without changing any
legitimate path, since a real onERC1155Received or native transfer costs
far less. Across the conduit and zone suites with the fuzz cache cleared
between runs, this moves 15 trials from roughly a fifth to a half
failing to 15 of 15 passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryanio
ryanio merged commit 2d34642 into main Aug 25, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants