fix(ci): retry cosign transparency-log writes and upload SBOMs first - #10232
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 1 file
Confidence score: 5/5
- In
.github/workflows/ci-docker-image.yml, duplicating the exponential-backoff retry loop acrossSign manifestandAttest SBOMscreates a maintenance-drift risk where one path may get updated while the other lags, leading to inconsistent CI reliability—extract the shared retry logic into a reusable step/composite action so both commands stay aligned.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name=".github/workflows/ci-docker-image.yml">
<violation number="1" location=".github/workflows/ci-docker-image.yml:217">
P3: The retry-with-exponential-backoff loop is duplicated almost verbatim between the `Sign manifest` step (5 attempts, 15s base) and the `Attest SBOMs` step — only the inner cosign subcommand differs. This is the exact duplication the PR elsewhere sets out to avoid ("share a single retry helper"), and it leaves the magic constants (attempt count `5`, base delay `15`) and the backoff formula in two places that can drift apart. Consider extracting a small reusable helper (e.g., a `retry_cosign` function or a shell snippet) so the attempt count/delays are defined once and the warning/error formatting stays consistent.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
| if [ "${attempt}" -eq 5 ]; then | ||
| break | ||
| fi | ||
| delay=$((15 * 2 ** (attempt - 1))) |
There was a problem hiding this comment.
P3: The retry-with-exponential-backoff loop is duplicated almost verbatim between the Sign manifest step (5 attempts, 15s base) and the Attest SBOMs step — only the inner cosign subcommand differs. This is the exact duplication the PR elsewhere sets out to avoid ("share a single retry helper"), and it leaves the magic constants (attempt count 5, base delay 15) and the backoff formula in two places that can drift apart. Consider extracting a small reusable helper (e.g., a retry_cosign function or a shell snippet) so the attempt count/delays are defined once and the warning/error formatting stays consistent.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/ci-docker-image.yml, line 217:
<comment>The retry-with-exponential-backoff loop is duplicated almost verbatim between the `Sign manifest` step (5 attempts, 15s base) and the `Attest SBOMs` step — only the inner cosign subcommand differs. This is the exact duplication the PR elsewhere sets out to avoid ("share a single retry helper"), and it leaves the magic constants (attempt count `5`, base delay `15`) and the backoff formula in two places that can drift apart. Consider extracting a small reusable helper (e.g., a `retry_cosign` function or a shell snippet) so the attempt count/delays are defined once and the warning/error formatting stays consistent.</comment>
<file context>
@@ -199,9 +199,27 @@ jobs:
+ if [ "${attempt}" -eq 5 ]; then
+ break
+ fi
+ delay=$((15 * 2 ** (attempt - 1)))
+ echo "::warning::cosign sign failed (attempt ${attempt}/5), retrying in ${delay}s"
+ sleep "${delay}"
</file context>
There was a problem hiding this comment.
Addressed in 67af96b: both jobs now carry a byte-identical retry() function taking the command as arguments ("$@", no string/eval layer), replacing the inline loop / attest_with_retry pair. The jobs run on separate runners, so the copies can't be genuinely shared without a checkout or a third-party action in the signing path — the identical text is the drift guard, verified by parsing the workflow and asserting the two bodies equal. The exponential backoff and the COSIGN_RETRY_* env tunables are gone too: five attempts 60s apart cover the same ~4-minute window as the old 15/30/60/120 schedule.
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 2 unresolved issues from previous reviews.
Re-trigger cubic
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved issue from previous reviews.
Re-trigger cubic
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Shadow auto-approve: would auto-approve. Add retry with exponential backoff to cosign sign and attest commands in CI, and upload SBOM artifacts before attesting to prevent transient Sigstore outages from blocking releases.
Re-trigger cubic
The v1.10.7 release failed in publish-docker-image / sbom (run
31521732131) when `cosign attest` could not write to the Sigstore
transparency log:
Post "https://rekor.sigstore.dev/api/v1/log/entries":
giving up after 2 attempt(s)
Rekor was only briefly unavailable — the sign job had succeeded against
the same infrastructure five minutes earlier, and the identical sbom job
passed on the next nightly run with no code change. But cosign retries a
failed log write only twice internally and exposes no knob to raise
that, so a blip that short was enough to fail the release. The fallout
was disproportionate: the SBOM upload step never ran, so v1.10.7 is the
only release since v1.10.0 with no SBOM artifacts, and
repository-dispatch (which needs publish-docker-image) was skipped, so
downstream repos were never notified of the release.
Three changes:
- Wrap cosign sign and both cosign attest calls in a retry() helper:
five attempts 60 seconds apart, covering a ~4-minute outage. The
helper takes the command as arguments ("$@", no string/eval layer)
and is pasted verbatim into the sign and sbom jobs — they run on
separate runners, so it cannot be shared without a checkout or a
third-party action in the signing path.
- Upload the SBOM artifacts before attesting them, so a
transparency-log outage can no longer cost us the SBOMs themselves.
Attestation failures still fail the job, which is the correct outcome
for a supply-chain step.
- Set overwrite: true on the SBOM upload. Artifacts are scoped to the
run rather than the attempt, so with the upload now preceding a step
that can fail, a re-run would otherwise die at the upload step
because the artifact name already exists from the earlier attempt.
Every caller passes a version unique to its invocation, so the only
artifact this can replace is the same SBOM from a previous attempt of
the same run.
Verified by extracting both run scripts from the parsed workflow and
executing them against a stubbed cosign: immediate success (1 call, no
sleeps), recovery after 3 failures (exit 0, 3 sleeps), budget
exhaustion (5 calls, 4 sleeps, exit 1), and a terminally failing first
attest stops the script before the second attest runs. The two retry()
bodies are asserted byte-identical, and actionlint passes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
67af96b to
461f485
Compare
2 participants
Problem
The v1.10.7 release failed in
publish-docker-image / sbom(run 31521732131) at theAttest SBOM (SPDX)step:Rekor was only briefly unavailable. Evidence it was transient, not a config break:
signjob succeeded at 18:21:53 against the same Sigstore infrastructure;sbomfailed at 18:26–18:27.sbomjob passed on the next run (31549394917) with no code change.cosign retries a failed transparency-log write only twice before giving up, which is short enough that a blip of a few minutes fails an entire release.
Why it mattered more than it should have
Attest SBOM (CycloneDX)andUpload SBOM artifactsnever ran, so:sbom-infrahub-v1.10.0…v1.10.6sequence (since re-run manually).repository-dispatchwas skipped. It declaresneeds: publish-docker-image(release.yml:137-141), so downstream repos were never notified of v1.10.7.Changes
cosign signand bothcosign attestcalls. Both talk to the same transparency log and are equally exposed; onlyattesthappened to lose the coin flip this time.The two
Attest SBOM (*)steps are consolidated into oneAttest SBOMsstep so the retry helper is defined once.Verification
yamllint -c .yamllint.ymlpasses on the modified workflow.bash -ewith a stubbedcosignfor three cases: succeeds first try (exit 0), recovers after 3 failures (exit 0, the real-world scenario), and exhausts all 5 attempts (exit 1, step fails).cosign-installer@v4.1.2installs) that no flag semantics changed; the cosign invocations themselves are unchanged apart from moving the image reference into anIMAGEenv var.Deliberately not changed — needs a decision
cosign signis pinned to the old signing path (--new-bundle-format=false --use-signing-config=false), added in 550d134 with the note "Use old bundle format and disable signing config until we upgrade Harbor to 2.15." The twocosign attestcalls never received those flags, so attestations still run through the new sigstore-go bundle path — which is precisely the code path that emittedsigning bundle: error signing bundlehere.I have left that asymmetry alone: flipping the attestation bundle format changes what is stored in the registry and how consumers verify it, which is a supply-chain behaviour change rather than a resilience fix. @fatih-acar — was excluding
attestfrom those flags intentional, or should it matchsign?