Skip to content

fix: preserve complete CI test output and exit status - #1327

Merged
exKAZUu merged 13 commits into
mainfrom
codex/preserve-ci-test-output
Sep 14, 2026
Merged

fix: preserve complete CI test output and exit status#1327
exKAZUu merged 13 commits into
mainfrom
codex/preserve-ci-test-output

Conversation

@exKAZUu

@exKAZUu exKAZUu commented Sep 14, 2026

Copy link
Copy Markdown
Member

Why

WillBooster/review-booster#329 needed a local CI shell wrapper to retain failure diagnostics. Buffered test output followed by immediate process exit can lose data when stdout is piped; central capture avoids repeating that workaround.

Requirements

  • required: Improve both. $review-booster -> $complete-pr -> $release on each PR.
  • required: Preserve CI test selection and failing command exit status.
  • chosen: Save streaming test-on-ci stdout/stderr in .wb/test-ci.log, overwriting it per run while preserving it during dry runs.
  • chosen: Reuse verification output capture with a streaming mode and await both output streams before returning a test failure.
  • chosen: Generated standalone workflows capture custom test/ci output only when raw artifact uploads are enabled; both require the repository variable UPLOAD_TEST_LOG=true and use 14-day retention. GitHub masks console output but not artifact files, so default-on uploads were rejected to preserve secret protection. Opted-out commands execute directly to avoid an extra workflow log; wb local capture remains independent so diagnostics stay available locally even without artifact uploads.

Customer Summary

CI tests retain their complete output locally while continuing to stream diagnostics. Generated standalone workflows can retain test logs as downloadable artifacts after failures when the repository opts in for secret-free output.

Technical Summary

testOnCi.ts handles command failures after output capture has flushed, preserving the exit status. The verification reporter's optional streaming mode writes output to disk as it arrives and forwards it to the console. selfContainedWorkflow.ts captures the test command with tee, reads its original pipeline exit code, and uploads the log using the OS, job check-run ID and run attempt when UPLOAD_TEST_LOG=true. A failed log write fails an opted-in run while preserving any nonzero test code. The tee subprocess ignores file-size-limit signals so it can keep draining output after a log failure. Local capture handles short writes and reports write, tail-read and close errors after the command finishes. Quiet verification falls back to console output after a log-write failure so subsequent test diagnostics remain visible. Captured E2E commands inherit stdin so EOF remains observable. Streaming subprocesses skip accumulating a second in-memory copy of output; other spawn callers retain the existing collection default. Forwarded child streams honor stdout/stderr backpressure by pausing until the destination drains. Captured commands preserve terminal colors while respecting explicit color preferences. The README documents local log errors and console fallback for all verification commands.

Testing

  • npm_config_prefer_offline=true bun run verify-full: passed in 1m 22s. A prior unmodified registry-backed test timed out against npm.flatt.tech; cached npm metadata avoided the proxy outage without changing registry configuration.
  • A real E2E command emits 160 MiB through a wrapper with a 96 MiB heap and retains its full log; the before-fix reproduction exhausted the heap.
  • A slow-reader probe for 160 MiB completed with full output and reduced peak RSS from 271 MiB to 93 MiB; the automated large-output test also drains through a pipe.
  • A real terminal probe confirmed captured unit and E2E commands preserve colors consistently. Final documentation verification passed with bun run verify; GitHub CI and review-booster completed on the final head.
  • A targeted CLI fault-injection check confirms a delayed close error retains exit 7 and prints the incomplete-log diagnostic.
  • Real CLI tests cover complete large Unicode stdout/stderr for exit codes 0 and 7, log replacement, and dry-run preservation.
  • The generated workflow's Bash step runs against real commands, checking complete stdout/stderr output and exit codes 0 and 7.
  • Real file-size-limit checks cover local capture and tee failures with passing and failing tests, and opted-out workflow tests prove no log allocation occurs. An E2E fixture reads stdin to EOF while its output is captured. Existing verification-output tests continue to pass.

Notes

Reusable workflow capture and failure-time uploads are handled by WillBooster/reusable-workflows#518. Generated workflow changes apply on the next wbfy run.

Save streaming test-on-ci output and flush diagnostics before returning a
failure. Retain test logs in generated standalone workflows as artifacts.

Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
@exKAZUu exKAZUu self-assigned this Sep 14, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the reliability of CI test diagnostics by ensuring that complete test output and command exit statuses are always preserved. It introduces local logging for wb test-on-ci commands with streaming capabilities and enhances generated CI workflows to automatically capture and upload test logs as artifacts. These changes prevent the loss of critical debugging information, making it easier to diagnose and resolve CI failures.

Highlights

  • Enhanced CI Test Output Preservation: The testOnCi command now streams raw output to .wb/test-ci.log and ensures that the exit status of failed commands is retained, preventing loss of diagnostic information.
  • Streaming Verification Output: The startVerificationOutput utility has been improved to allow direct streaming of output to the console while simultaneously writing it to a log file, offering immediate feedback during CI runs.
  • Automated Test Log Artifact Uploads: Generated standalone CI workflows are updated to capture test/ci command output using tee and upload it as a unique log artifact to GitHub Actions, with a 14-day retention period for comprehensive post-failure analysis.
  • Improved Test Coverage: New tests have been added to verify the complete preservation of CI output, including large Unicode content, and the correct handling of various exit codes (0 and 7) for the test-on-ci command.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.


CI logs now flow, Failures caught, exit codes known, No more lost output.

Footnotes

  1. Review the Generative AI Prohibited Use Policy, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the 'wb test-on-ci' command to stream raw output and overwrite '.wb/test-ci.log' on each run. It introduces a 'streamOutput' option in 'startVerificationOutput' to allow streaming output directly while still logging to a file. Additionally, the self-contained test workflow generator in 'wbfy' is updated to run tests in a wrapper that pipes output to a log file and uploads it as a GitHub Actions artifact. Relevant unit tests are added and updated to verify these changes. There are no review comments, so I have no feedback to provide.

exKAZUu and others added 12 commits September 14, 2026 14:50
Capture workflow logs only for opted-in repositories, propagate tee failures,
and label artifacts by platform. Preserve stdin for captured E2E commands.

Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
Keep inherited stdin for pooled commands as well as E2E execution.

Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
Retain local diagnostics independently of optional workflow artifacts and
record that incomplete logs fail otherwise successful capture.

Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
Co-authored-by: WillBooster (Codex CLI) <agent@willbooster.com>
@exKAZUu
exKAZUu merged commit b15f713 into main Sep 14, 2026
5 checks passed
@exKAZUu
exKAZUu deleted the codex/preserve-ci-test-output branch September 14, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant