Skip to content

Remove screenshot test PR comment workflow; replace with inline CI error and retries#2863

Open
Copilot wants to merge 5 commits into
masterfrom
copilot/remove-screenshot-test-pr-comment
Open

Remove screenshot test PR comment workflow; replace with inline CI error and retries#2863
Copilot wants to merge 5 commits into
masterfrom
copilot/remove-screenshot-test-pr-comment

Conversation

Copilot AI commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

The screenshot-test-comment.yml workflow (pull_request_target) always failed on first run for fork PRs because the workflow requires manual approval before running — meaning the screenshot test it was waiting on hadn't run yet.

Changes

  • Deleted .github/workflows/screenshot-test-comment.yml — removes the fragile PR comment workflow entirely
  • Added a Report Screenshot Test failure step to the ScreenshotTests job in main.yml using the ::error:: workflow command, following the same pattern already used by JavadocDoclint:
- name: Report Screenshot Test failure
  if: failure()
  run: |
    echo "::error title=Screenshot tests failed::Screenshot tests have failed.%0A%0A"\
    "To review the differences:%0A"\
    "  1. Download the 'screenshot-test-report' artifact from this run.%0A"\
    ...

The error message preserves the key guidance from the old PR comment: how to download the report artifact, what to do for unintentional vs. intentional visual changes, and a link to the README.

  • Added retry logic to desktop screenshot tests (main.yml) — the Gradle test task is retried up to 3 times before the job is marked as failed, reducing noise from transient failures.
  • Added retry logic to Android screenshot tests (android-screenshot.yml) — the connectedDebugAndroidTest Gradle task is similarly retried up to 3 times, with an adb uninstall before each attempt to ensure a clean state. The job timeout was increased from 20 to 60 minutes to accommodate the retries.

Copilot AI changed the title [WIP] Remove screenshot test CI task and replace with error message Remove screenshot test PR comment workflow; replace with inline CI error Jun 9, 2026
Copilot AI requested a review from riccardobl June 9, 2026 20:43
@riccardobl riccardobl marked this pull request as ready for review June 9, 2026 20:44
@riccardobl

Copy link
Copy Markdown
Member

@copilot can you make screenshot tests retry up to 3 times if they fails before marking the workflow as failed?

Copilot AI commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

@copilot can you make screenshot tests retry up to 3 times if they fails before marking the workflow as failed?

Done — the Test with Gradle Wrapper step now loops up to 3 times, exiting immediately on success and only failing the job after all 3 attempts are exhausted. Committed in 8a3fe26.

Copilot AI changed the title Remove screenshot test PR comment workflow; replace with inline CI error Remove screenshot test PR comment workflow; replace with inline CI error and retries Jun 9, 2026
Copilot AI requested a review from riccardobl June 9, 2026 20:49
@riccardobl

Copy link
Copy Markdown
Member

@copilot can you do the same for android tests?

Copilot AI commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

@copilot can you do the same for android tests?

Done — the Android test script now retries the connectedDebugAndroidTest Gradle task up to 3 times (with an adb uninstall before each attempt for a clean state), and only exits with a failure after all 3 attempts. The job timeout was also bumped from 20 to 60 minutes to accommodate the retries. Committed in the latest push.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove screenshot test PR comment CI task and failure message

2 participants