Skip to content

Make verification and PR updates resilient across consumer setups - #36

Merged
kaxil merged 2 commits into
mainfrom
resilient-verification
Nov 10, 2026
Merged

kaxil merged 2 commits into
mainfrom
resilient-verification

Conversation

@kaxil

@kaxil kaxil commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

On 2026-11-06, one consumer's DAG integrity check ran inside the image, 181 tests passed, and the PR still said "Verification — skipped" and listed 19 failures as "pre-existing". Its pytest addopts (--color=yes --no-header --tb=short) changed the output text parse_check.py scrapes. The same day, every consumer we checked got "could not determine security fixes", because the Runtime release notes now mark each build as **Astro Runtime 3.3-12**. Both bugs share one cause: the action reads text it doesn't own, and when that text changes it quietly gives a weaker answer instead of a loud one.

This PR stops reading that text where it can, and adds a check across every consumer's setup so a fix for one consumer can't break another.

  • The parse verdict comes from structured data. In the throwaway copy, verify.sh appends a small pytest reporter to .astro/conftest.py. It records each DAG file's outcome from pytest's report objects and writes one OTTO_PARSE_RESULT {json} line. The project's own integrity test, conftest hooks, fixtures and options are unchanged. A per-run nonce, passed through the copy's .env, proves the line came from the integrity container. The reporter also forces a full run, so a project's -x can't leave files untested. With no result line, the run is no verdict, never a pass.
  • The baseline is "target minus this run's edits". Before the run changes anything, run_changes.py snapshots the project as a git tree. The verifier's baseline reverts exactly the changed paths in a copy of the working tree, and open-pr stages exactly those paths. Before, the baseline was a clean git worktree of HEAD, so a gitignored .env the workflow wrote existed only on the target side. That made an upgrade-only failure in an env-dependent DAG read as pre-existing, and the run passed. The asymmetric baseline came in with f23fcbb (fix: Verify like Airflow parses, baseline-diff pre-existing failures, and handle PEP 503 pins #4).
  • Release notes: the parser matches a line holding only Astro Runtime <tag>, whatever markup wraps it. It also keeps per-image scopes like "(Python 3.13 image)", so a fix isn't credited to images that don't get it.
  • New effective-verify-level output and PR warning: when a fallback produced the verdict (for example parse couldn't run and import did), the PR leads with "Verified at import level, not parse".
  • open-pr:
    • When the tree is unchanged, it still refreshes the PR's title, body and labels instead of exiting. Before, a stale verification section stayed until the target moved (since 51d37cb).
    • It stages only what the run changed, so an untracked secret file the workflow wrote first isn't committed.
  • Projects without .astro/ scaffolding now get a parse-level verdict. The copy gets the CLI's default integrity test and config.

Evidence

Each row below is a fixture copying one real consumer's setup (e2e/consumers/). Each ran through the action's tokenless dry-run path, building the real images with Docker and astro CLI 1.47.0, against v0.6.2 and against this branch:

Fixture Consumer setup it copies v0.6.2 This branch
plain plain project, bump-blocking-pins passed (parse), security shape-mismatch passed (parse), security ok
pytest-addopts-env project addopts, gitignored .env written by the workflow passed (parse) passed (parse)
pytest-addopts-env + upgrade-only failure same passed, failure listed as "pre-existing" failed, TypeError listed as NEW
nested-standalone project-path: airflow, dev.mode: standalone passed (parse) passed (parse)
floating-base-secret floating -base tag, build secret passed (parse) passed (parse)
astro-project no .astro/ scaffolding passed via import fallback passed (parse)

Some extra context on these rows:

  • addopts row: v0.6.2 already includes Read coloured parse output and the 'Dags' spelling #35, so that row passes on both versions here, but pytest's output text is still a moving target. The new test_parse_check.py cases show v0.6.2 also misreads a clean run under -q --tb=no and -v --tb=line.
  • Fallback warning: with Docker unavailable, the plain fixture shows the new banner: Verified at import level, not parse.
  • Unit tests: 362 pass. New ones cover the reporter under four addopts shapes, the project-conftest hook coexistence, the snapshot/stage/revert rules, open-pr against a real bare remote with a fake gh, and verify.sh's parse path end to end with a fake astro.
  • CI: the consumer-shapes job runs this matrix on every PR and daily, against the latest CLI and the live release-notes page. That way upstream drift shows up there before a consumer's Friday run.

Design rationale

Why append to .astro/conftest.py instead of replacing the integrity test. Replacing it would drop whatever mocks a customer added. Appending keeps their conftest loading as itself. The reporter's hooks use unique pytest_otto_* names registered through pluggy's specname, so they can't overwrite a project hook of the same name.

The image-build failure is still detected from CLI text. When the build fails, the reporter never runs, so the CLI's message is the only evidence. It's matched case-insensitively. If the wording drifts, the run reads as no verdict and falls back with the warning; it never passes.

Known issues

  • The import-level baseline copies the files git can see outside the project, but not gitignored files there. A DAG reading a gitignored file above the project still behaves differently on the two sides.
  • The snapshot hashes untracked, non-ignored files into the scratch index once per run. A multi-GB untracked file that isn't ignored will slow the detect step.
  • In Runtime images with pluggy < 1.0 (Airflow 2.1 era), the reporter fails to load. Those runs fall back with the warning; they don't pass.

Read the parse-level verdict from a pytest reporter appended to the copy's
.astro/conftest.py instead of scraping pytest and Astro CLI text, so project
addopts and CLI wording no longer decide the result. Build the comparison
baseline from the same working-tree copy minus the run's own edits, so a
gitignored .env written by the workflow no longer turns an upgrade-only
failure into "pre-existing". Anchor Runtime release-notes parsing on the
version tag, keep per-image fix scopes, add an effective-verify-level output
with a PR warning when a fallback produced the verdict, stage only what the
run changed, refresh the PR description on unchanged trees, and add a
consumer-shape e2e matrix that also runs daily against the latest CLI and
the live release notes.
@kaxil
kaxil requested a review from a team as a code owner November 10, 2026 20:59
@kaxil
kaxil requested a balanced review from Copilot November 10, 2026 20:59
open-pr.sh wrote gh pr create stderr to /tmp/otto-upgrade even when the
caller set another WORKDIR, so a failed create died on a missing directory
instead of reporting the error.
@kaxil
kaxil merged commit 79bc5b7 into main Nov 10, 2026
11 checks passed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical reporting gaps can let unfinished tests, collection errors, and new import failures be treated as pre-existing failures, causing baseline comparison to incorrectly report a pass.

3 open issues
What changed in this PR

This PR strengthens the upgrade action’s verification and rolling-PR updates across consumer configurations, reducing reliance on formatted output and preserving workflow-provided files during baseline checks.

Changes:

  • Adds structured parse results, default integrity-test scaffolding, and explicit fallback-level reporting.
  • Uses pre-run snapshots for baseline restoration and selective staging, while refreshing unchanged PR descriptions.
  • Supports updated release-note formatting and image-specific security fixes, with regression tests and a daily consumer matrix.
File Description
tests/​test_verify_parse_level.py Exercises parse verification, baselines, fallback, and symlink handling.
tests/​test_verify_build_secrets.py Updates comments describing build copies.
tests/​test_security_fixes.py Tests bold release headings and image scopes.
tests/​test_run_changes.py Tests snapshot, staging, and restoration behavior.
tests/​test_parse_check.py Tests structured results across pytest configurations.
tests/​test_open_pr.py Tests selective commits and unchanged-PR refreshes.
tests/​test_build_pr_body.py Tests fallback warnings and scoped security reporting.
tests/​fixtures/​runtime-release-notes-bold-2026-11.md Supplies release-note formatting examples.
tests/​conftest.py Adds shared Git and snapshot helpers.
scripts/​verify.sh Prepares parse copies, restores baselines, and reports effective levels.
scripts/​security_fixes.py Recognizes more heading formats and preserves image scopes.
scripts/​run_changes.py Tracks, stages, and restores run-specific changes.
scripts/​parse_probe/​dag_integrity_default_af3.py Supplies the default Airflow 3 integrity test.
scripts/​parse_probe/​dag_integrity_default_af2.py Supplies the default Airflow 2 integrity test.
scripts/​parse_probe/​conftest.py Emits structured pytest outcomes.
scripts/​parse_check.py Reads nonce-matched results instead of pytest summaries.
scripts/​open-pr.sh Stages run changes and refreshes unchanged PR metadata.
scripts/​build_pr_body.py Displays fallback warnings and security-fix scopes.
README.md Documents verification, staging, and the new output.
pyproject.toml Excludes vendored integrity templates from Ruff.
e2e/​README.md Documents consumer fixtures and scheduled checks.
e2e/​consumers/​pytest-addopts-env/​requirements.txt Pins a provider for upgrade testing.
e2e/​consumers/​pytest-addopts-env/​pyproject.toml Configures consumer-style pytest output options.
e2e/​consumers/​pytest-addopts-env/​packages.txt Provides empty system-package scaffolding.
e2e/​consumers/​pytest-addopts-env/​fixture.env Supplies the workflow-created environment value.
e2e/​consumers/​pytest-addopts-env/​Dockerfile Pins the environment-dependent fixture’s Runtime.
e2e/​consumers/​pytest-addopts-env/​dags/​plain_dag.py Adds a clean control DAG.
e2e/​consumers/​pytest-addopts-env/​dags/​env_dependent_dag.py Models an environment-dependent upgrade regression.
e2e/​consumers/​pytest-addopts-env/​.gitignore Keeps the generated .env untracked.
e2e/​consumers/​pytest-addopts-env/​.astro/​test_dag_integrity_default.py Provides the fixture’s integrity test.
e2e/​consumers/​pytest-addopts-env/​.astro/​config.yaml Defines the Astro fixture project.
e2e/​consumers/​plain/​requirements.txt Pins a provider for the plain-project upgrade.
e2e/​consumers/​plain/​packages.txt Provides empty system-package scaffolding.
e2e/​consumers/​plain/​Dockerfile Pins the plain fixture’s Runtime.
e2e/​consumers/​plain/​dags/​plain_dag.py Adds a minimal clean DAG.
e2e/​consumers/​plain/​.astro/​test_dag_integrity_default.py Provides the plain fixture’s integrity test.
e2e/​consumers/​plain/​.astro/​config.yaml Defines the plain Astro project.
e2e/​consumers/​nested-standalone/​airflow/​requirements.txt Pins a provider in the nested project.
e2e/​consumers/​nested-standalone/​airflow/​packages.txt Provides empty system-package scaffolding.
e2e/​consumers/​nested-standalone/​airflow/​Dockerfile Models Runtime and environment settings in a nested project.
e2e/​consumers/​nested-standalone/​airflow/​dags/​plain_dag.py Adds a clean nested-project DAG.
e2e/​consumers/​nested-standalone/​airflow/​.astro/​test_dag_integrity_default.py Provides the nested fixture’s integrity test.
e2e/​consumers/​nested-standalone/​airflow/​.astro/​config.yaml Enables standalone development mode.
e2e/​consumers/​floating-base-secret/​requirements.txt Includes pinned and unpinned providers.
e2e/​consumers/​floating-base-secret/​packages.txt Adds a system-package installation case.
e2e/​consumers/​floating-base-secret/​Dockerfile Models a floating base image with a build secret.
e2e/​consumers/​floating-base-secret/​dags/​plain_dag.py Adds a clean build-secret fixture DAG.
e2e/​consumers/​floating-base-secret/​.astro/​test_dag_integrity_default.py Provides the build-secret fixture’s integrity test.
e2e/​consumers/​floating-base-secret/​.astro/​config.yaml Configures the environment-backed build secret.
action.yaml Captures the pre-run snapshot and exposes effective verification.
.github/​workflows/​e2e-otto-upgrade.yaml Adds the consumer matrix and daily upstream checks.

Review effort: Balanced

Comment thread scripts/parse_check.py
# against the baseline, its failures could clear while the files it never
# reached go unchecked.
# Something failed to collect, so some DAG file may never have been tested.
if collect_errors and not failures:
outcome = yield
report = outcome.get_result()
if call.excinfo is not None:
report._otto_msg = str(call.excinfo.value)

@_otto_pytest.hookimpl(specname="pytest_runtest_logreport")
def pytest_otto_logreport(report):
_otto_ran.add(report.nodeid)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants