Repository navigation
Make verification and PR updates resilient across consumer setups - #36
Merged
Merged
Conversation
Read the parse-level verdict from a pytest reporter appended to the copy's .astro/conftest.py instead of scraping pytest and Astro CLI text, so project addopts and CLI wording no longer decide the result. Build the comparison baseline from the same working-tree copy minus the run's own edits, so a gitignored .env written by the workflow no longer turns an upgrade-only failure into "pre-existing". Anchor Runtime release-notes parsing on the version tag, keep per-image fix scopes, add an effective-verify-level output with a PR warning when a fallback produced the verdict, stage only what the run changed, refresh the PR description on unchanged trees, and add a consumer-shape e2e matrix that also runs daily against the latest CLI and the live release notes.
open-pr.sh wrote gh pr create stderr to /tmp/otto-upgrade even when the caller set another WORKDIR, so a failed create died on a missing directory instead of reporting the error.
There was a problem hiding this comment.
🟡 Changes recommended
Critical reporting gaps can let unfinished tests, collection errors, and new import failures be treated as pre-existing failures, causing baseline comparison to incorrectly report a pass.
3 open issues
What changed in this PR
This PR strengthens the upgrade action’s verification and rolling-PR updates across consumer configurations, reducing reliance on formatted output and preserving workflow-provided files during baseline checks.
Changes:
- Adds structured parse results, default integrity-test scaffolding, and explicit fallback-level reporting.
- Uses pre-run snapshots for baseline restoration and selective staging, while refreshing unchanged PR descriptions.
- Supports updated release-note formatting and image-specific security fixes, with regression tests and a daily consumer matrix.
| File | Description |
|---|---|
tests/test_verify_parse_level.py |
Exercises parse verification, baselines, fallback, and symlink handling. |
tests/test_verify_build_secrets.py |
Updates comments describing build copies. |
tests/test_security_fixes.py |
Tests bold release headings and image scopes. |
tests/test_run_changes.py |
Tests snapshot, staging, and restoration behavior. |
tests/test_parse_check.py |
Tests structured results across pytest configurations. |
tests/test_open_pr.py |
Tests selective commits and unchanged-PR refreshes. |
tests/test_build_pr_body.py |
Tests fallback warnings and scoped security reporting. |
tests/fixtures/runtime-release-notes-bold-2026-11.md |
Supplies release-note formatting examples. |
tests/conftest.py |
Adds shared Git and snapshot helpers. |
scripts/verify.sh |
Prepares parse copies, restores baselines, and reports effective levels. |
scripts/security_fixes.py |
Recognizes more heading formats and preserves image scopes. |
scripts/run_changes.py |
Tracks, stages, and restores run-specific changes. |
scripts/parse_probe/dag_integrity_default_af3.py |
Supplies the default Airflow 3 integrity test. |
scripts/parse_probe/dag_integrity_default_af2.py |
Supplies the default Airflow 2 integrity test. |
scripts/parse_probe/conftest.py |
Emits structured pytest outcomes. |
scripts/parse_check.py |
Reads nonce-matched results instead of pytest summaries. |
scripts/open-pr.sh |
Stages run changes and refreshes unchanged PR metadata. |
scripts/build_pr_body.py |
Displays fallback warnings and security-fix scopes. |
README.md |
Documents verification, staging, and the new output. |
pyproject.toml |
Excludes vendored integrity templates from Ruff. |
e2e/README.md |
Documents consumer fixtures and scheduled checks. |
e2e/consumers/pytest-addopts-env/requirements.txt |
Pins a provider for upgrade testing. |
e2e/consumers/pytest-addopts-env/pyproject.toml |
Configures consumer-style pytest output options. |
e2e/consumers/pytest-addopts-env/packages.txt |
Provides empty system-package scaffolding. |
e2e/consumers/pytest-addopts-env/fixture.env |
Supplies the workflow-created environment value. |
e2e/consumers/pytest-addopts-env/Dockerfile |
Pins the environment-dependent fixture’s Runtime. |
e2e/consumers/pytest-addopts-env/dags/plain_dag.py |
Adds a clean control DAG. |
e2e/consumers/pytest-addopts-env/dags/env_dependent_dag.py |
Models an environment-dependent upgrade regression. |
e2e/consumers/pytest-addopts-env/.gitignore |
Keeps the generated .env untracked. |
e2e/consumers/pytest-addopts-env/.astro/test_dag_integrity_default.py |
Provides the fixture’s integrity test. |
e2e/consumers/pytest-addopts-env/.astro/config.yaml |
Defines the Astro fixture project. |
e2e/consumers/plain/requirements.txt |
Pins a provider for the plain-project upgrade. |
e2e/consumers/plain/packages.txt |
Provides empty system-package scaffolding. |
e2e/consumers/plain/Dockerfile |
Pins the plain fixture’s Runtime. |
e2e/consumers/plain/dags/plain_dag.py |
Adds a minimal clean DAG. |
e2e/consumers/plain/.astro/test_dag_integrity_default.py |
Provides the plain fixture’s integrity test. |
e2e/consumers/plain/.astro/config.yaml |
Defines the plain Astro project. |
e2e/consumers/nested-standalone/airflow/requirements.txt |
Pins a provider in the nested project. |
e2e/consumers/nested-standalone/airflow/packages.txt |
Provides empty system-package scaffolding. |
e2e/consumers/nested-standalone/airflow/Dockerfile |
Models Runtime and environment settings in a nested project. |
e2e/consumers/nested-standalone/airflow/dags/plain_dag.py |
Adds a clean nested-project DAG. |
e2e/consumers/nested-standalone/airflow/.astro/test_dag_integrity_default.py |
Provides the nested fixture’s integrity test. |
e2e/consumers/nested-standalone/airflow/.astro/config.yaml |
Enables standalone development mode. |
e2e/consumers/floating-base-secret/requirements.txt |
Includes pinned and unpinned providers. |
e2e/consumers/floating-base-secret/packages.txt |
Adds a system-package installation case. |
e2e/consumers/floating-base-secret/Dockerfile |
Models a floating base image with a build secret. |
e2e/consumers/floating-base-secret/dags/plain_dag.py |
Adds a clean build-secret fixture DAG. |
e2e/consumers/floating-base-secret/.astro/test_dag_integrity_default.py |
Provides the build-secret fixture’s integrity test. |
e2e/consumers/floating-base-secret/.astro/config.yaml |
Configures the environment-backed build secret. |
action.yaml |
Captures the pre-run snapshot and exposes effective verification. |
.github/workflows/e2e-otto-upgrade.yaml |
Adds the consumer matrix and daily upstream checks. |
Review effort: Balanced
| # against the baseline, its failures could clear while the files it never | ||
| # reached go unchecked. | ||
| # Something failed to collect, so some DAG file may never have been tested. | ||
| if collect_errors and not failures: |
| outcome = yield | ||
| report = outcome.get_result() | ||
| if call.excinfo is not None: | ||
| report._otto_msg = str(call.excinfo.value) |
|
|
||
| @_otto_pytest.hookimpl(specname="pytest_runtest_logreport") | ||
| def pytest_otto_logreport(report): | ||
| _otto_ran.add(report.nodeid) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

On 2026-11-06, one consumer's DAG integrity check ran inside the image, 181 tests passed, and the PR still said "Verification — skipped" and listed 19 failures as "pre-existing". Its pytest
addopts(--color=yes --no-header --tb=short) changed the output textparse_check.pyscrapes. The same day, every consumer we checked got "could not determine security fixes", because the Runtime release notes now mark each build as**Astro Runtime 3.3-12**. Both bugs share one cause: the action reads text it doesn't own, and when that text changes it quietly gives a weaker answer instead of a loud one.This PR stops reading that text where it can, and adds a check across every consumer's setup so a fix for one consumer can't break another.
.astro/conftest.py. It records each DAG file's outcome from pytest's report objects and writes oneOTTO_PARSE_RESULT {json}line. The project's own integrity test, conftest hooks, fixtures and options are unchanged. A per-run nonce, passed through the copy's.env, proves the line came from the integrity container. The reporter also forces a full run, so a project's-xcan't leave files untested. With no result line, the run is no verdict, never a pass.run_changes.pysnapshots the project as a git tree. The verifier's baseline reverts exactly the changed paths in a copy of the working tree, and open-pr stages exactly those paths. Before, the baseline was a cleangit worktreeof HEAD, so a gitignored.envthe workflow wrote existed only on the target side. That made an upgrade-only failure in an env-dependent DAG read as pre-existing, and the run passed. The asymmetric baseline came in with f23fcbb (fix: Verify like Airflow parses, baseline-diff pre-existing failures, and handle PEP 503 pins #4).Astro Runtime <tag>, whatever markup wraps it. It also keeps per-image scopes like "(Python 3.13 image)", so a fix isn't credited to images that don't get it.effective-verify-leveloutput and PR warning: when a fallback produced the verdict (for exampleparsecouldn't run andimportdid), the PR leads with "Verified atimportlevel, notparse"..astro/scaffolding now get a parse-level verdict. The copy gets the CLI's default integrity test and config.Evidence
Each row below is a fixture copying one real consumer's setup (
e2e/consumers/). Each ran through the action's tokenless dry-run path, building the real images with Docker and astro CLI 1.47.0, against v0.6.2 and against this branch:plainbump-blocking-pinsshape-mismatchokpytest-addopts-envaddopts, gitignored.envwritten by the workflowpytest-addopts-env+ upgrade-only failureTypeErrorlisted as NEWnested-standaloneproject-path: airflow,dev.mode: standalonefloating-base-secret-basetag, build secretastro-project.astro/scaffoldingSome extra context on these rows:
addoptsrow: v0.6.2 already includes Read coloured parse output and the 'Dags' spelling #35, so that row passes on both versions here, but pytest's output text is still a moving target. The newtest_parse_check.pycases show v0.6.2 also misreads a clean run under-q --tb=noand-v --tb=line.plainfixture shows the new banner:Verified at import level, not parse.addoptsshapes, the project-conftest hook coexistence, the snapshot/stage/revert rules, open-pr against a real bare remote with a fakegh, and verify.sh's parse path end to end with a fakeastro.consumer-shapesjob runs this matrix on every PR and daily, against the latest CLI and the live release-notes page. That way upstream drift shows up there before a consumer's Friday run.Design rationale
Why append to
.astro/conftest.pyinstead of replacing the integrity test. Replacing it would drop whatever mocks a customer added. Appending keeps their conftest loading as itself. The reporter's hooks use uniquepytest_otto_*names registered through pluggy'sspecname, so they can't overwrite a project hook of the same name.The image-build failure is still detected from CLI text. When the build fails, the reporter never runs, so the CLI's message is the only evidence. It's matched case-insensitively. If the wording drifts, the run reads as no verdict and falls back with the warning; it never passes.
Known issues