Skip to content

Close three ways a partial integrity run could still pass - #37

Merged
kaxil merged 3 commits into
mainfrom
probe-completeness-fixes
Nov 10, 2026
Merged

kaxil merged 3 commits into
mainfrom
probe-completeness-fixes

Conversation

@kaxil

@kaxil kaxil commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #36, from Copilot's review there and a review of the first fix. Each change closes a way a cut-short or partly broken integrity run could still pass when the baseline already had a failure.

  • An incomplete session is never compared with the baseline. A session is incomplete when part of it failed to collect, it stopped before every test ran, or pytest exited abnormally. Failures from such a session still fail the run, and verify.sh lists them all instead of comparing them with the baseline. In a comparison, a shared failure could clear while files that never ran went unchecked. An incomplete session with no failures is no verdict, and falls back with the PR warning. rc 5 again means only "collection failed and nothing ran".
  • A test counts as run only once its teardown reports, so one interrupted mid-call leaves the session incomplete.
  • The probe records the exception class as its own field. A custom integrity test that raises ModuleNotFoundError itself then reads as an import break in the baseline comparison, even with a long or multi-line message. The default template is unaffected: its class still comes from DagBag's last traceback line.

Tests in tests/test_parse_check.py cover each case. tests/test_verify_parse_level.py adds an end-to-end case: a project plugin stops the run at a failure the baseline shares, and verification must fail instead of passing it as pre-existing.

I also ran the action at this branch from a separate private repo, as a tokenless dry-run against two projects there, side by side with v0.6.2:

  • One project is on Runtime 3.1-13 (Airflow 3), the other on Runtime 12.5.0 (Airflow 2).
  • The reporter produced its result line on both image lines (4 collected, 4 ran, nonce matched).
  • Both versions gave the same verdict, a genuine image-build failure at the target.
  • This branch also listed the security fixes (75 and 57) that v0.6.2 reported as unknown.

A collection error beside test failures now means no verdict, a test counts
as run only once its teardown reports, and failure messages keep the
exception class, so a directly raised ModuleNotFoundError reads as an
import failure in the baseline comparison.
A collection error, a stop before every test ran, or an abnormal pytest exit
now marks the session incomplete. Failures it found still fail the run, but
verify.sh shows them all instead of comparing with the baseline, where a
shared failure could clear while the files that never ran go unchecked.
With no failures, an incomplete session stays no verdict. The probe records
the exception class as its own field instead of prefixing the message.
@kaxil
kaxil requested a review from a team as a code owner November 10, 2026 21:33
An incomplete target session is never compared, so its baseline image is no
longer built. Its report now opens with the warning (with the reason escaped
for Markdown), and an error a test raised itself keeps its class and the
first line of its message, where the module or symbol is named.
@kaxil
kaxil merged commit a4bd06b into main Nov 10, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant