Repository navigation
Close three ways a partial integrity run could still pass - #37
Merged
Merged
Conversation
A collection error beside test failures now means no verdict, a test counts as run only once its teardown reports, and failure messages keep the exception class, so a directly raised ModuleNotFoundError reads as an import failure in the baseline comparison.
A collection error, a stop before every test ran, or an abnormal pytest exit now marks the session incomplete. Failures it found still fail the run, but verify.sh shows them all instead of comparing with the baseline, where a shared failure could clear while the files that never ran go unchecked. With no failures, an incomplete session stays no verdict. The probe records the exception class as its own field instead of prefixing the message.
An incomplete target session is never compared, so its baseline image is no longer built. Its report now opens with the warning (with the reason escaped for Markdown), and an error a test raised itself keeps its class and the first line of its message, where the module or symbol is named.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #36, from Copilot's review there and a review of the first fix. Each change closes a way a cut-short or partly broken integrity run could still pass when the baseline already had a failure.
verify.shlists them all instead of comparing them with the baseline. In a comparison, a shared failure could clear while files that never ran went unchecked. An incomplete session with no failures is no verdict, and falls back with the PR warning. rc 5 again means only "collection failed and nothing ran".ModuleNotFoundErroritself then reads as an import break in the baseline comparison, even with a long or multi-line message. The default template is unaffected: its class still comes from DagBag's last traceback line.Tests in
tests/test_parse_check.pycover each case.tests/test_verify_parse_level.pyadds an end-to-end case: a project plugin stops the run at a failure the baseline shares, and verification must fail instead of passing it as pre-existing.I also ran the action at this branch from a separate private repo, as a tokenless dry-run against two projects there, side by side with v0.6.2:
unknown.