Repository navigation
test: add a Hypothesis property test suite with a nightly run - #2869
Open
tonyandrewmeyer wants to merge 12 commits into
Open
tonyandrewmeyer wants to merge 12 commits into
tonyandrewmeyer wants to merge 12 commits into
Conversation
The property tests get their own directory, tox environment and dependency group, so the unit tests and the unit environment don't depend on Hypothesis. Settings profiles in the suite's conftest decide how hard each run looks: dev by default, a small derandomised ci profile, and a large nightly profile. The first two properties are the round-trips between scenario's statuses and ports and their ops equivalents. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pull requests and pushes to main run the ci profile. The scheduled nightly run, and manual runs, use the nightly profile with a much larger example budget, and a scheduled failure opens or updates an issue through notify-scheduled-failure.yaml, like the other scheduled workflows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tonyandrewmeyer
force-pushed
the
rainy/hypothesis-pilot-trimmed-v2
branch
from
November 10, 2026 02:51
b7c8faf to
662ab2f
Compare
tonyandrewmeyer
marked this pull request as ready for review
November 10, 2026 03:50
benhoyt
reviewed
Nov 10, 2026
benhoyt
left a comment
Collaborator
There was a problem hiding this comment.
I'm good with the idea and running a short version on PR (longer on nightly schedule), however, I'm pretty uncertain about adding these to the main branch (I see your comment in the PR body -- thanks). My concerns:
- The new dependency in main (and its sub-dependencies) -- thinking of distros and package registries.
- I think for this to be effective we'll want to be adding quite a few test cases, which will be a lot of code to review and so on. I think if we put it on a "basically AI-managed branch" (or fork) we can do a lot more lightly-reviewed spraying, and just push (unit) test cases and fixes to main when it finds an issue.
Additionally, the specific examples here, the Status and Port (data) classes, don't sell it for me. Aren't they basically testing that the dataclass initialisers work? An example that's actually doing significant work would be more interesting -- for example, parsing multipart bodies in pebble.py or similar.
Perhaps worth further discuss in our 1:1 tomorrow (or as a team if needed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub only schedules workflows from the default branch, so the nightly run moves to a workflow on main that checks this branch out. This workflow keeps the small run for pushes and pull requests to property-tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its check is then "property" rather than a bare "test", which a branch ruleset can require without confusing it with the unit matrix's "test (...)" checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The round-trip property only held because the strategy gave the unknown status a single empty-message value. An unknown status with a message comes back without it, so the round-trip now covers the statuses that carry a message, and the unknown case has its own property. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port._to_ops() copies two fields into a dataclass with no validation, so checking that it copies them tested nothing. The ports' real rule is the range check: TCP and UDP accept 1-65535 and ICMP accepts no port. The integers are unbounded, and the range edges are pinned with @example, since random integers almost never land on them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It runs Hypothesis's usual 100 examples, like dev; what sets it apart is that it's derandomised. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 60 minutes was headroom for the nightly profile, which now runs from main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deadline is noise on CI machines, but locally it's what notices a change that makes something accidentally slow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a Hypothesis property test suite in
test/property/on the long-livedproperty-testsbranch, with a short run on every PR to that branch and a much longer one every night. The first properties cover the glue between Scenario and ops:_EntityStatusround-tripping through_to_ops()andfrom_ops()with arbitrary messages, an unknown status dropping its message on the way (the one status that doesn't round-trip), andTCPPort,UDPPortandICMPPortaccepting exactly the port numbers they should, with the edges of the range pinned with@example.propertydependency group (it also brings insortedcontainers), and only the newtox -e propertyenvironment andlintinstall it. Theunitenvironment doesn't, and the unit and coverage runs ignoretest/property/.test/property/conftest.py, picked withHYPOTHESIS_PROFILE, decide how hard each run looks:ciis derandomised so a PR's result doesn't depend on luck, andnightlyis 10,000 examples with fresh randomness. The tests themselves don't setmax_examples.property-tests.yamlruns theciprofile on PRs and pushes toproperty-tests. GitHub only schedules workflows from the default branch, so the nightly run is a separate workflow onmain, in ci: run the property-tests branch's property tests nightly #2897. It checks this branch out, merges in the currentmainso that it tests today's ops, and a scheduled failure opens an issue throughnotify-scheduled-failure.yaml, like the other scheduled workflows.CONTRIBUTING.mdsays how to run them, that property test PRs go againstproperty-tests, and that a counterexample the nightly run finds should be fixed onmainand then pinned with@example(...)here.The aim is to have a lot more property tests than these over time. They live on a branch rather than
mainfor now (as Pebble's do), which keeps the new dependency and the review load of a lot of generated tests offmainwhile we find out how much they catch.mainis merged into the branch regularly, and fixes for anything the tests find go tomain.