Skip to content

test: add a Hypothesis property test suite with a nightly run - #2869

Open
tonyandrewmeyer wants to merge 12 commits into
canonical:property-testsfrom
tonyandrewmeyer:rainy/hypothesis-pilot-trimmed-v2
Open

tonyandrewmeyer wants to merge 12 commits into
canonical:property-testsfrom
tonyandrewmeyer:rainy/hypothesis-pilot-trimmed-v2

Conversation

@tonyandrewmeyer

@tonyandrewmeyer tonyandrewmeyer commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

This adds a Hypothesis property test suite in test/property/ on the long-lived property-tests branch, with a short run on every PR to that branch and a much longer one every night. The first properties cover the glue between Scenario and ops: _EntityStatus round-tripping through _to_ops() and from_ops() with arbitrary messages, an unknown status dropping its message on the way (the one status that doesn't round-trip), and TCPPort, UDPPort and ICMPPort accepting exactly the port numbers they should, with the edges of the range pinned with @example.

  • Hypothesis is in a new property dependency group (it also brings in sortedcontainers), and only the new tox -e property environment and lint install it. The unit environment doesn't, and the unit and coverage runs ignore test/property/.
  • Settings profiles in test/property/conftest.py, picked with HYPOTHESIS_PROFILE, decide how hard each run looks: ci is derandomised so a PR's result doesn't depend on luck, and nightly is 10,000 examples with fresh randomness. The tests themselves don't set max_examples.
  • property-tests.yaml runs the ci profile on PRs and pushes to property-tests. GitHub only schedules workflows from the default branch, so the nightly run is a separate workflow on main, in ci: run the property-tests branch's property tests nightly #2897. It checks this branch out, merges in the current main so that it tests today's ops, and a scheduled failure opens an issue through notify-scheduled-failure.yaml, like the other scheduled workflows.
  • CONTRIBUTING.md says how to run them, that property test PRs go against property-tests, and that a counterexample the nightly run finds should be fixed on main and then pinned with @example(...) here.

The aim is to have a lot more property tests than these over time. They live on a branch rather than main for now (as Pebble's do), which keeps the new dependency and the review load of a lot of generated tests off main while we find out how much they catch. main is merged into the branch regularly, and fixes for anything the tests find go to main.

tonyandrewmeyer and others added 2 commits November 10, 2026 15:51
The property tests get their own directory, tox environment and
dependency group, so the unit tests and the unit environment don't
depend on Hypothesis. Settings profiles in the suite's conftest decide
how hard each run looks: dev by default, a small derandomised ci
profile, and a large nightly profile.

The first two properties are the round-trips between scenario's
statuses and ports and their ops equivalents.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pull requests and pushes to main run the ci profile. The scheduled
nightly run, and manual runs, use the nightly profile with a much
larger example budget, and a scheduled failure opens or updates an
issue through notify-scheduled-failure.yaml, like the other scheduled
workflows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tonyandrewmeyer
tonyandrewmeyer force-pushed the rainy/hypothesis-pilot-trimmed-v2 branch from b7c8faf to 662ab2f Compare November 10, 2026 02:51
@tonyandrewmeyer tonyandrewmeyer changed the title test: add Hypothesis property tests for status/port conversion glue test: add a Hypothesis property test suite with a nightly run Nov 10, 2026
@tonyandrewmeyer
tonyandrewmeyer marked this pull request as ready for review November 10, 2026 03:50

@benhoyt benhoyt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm good with the idea and running a short version on PR (longer on nightly schedule), however, I'm pretty uncertain about adding these to the main branch (I see your comment in the PR body -- thanks). My concerns:

  • The new dependency in main (and its sub-dependencies) -- thinking of distros and package registries.
  • I think for this to be effective we'll want to be adding quite a few test cases, which will be a lot of code to review and so on. I think if we put it on a "basically AI-managed branch" (or fork) we can do a lot more lightly-reviewed spraying, and just push (unit) test cases and fixes to main when it finds an issue.

Additionally, the specific examples here, the Status and Port (data) classes, don't sell it for me. Aren't they basically testing that the dataclass initialisers work? An example that's actually doing significant work would be more interesting -- for example, parsing multipart bodies in pebble.py or similar.

Perhaps worth further discuss in our 1:1 tomorrow (or as a team if needed).

Comment thread test/property/test_scenario_state.py Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tonyandrewmeyer and others added 2 commits November 12, 2026 15:31
GitHub only schedules workflows from the default branch, so the nightly
run moves to a workflow on main that checks this branch out. This
workflow keeps the small run for pushes and pull requests to
property-tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its check is then "property" rather than a bare "test", which a
branch ruleset can require without confusing it with the unit matrix's
"test (...)" checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tonyandrewmeyer and others added 7 commits November 14, 2026 00:19
The round-trip property only held because the strategy gave the
unknown status a single empty-message value. An unknown status with a
message comes back without it, so the round-trip now covers the
statuses that carry a message, and the unknown case has its own
property.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Port._to_ops() copies two fields into a dataclass with no validation,
so checking that it copies them tested nothing. The ports' real rule is
the range check: TCP and UDP accept 1-65535 and ICMP accepts no port.
The integers are unbounded, and the range edges are pinned with
@example, since random integers almost never land on them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It runs Hypothesis's usual 100 examples, like dev; what sets it apart
is that it's derandomised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 60 minutes was headroom for the nightly profile, which now runs from
main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deadline is noise on CI machines, but locally it's what notices a
change that makes something accidentally slow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants