test(webhook): identify the render test's own flow run instead of counting

[fatih-acar]

Why

test_render.py::TestWebhookRender::test_branchless_event_triggers_webhook_process failed on 5
unrelated PRs against release-1.11 between Aug 13 and Aug 18, always with
AssertionError: webhook-process deployment was not run; server-side parameter render failed.
Server-side rendering is fine — the test's own assertion is the bug, and it cannot pass under the
conditions CI now hits.

The test compared flow-run counts before and after posting an event:

runs_before = len(await prefect_client.read_flow_runs(deployment_filter=deployment_filter))
...
assert runs_after > runs_before

read_flow_runs() returns at most PREFECT_API_DEFAULT_LIMIT (200) rows, and the API rejects a
larger limit outright (422 Invalid limit: must be less than or equal to 200). Once the
webhook-process deployment has >= 200 runs, runs_before == runs_after == 200 and the comparison
can never be true. Every one of the five failures printed exactly 200, on four different runners.

Those runs come from the webhook classes: TestWebhookConfigure::test_configure_all calls
reconcile-all, and nothing deletes the automations afterwards. A surviving all-branches automation
turns every event any later test class emits into a scheduled webhook-process run (nothing executes
them — the functional suite runs no Prefect worker). Under --dist loadscope the test therefore
fails iff a leaking webhook class landed on the same worker before it:

job	configure	render	result
94380453769	gw1	gw1	FAILED
94396911944	gw0	gw0	FAILED
94730761717	gw2	gw2	FAILED
95019333463	gw0	gw0	FAILED
95765302237	gw2	gw2	FAILED
94746119281 (green retry of 94730761717)	gw3	gw1	PASSED

Non-goal: no production code changes. Nothing in infrahub/ is at fault here.

What changed

• test_render.py identifies the run it caused — newest-first read, filtered to run ids not seen
  before the event and to this webhook's webhook_id — instead of counting rows, and then asserts
  the rendered parameters are the plain strings the test exists to guard (event_id, event_type,
  event_occured_at, an empty branch_name for the branch-less event, and the payload). The
  previous version only proved that some run appeared.
• conftest.py gains an autouse class-scoped fixture that deletes webhook::-prefixed automations
  at class teardown. Automations outlive the class that created them while the webhook node behind
  them is dropped with the class database.
• dev/knowledge/backend/testing.md records both traps under a new "Prefect Server State Outlives
  the Test Class" subsection.

How to review

The interesting question is whether run identity is robust where counting was not: a new
automation-triggered run is the newest by expected start time, so it lands at the top of the page
regardless of how many older runs exist, and the webhook_id filter keeps another automation's run
from being mistaken for this one.

How to test

# The failing CI ordering, in one process: configure runs before render, same Prefect server
uv run pytest backend/tests/functional/webhook -v

29 passed locally in 2m47s, including the co-located ordering that fails in CI.

The 200-row cap was verified directly against a live Prefect test server: with 205 flow runs,
read_flow_runs() returns 200, limit=1000 is a 422, and offset=200 reveals the remaining 5.

The teardown makes the suite faster, not slower

Measured over backend/tests/functional/webhook followed by one event-heavy class
(ipam/test_ipam_utilization.py) in a single process — the CI shape, where webhook classes run
before event-heavy ones on the same worker — two pairs, alternating:

	pytest duration	webhook-process runs left	leftover webhook:: automations
with cleanup	182s / 201s	1	0
without cleanup	235s / 233s	243 / 216	1

The cleanup is worth 33-53s (14-22%) on that slice, because the leaked automation generates
216-243 scheduled flow runs — each one a trigger evaluation plus rows in the same SQLite database
the tests are waiting on. It also reproduces the failure condition locally: a single event-heavy
class is enough to push the count past 200, and a CI worker runs ~15 of the suite's 62 classes.

The teardown call itself, timed against a live server (gather + deletes per class teardown):

webhook automations deleted	total automations on server	gather	deletes	teardown
2	2	5.5ms	6.9ms	12.4ms
4	10	2.0ms	11.3ms	13.3ms
10	50	3.8ms	33.5ms	37.2ms
50	200	12.8ms	150.1ms	162.9ms
200	300	17.1ms	12046ms	12.1s

Paging through all automations is cheap and flat (2-17ms even at 300). The cost is entirely in the
DELETE calls, and it degrades badly past ~50 — 60ms each at 200, presumably trigger-service
reconfiguration serialized through SQLite. The webhook classes register 1-4 automations each, so
real cost is ~12-13ms per class teardown, ~0.1s across the package. Worth knowing if a future test
ever registers automations in bulk.

Impact & rollout

• Backward compatibility: test-only change, no runtime behavior touched.
• Deployment notes: safe to merge; no changelog fragment since nothing user-facing changed.

develop and release-1.11 carry a byte-identical test_render.py (same blob hash) and are
vulnerable to the same failure, so they need this via the usual merge-forward. Other functional
suites (computed attributes, action rules) leave their own automations on the shared Prefect server
— same class of leak, different blast radius, deliberately out of scope here.

Checklist

• Tests added/updated
• Changelog entry added — not
  user-facing
• External docs updated (if user-facing or ops-facing change)
• Internal .md docs updated
• I have reviewed AI generated content

🤖 Generated with Claude Code

[codspeed-hq]

Merging this PR will not alter performance

✅ 13 untouched benchmarks

---

_{Comparing fac-fix-webhook-render-flake (d3d06ab) with stable (c6f1fa1)}

[cubic-dev-ai]

No issues found across 3 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Shadow auto-approve: would auto-approve. Test-only changes: fix flaky test by identifying runs by identity instead of count, add automation cleanup fixture, update testing docs.

Re-trigger cubic</sub>