Skip to content

test: widen timing margins in the quick end-to-end test - #12

Merged
Prajwal7-2002 merged 1 commit into
mainfrom
fix/flaky-quick-test
Nov 5, 2026
Merged

Prajwal7-2002 merged 1 commit into
mainfrom
fix/flaky-quick-test

Conversation

@Prajwal7-2002

Copy link
Copy Markdown
Owner

Summary

test_quick_end_to_end_on_mock_server asserts that servebench quick finds a capacity of exactly 3 users on the mock server. The mock's TTFT was 40 ms × requests in flight against a 150 ms target, which left only 30 ms of headroom at 3 users (120 ms). A slow macOS runner exceeded that and the test failed once on PR #11 (passed on re-run); main has had similar intermittent CI failures since October.

Change: 200 ms steps and a 700 ms target, so 3 users = 600 ms (100 ms headroom) and 4 users = 800 ms (100 ms over). What the test checks is unchanged; no production code changes.

Test verification

  • The test passes locally; runtime went from 22 s to 31 s.
  • ruff format --check clean.

TTFT steps of 40 ms against a 150 ms target left 30 ms of headroom at 3
users; a slow macOS runner turned 3 users into 2 (failed once on PR #11).
Use 200 ms steps and a 700 ms target: 100 ms margin on each side.
@Prajwal7-2002
Prajwal7-2002 merged commit e3d1b16 into main Nov 5, 2026
7 checks passed
@Prajwal7-2002
Prajwal7-2002 deleted the fix/flaky-quick-test branch November 5, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant