Skip to content

fix(proxy-stress-test): make the lane's assertions reproducible - #238

Merged
slayerjain merged 1 commit into
mainfrom
fix/proxy-stress-test-deterministic-upstream
Jul 31, 2026
Merged

fix(proxy-stress-test): make the lane's assertions reproducible#238
slayerjain merged 1 commit into
mainfrom
fix/proxy-stress-test-deterministic-upstream

Conversation

@slayerjain

Copy link
Copy Markdown
Member

What

Every replay of proxy-stress-test failed 3 of its 5 test cases — in CI and locally — and had done for at least two weeks. Neither cause is keploy; both are in this sample.

1. The endpoints assert on their own wall-clock time

main.go returns elapsed time in the response body:

line endpoint field
321 /api/transfer result.Duration = time.Since(start).String()
400 /api/batch-transfer Duration: time.Since(start).String()
463 /api/post-transfer "duration": time.Since(start).String()

Those are exactly the three tests that fail. get-health-1 and get-health-2 pass. A recorded "30.031440402s" will never equal a replayed "44.525254ms".

Fixed with a checked-in keploy.yml declaring duration as global body noise. Checked in rather than patched from CI so ./test.sh — which already enforces report status — behaves the same as the pipeline and the two cannot drift. keploy config --generate is a no-op when the file exists, so keploy's CI script keeps working unchanged.

2. The sample recorded against the public internet

It drove 20 concurrent HTTPS requests at https://httpbin.org during record, so whatever httpbin did that minute got baked into the expected body. One CI run recorded:

"succeeded": 6, "failed": 14, "status": 503     <- 14 of 20 hit the 30s timeout

Replay serves those from mocks in milliseconds and legitimately produced 20 / 0 / 200. The timeouts also ate one record window badly enough that only 3 of 5 test cases were captured at all.

Replaced with a local Caddy stub on the compose network. It serves real HTTPS on 443 via tls internal, so the CONNECT-tunnel and TLS-cert-caching paths this sample exists to stress are untouched — only the far end moves in-cluster, where it is deterministic. No CA plumbing is needed: the app already sets InsecureSkipVerify to accept keploy's MITM cert, and keploy never validates upstream certs either. The /get body is padded to roughly httpbin's size so the tunnel carries comparable data.

Verification

Full record → replay cycle, locally, against a keploy built from main with the private parsers:

before after
replay result 2 passed / 3 failed 5 passed / 0 failed
test-set-0-report.yaml FAILED PASSED
recorded batch fields succeeded: 6, failed: 14, status: 503 (CI) succeeded: 20, failed: 0, status: 200

Also verified directly: the stub reaches healthy, curl -k -x http://proxy:3128 https://httpstub/get returns 200 through the CONNECT tunnel, and 20 concurrent GETs all return 200.

Related

  • keploy/keploy PR that arms the lane's gate (it was reporting success through all of the above). That PR requires this keploy.yml to be present and fails loudly if it is not, so land this one first.

Residual, stated plainly

/api/batch-transfer still asserts exact succeeded/failed counts over 20 concurrent connections. The stub removes the internet as a variable but not concurrency — if one connection fails under CI load, three body fields flip at once. With the gate armed that becomes a red lane rather than an invisible one, which is the right trade, but it is the thing most likely to bite next.

Every replay of this sample failed 3 of its 5 test cases, in CI and locally,
and had done for at least two weeks. Neither cause was keploy.

1. The three stress endpoints report their own wall-clock elapsed time in the
   response body — `Duration: time.Since(start).String()` in main.go on
   /api/transfer (:321), /api/batch-transfer (:400) and /api/post-transfer
   (:463). Those are exactly the three test cases that fail; get-health-1 and
   get-health-2 pass. A recorded "30.031440402s" is never going to equal a
   replayed "44.525254ms".

   Fixed by checking in a keploy.yml that declares `duration` global body
   noise. Checked in rather than patched by CI so `./test.sh` — which already
   enforces report status — gets the same behaviour, and the two cannot drift.
   `keploy config --generate` is a no-op when the file exists, so the CI script
   keeps working unchanged.

2. The sample drove 20 concurrent HTTPS requests at https://httpbin.org during
   RECORD, so whatever the public internet did that minute got baked into the
   expected body. A CI run recorded "succeeded": 6, "failed": 14, "status":
   503 after 14 of 20 calls hit the 30s timeout; replay serves those from mocks
   in milliseconds and legitimately produced 20 / 0 / 200. The timeouts also
   ate one record window badly enough that only 3 of 5 test cases were
   captured.

   Fixed with a local Caddy stub. It serves real HTTPS on 443 with a
   `tls internal` cert, so the CONNECT-tunnel and TLS-cert-caching paths this
   sample exists to stress are untouched — only the far end moves in-cluster,
   where it is deterministic. Nothing has to trust the cert: the app already
   sets InsecureSkipVerify to accept keploy's MITM cert, and keploy never
   validates upstream certs either. The /get body is padded to roughly
   httpbin's response size so the tunnel still carries comparable data.

Verified locally against a full record -> replay cycle: before, 2 passed /
3 failed with test-set-0 FAILED; after, 5 passed / 0 failed with test-set-0
PASSED, and the recorded batch fields are a deterministic succeeded=20,
failed=0, status=200.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
@slayerjain
slayerjain merged commit 572f0ee into main Jul 31, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant