fix(proxy-stress-test): make the lane's assertions reproducible - #238
Merged
Merged
Conversation
Every replay of this sample failed 3 of its 5 test cases, in CI and locally, and had done for at least two weeks. Neither cause was keploy. 1. The three stress endpoints report their own wall-clock elapsed time in the response body — `Duration: time.Since(start).String()` in main.go on /api/transfer (:321), /api/batch-transfer (:400) and /api/post-transfer (:463). Those are exactly the three test cases that fail; get-health-1 and get-health-2 pass. A recorded "30.031440402s" is never going to equal a replayed "44.525254ms". Fixed by checking in a keploy.yml that declares `duration` global body noise. Checked in rather than patched by CI so `./test.sh` — which already enforces report status — gets the same behaviour, and the two cannot drift. `keploy config --generate` is a no-op when the file exists, so the CI script keeps working unchanged. 2. The sample drove 20 concurrent HTTPS requests at https://httpbin.org during RECORD, so whatever the public internet did that minute got baked into the expected body. A CI run recorded "succeeded": 6, "failed": 14, "status": 503 after 14 of 20 calls hit the 30s timeout; replay serves those from mocks in milliseconds and legitimately produced 20 / 0 / 200. The timeouts also ate one record window badly enough that only 3 of 5 test cases were captured. Fixed with a local Caddy stub. It serves real HTTPS on 443 with a `tls internal` cert, so the CONNECT-tunnel and TLS-cert-caching paths this sample exists to stress are untouched — only the far end moves in-cluster, where it is deterministic. Nothing has to trust the cert: the app already sets InsecureSkipVerify to accept keploy's MITM cert, and keploy never validates upstream certs either. The /get body is padded to roughly httpbin's response size so the tunnel still carries comparable data. Verified locally against a full record -> replay cycle: before, 2 passed / 3 failed with test-set-0 FAILED; after, 5 passed / 0 failed with test-set-0 PASSED, and the recorded batch fields are a deterministic succeeded=20, failed=0, status=200. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every replay of
proxy-stress-testfailed 3 of its 5 test cases — in CI and locally — and had done for at least two weeks. Neither cause is keploy; both are in this sample.1. The endpoints assert on their own wall-clock time
main.goreturns elapsed time in the response body:/api/transferresult.Duration = time.Since(start).String()/api/batch-transferDuration: time.Since(start).String()/api/post-transfer"duration": time.Since(start).String()Those are exactly the three tests that fail.
get-health-1andget-health-2pass. A recorded"30.031440402s"will never equal a replayed"44.525254ms".Fixed with a checked-in
keploy.ymldeclaringdurationas global body noise. Checked in rather than patched from CI so./test.sh— which already enforces report status — behaves the same as the pipeline and the two cannot drift.keploy config --generateis a no-op when the file exists, so keploy's CI script keeps working unchanged.2. The sample recorded against the public internet
It drove 20 concurrent HTTPS requests at
https://httpbin.orgduring record, so whatever httpbin did that minute got baked into the expected body. One CI run recorded:Replay serves those from mocks in milliseconds and legitimately produced
20 / 0 / 200. The timeouts also ate one record window badly enough that only 3 of 5 test cases were captured at all.Replaced with a local Caddy stub on the compose network. It serves real HTTPS on 443 via
tls internal, so the CONNECT-tunnel and TLS-cert-caching paths this sample exists to stress are untouched — only the far end moves in-cluster, where it is deterministic. No CA plumbing is needed: the app already setsInsecureSkipVerifyto accept keploy's MITM cert, and keploy never validates upstream certs either. The/getbody is padded to roughly httpbin's size so the tunnel carries comparable data.Verification
Full record → replay cycle, locally, against a keploy built from
mainwith the private parsers:test-set-0-report.yamlsucceeded: 6, failed: 14, status: 503(CI)succeeded: 20, failed: 0, status: 200Also verified directly: the stub reaches
healthy,curl -k -x http://proxy:3128 https://httpstub/getreturns 200 through the CONNECT tunnel, and 20 concurrent GETs all return 200.Related
keploy.ymlto be present and fails loudly if it is not, so land this one first.Residual, stated plainly
/api/batch-transferstill asserts exactsucceeded/failedcounts over 20 concurrent connections. The stub removes the internet as a variable but not concurrency — if one connection fails under CI load, three body fields flip at once. With the gate armed that becomes a red lane rather than an invisible one, which is the right trade, but it is the thing most likely to bite next.