Skip to content

phase 6a - replay functionality [source cache-scoped] #1045

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kans wants to merge 1 commit into
base: main
Choose a base branch
from
Open

phase 6a - replay functionality [source cache-scoped] #1045

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,14 +89,14 @@ demo-crash-check: crash-check ## Deprecated alias for crash-check.

.PHONY: checkpoint-cut-check
checkpoint-cut-check: ## Resume from every durable checkpoint cut.
BATON_CUT_SWEEP=full go test -v -count=1 -timeout=30m -run TestCheckpointCutEnumeration ./pkg/sync
BATON_TEST_EXTRA=1 BATON_CUT_SWEEP=full go test -v -count=1 -timeout=30m -run TestCheckpointCutEnumeration ./pkg/sync

.PHONY: interrupt-check
interrupt-check: checkpoint-cut-check crash-check ## Run in-process cut and real-process interruption checks.

.PHONY: race-check
race-check: ## Run the complete Go suite with the race detector.
go test -race -tags=baton_lambda_support -count=1 -timeout=45m ./...
BATON_TEST_EXTRA=1 go test -race -tags=baton_lambda_support -count=1 -timeout=45m ./...

@github-actions github-actions Bot Aug 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: ci.yaml and main.yaml run bare go test ./... with neither tier variable set, so every test newly gated by testtier.RequireExtra/RequireNightly now skips in all PR and main runs — including several that previously ran unconditionally (TestParallelQueueExhaustiveInterleavings, TestTopologicalMergeDifferentialRandom{,Store}, TestTopologicalMergeChunkedDirtyFlush, TestCleanupContextDeadlineExceeded) and the whole non-short set (TestC1ZIntegrity, TestWALCheckpointRace, all TestChaosConnector*Corpus). race-check here sets only BATON_TEST_EXTRA=1, so the chaos corpora that the deleted comment said "race-check already includes" now run only under nightly. Worth confirming that's the intended net coverage, or gating the cheap deterministic ones (e.g. TestTopologicalMergeChunkedDirtyFlush) at a lower tier.


# Nightly race shards. A serial instrumented ./... sweep is hours of work, and
# nearly all of it sits in a handful of packages, so the nightly workflow runs
Expand Down Expand Up @@ -158,13 +158,20 @@ race-shard-list: ## Print the packages in race shard SHARD.
# An empty list cannot get past the capture — the list ends in grep, which exits
# 1 when it matches nothing — so this reports it here rather than testing $$pkgs
# afterwards, which would never run.
#
# The shards are the nightly instrumented sweep, so they run both opt-in tiers.
# nightly.yaml invokes these directly rather than through test-nightly, so
# without the tier variables here the guarded tests would skip in every shard
# and the sweep would quietly cover less than the un-sharded race-check does.
# Extra is implied by nightly: a tier that runs the randomized and full-corpus
# cases has no reason to drop the deterministic long ones.
.PHONY: race-check-shard
race-check-shard: ## Race-check one nightly shard, for example SHARD=dotc1z.
@pkgs=$$($(MAKE) --no-print-directory race-shard-list SHARD=$(SHARD)) || { \
echo "race-check-shard: no packages for SHARD='$(SHARD)' (see above)" >&2; \
exit 1; \
}; \
set -x; go test -race -tags=$(RACE_TAGS) -count=1 -timeout=$(RACE_SHARD_TIMEOUT) $$pkgs
set -x; BATON_TEST_EXTRA=1 BATON_TEST_NIGHTLY=1 go test -race -tags=$(RACE_TAGS) -count=1 -timeout=$(RACE_SHARD_TIMEOUT) $$pkgs

# Each shard's list is captured rather than piped straight into wc, for the same
# reason as race-check-shard above: at the head of a pipe its exit status is
Expand Down Expand Up @@ -223,7 +230,7 @@ fuzz-smoke: ## Run each native Go fuzzer for FUZZ_TIME (default 30s).

.PHONY: differential-check
differential-check: ## Differential-fuzz SQLite and Pebble for DIFFERENTIAL_TIME.
BATON_EXPAND_FUZZ_DURATION=$(DIFFERENTIAL_TIME) go test -v -count=1 -timeout=30m -run '^TestFullPipelineDifferentialFuzz$$' ./pkg/sync/expand
BATON_TEST_EXTRA=1 BATON_EXPAND_FUZZ_DURATION=$(DIFFERENTIAL_TIME) go test -v -count=1 -timeout=30m -run '^TestFullPipelineDifferentialFuzz$$' ./pkg/sync/expand

.PHONY: bench-smoke
bench-smoke: ## Run the bounded checkpoint cost benchmarks once.
Expand All @@ -236,7 +243,7 @@ bench: ## Run curated checkpoint and medium full-sync benchmarks.

.PHONY: scheduler-soak
scheduler-soak: ## Run randomized scheduler cases under race detection.
BATON_SOAK_ITERATIONS=$(SOAK_ITERATIONS) go test -race -v -count=1 -timeout=30m -run TestSchedulerSoakRandomizedFanoutWithFailures ./pkg/sync
BATON_TEST_NIGHTLY=1 BATON_SOAK_ITERATIONS=$(SOAK_ITERATIONS) go test -race -v -count=1 -timeout=30m -run TestSchedulerSoakRandomizedFanoutWithFailures ./pkg/sync

.PHONY: errorfs-soak
errorfs-soak: ## Sweep whole-sync Pebble crash points using errorfs.
Expand All @@ -249,17 +256,19 @@ chaos-check: ## Run bounded representative chaos checks under race detection.

.PHONY: chaos-full-check
chaos-full-check: ## Run every deterministic chaos corpus under race detection.
go test -race -count=1 -timeout=30m -run '^TestChaosConnector' ./pkg/sync
BATON_TEST_NIGHTLY=1 go test -race -count=1 -timeout=30m -run '^TestChaosConnector' ./pkg/sync

.PHONY: chaos-soak
chaos-soak: ## Run extended seeded chaos connector fanout schedules.
BATON_CHAOS_ITERATIONS=$(CHAOS_ITERATIONS) go test -race -v -count=1 -timeout=30m -run TestChaosConnectorSeededFanoutWithRetries ./pkg/sync

# race-check already includes chaos-full-check's complete deterministic corpus.
# Full deterministic chaos corpora are reserved for test-nightly.
.PHONY: test-extra
test-extra: export BATON_TEST_EXTRA=1
test-extra: race-check compat-check interrupt-check fuzz-smoke differential-check bench-smoke ## Run bounded confidence checks omitted from CI.

.PHONY: test-nightly
test-nightly: export BATON_TEST_NIGHTLY=1

@github-actions github-actions Bot Aug 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: neither new tier has an automated runner. .github/workflows/ci.yaml:39 and main.yaml:56 invoke bare go test ./... with neither BATON_TEST_EXTRA nor BATON_TEST_NIGHTLY set, and there is no schedule:/workflow_dispatch: workflow that calls make test-extra or make test-nightly. As a result the 21 newly gated tests — the full deterministic chaos corpora, scheduler soak, WAL-checkpoint race, C1Z integrity, the randomized source-cache lifecycle model, checkpoint-cut enumeration, exhaustive parallel-queue interleavings, and both differential suites — now run only when a human types a Make target, which is coverage removal rather than relocation. Consider adding a nightly scheduled job that runs make test-nightly.

test-nightly: ## Run extended confidence, fuzz, scheduler, and errorfs checks.
$(MAKE) test-extra FUZZ_TIME=$(NIGHTLY_FUZZ_TIME) DIFFERENTIAL_TIME=$(NIGHTLY_DIFFERENTIAL_TIME)
$(MAKE) scheduler-soak
Expand Down
19 changes: 17 additions & 2 deletions docs/TESTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,22 @@ Tests in this tier should be deterministic, self-contained, and reasonably
fast. A test that only skips on Windows with `testing.Short()` is still a CI
test on the other platforms.

Long-running tests use two repository-wide opt-ins:

- `BATON_TEST_EXTRA=1` enables deterministic exhaustive and boundary-scale
tests.
- `BATON_TEST_NIGHTLY=1` enables randomized, repeated, and full-corpus tests;
it also satisfies extra-tier guards.

The named Make targets set these variables. Direct `go test` invocations omit
both tiers unless the caller explicitly sets the corresponding variable to
exactly `1`; values such as `0` and `false` leave the tier disabled.

## Bounded checks omitted from CI

`make test-extra` is the memorable pre-merge/pre-release command. It composes:

- `make race-check` — the complete Go suite under the race detector.
- `make race-check` — ordinary and extra-tier tests under the race detector.
- `make compat-check` — exchanges real checkpoint artifacts between HEAD and
a pinned older SDK release. Override the old release with
`BATON_COMPAT_OLD_REF=<tag>`.
Expand All @@ -41,10 +52,14 @@ that are inappropriate for every pull request.

## Nightly checks

`make test-nightly` runs `test-extra` with longer fuzz durations, then adds:
`make test-nightly` enables both opt-in tiers, runs `test-extra` with longer
fuzz durations, and then adds:

- `make scheduler-soak` — randomized scheduler fan-out and failure histories
under the race detector.
- the complete deterministic chaos corpora and randomized source-cache,
integrity, and WAL lifecycle checks.
- `make chaos-soak` — extended seeded chaos connector fan-out schedules.
- `make errorfs-soak` — randomized whole-sync Pebble failure-point sweeps
against a crashable filesystem.

Expand Down
Loading
Loading