Skip to content

TOOLS-4263 Convert mongorestore bulk sizing tests to Go - #1091

Draft
autarch wants to merge 1 commit into
graphite-base/1091from
08-06-tools-4263_convert_mongorestore_bulk_sizing_tests_to_go
Draft

TOOLS-4263 Convert mongorestore bulk sizing tests to Go#1091
autarch wants to merge 1 commit into
graphite-base/1091from
08-06-tools-4263_convert_mongorestore_bulk_sizing_tests_to_go

Conversation

@autarch

@autarch autarch commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Adds integration/dumprestore/bulk_test.go: two DumpRestoreSuite methods, five cases in total, covering how mongorestore batches documents when writing them back.

JS -> Go mapping (all Go tests live in integration/dumprestore/bulk_test.go):

  • test/qa-tests/jstests/restore/large_bulk.js -> TestRestoreLargeBulk (integration/dumprestore/bulk_test.go) - multi-megabyte documents survive a dump and restore byte for byte, batched together rather than written one at a time. Worth recording what this test does not cover: the JS was added for TOOLS-939, where mongorestore batched by document count alone and could build a write command larger than the server would accept, but it never actually reached that threshold. The buffered bulk inserter flushes at MAX_MESSAGE_SIZE_BYTES minus 1MB, roughly 47MB (common/db/buffered_bulk.go), not at the 16MB MaxBSONSize that limits a single document, and the JS fixture totaled only about 35MB. Growing the fixture past 47MB would make every CI topology and server version pay for it, and the size-based flush is already covered by unit tests in common/db/buffered_bulk_test.go (the "byte limit 1" and "full buffer with max-length namespace" cases), so this test keeps only the assertion it can honestly make.
  • test/qa-tests/jstests/restore/duplicate_keys.js -> TestRestoreDuplicateKeys (integration/dumprestore/bulk_test.go) - restoring a dump over a collection that still holds most of the dumped documents inserts the missing ones while the rest produce duplicate key errors, which mongorestore reports as failures without failing the restore. This is the first coverage of --batchSize. The JS asserted only the exit code and the final document count, which cannot tell "the ten missing documents were reinserted" apart from "all fifty were overwritten"; this marks the surviving documents after the dump is taken, so the marker exists nowhere in the dump and would disappear if a document were overwritten, and it asserts the exact split of inserts to duplicate-key failures. The JS looped --batchSize from 1 to 99, which is 99 dump and restore cycles for one behavior; this uses the default, 1, 7 so that the final batch is partial rather than full, and 51 so that every document lands in a single batch.

@autarch
autarch force-pushed the 08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go branch from a464f74 to 9af89c7 Compare August 7, 2026 18:41
@autarch
autarch force-pushed the 08-06-tools-4263_convert_mongorestore_bulk_sizing_tests_to_go branch from 7e7a9c9 to 8fd09dd Compare August 7, 2026 18:42
@autarch
autarch force-pushed the 08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go branch from 9af89c7 to fa54faa Compare August 7, 2026 18:57
@autarch
autarch force-pushed the 08-06-tools-4263_convert_mongorestore_bulk_sizing_tests_to_go branch 2 times, most recently from fd416b5 to a19b7e3 Compare August 10, 2026 20:44
@autarch
autarch force-pushed the 08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go branch from fa54faa to 79a1fc6 Compare August 10, 2026 20:44
@autarch
autarch force-pushed the 08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go branch from 79a1fc6 to 7424f8b Compare August 11, 2026 16:19
@autarch
autarch force-pushed the 08-06-tools-4263_convert_mongorestore_bulk_sizing_tests_to_go branch 2 times, most recently from 8158853 to e4aa6bb Compare August 11, 2026 17:11
@autarch
autarch force-pushed the 08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go branch from 7424f8b to 871f7d6 Compare August 11, 2026 17:11
@autarch
autarch force-pushed the 08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go branch from 871f7d6 to 1f368aa Compare August 12, 2026 18:43
Adds `integration/dumprestore/bulk_test.go`: two `DumpRestoreSuite` methods, five cases in total, covering how mongorestore batches documents when writing them back.

JS -> Go mapping (all Go tests live in `integration/dumprestore/bulk_test.go`):

- `test/qa-tests/jstests/restore/large_bulk.js` -> `TestRestoreLargeBulk` (`integration/dumprestore/bulk_test.go`) - multi-megabyte documents survive a dump and restore byte for byte, batched together rather than written one at a time. Worth recording what this test does *not* cover: the JS was added for TOOLS-939, where mongorestore batched by document count alone and could build a write command larger than the server would accept, but it never actually reached that threshold. The buffered bulk inserter flushes at `MAX_MESSAGE_SIZE_BYTES` minus 1MB, roughly 47MB (`common/db/buffered_bulk.go`), not at the 16MB `MaxBSONSize` that limits a single document, and the JS fixture totaled only about 35MB. Growing the fixture past 47MB would make every CI topology and server version pay for it, and the size-based flush is already covered by unit tests in `common/db/buffered_bulk_test.go` (the "byte limit 1" and "full buffer with max-length namespace" cases), so this test keeps only the assertion it can honestly make.
- `test/qa-tests/jstests/restore/duplicate_keys.js` -> `TestRestoreDuplicateKeys` (`integration/dumprestore/bulk_test.go`) - restoring a dump over a collection that still holds most of the dumped documents inserts the missing ones while the rest produce duplicate key errors, which mongorestore reports as failures without failing the restore. This is the first coverage of `--batchSize`. The JS asserted only the exit code and the final document count, which cannot tell "the ten missing documents were reinserted" apart from "all fifty were overwritten"; this marks the surviving documents after the dump is taken, so the marker exists nowhere in the dump and would disappear if a document were overwritten, and it asserts the exact split of inserts to duplicate-key failures. The JS looped `--batchSize` from 1 to 99, which is 99 dump and restore cycles for one behavior; this uses the default, 1, 7 so that the final batch is partial rather than full, and 51 so that every document lands in a single batch.
@autarch
autarch force-pushed the 08-06-tools-4263_convert_mongorestore_bulk_sizing_tests_to_go branch from e4aa6bb to bd5eab2 Compare August 12, 2026 18:43
@autarch
autarch changed the base branch from 08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go to graphite-base/1091 August 13, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant