TOOLS-4263 Convert mongorestore bulk sizing tests to Go - #1091
Draft
autarch wants to merge 1 commit into
Draft
Conversation
Collaborator
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
This was referenced Aug 6, 2026
autarch
force-pushed
the
08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go
branch
from
August 7, 2026 18:41
a464f74 to
9af89c7
Compare
autarch
force-pushed
the
08-06-tools-4263_convert_mongorestore_bulk_sizing_tests_to_go
branch
from
August 7, 2026 18:42
7e7a9c9 to
8fd09dd
Compare
autarch
force-pushed
the
08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go
branch
from
August 7, 2026 18:57
9af89c7 to
fa54faa
Compare
autarch
force-pushed
the
08-06-tools-4263_convert_mongorestore_bulk_sizing_tests_to_go
branch
2 times, most recently
from
August 10, 2026 20:44
fd416b5 to
a19b7e3
Compare
autarch
force-pushed
the
08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go
branch
from
August 10, 2026 20:44
fa54faa to
79a1fc6
Compare
autarch
force-pushed
the
08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go
branch
from
August 11, 2026 16:19
79a1fc6 to
7424f8b
Compare
autarch
force-pushed
the
08-06-tools-4263_convert_mongorestore_bulk_sizing_tests_to_go
branch
2 times, most recently
from
August 11, 2026 17:11
8158853 to
e4aa6bb
Compare
autarch
force-pushed
the
08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go
branch
from
August 11, 2026 17:11
7424f8b to
871f7d6
Compare
autarch
force-pushed
the
08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go
branch
from
August 12, 2026 18:43
871f7d6 to
1f368aa
Compare
Adds `integration/dumprestore/bulk_test.go`: two `DumpRestoreSuite` methods, five cases in total, covering how mongorestore batches documents when writing them back. JS -> Go mapping (all Go tests live in `integration/dumprestore/bulk_test.go`): - `test/qa-tests/jstests/restore/large_bulk.js` -> `TestRestoreLargeBulk` (`integration/dumprestore/bulk_test.go`) - multi-megabyte documents survive a dump and restore byte for byte, batched together rather than written one at a time. Worth recording what this test does *not* cover: the JS was added for TOOLS-939, where mongorestore batched by document count alone and could build a write command larger than the server would accept, but it never actually reached that threshold. The buffered bulk inserter flushes at `MAX_MESSAGE_SIZE_BYTES` minus 1MB, roughly 47MB (`common/db/buffered_bulk.go`), not at the 16MB `MaxBSONSize` that limits a single document, and the JS fixture totaled only about 35MB. Growing the fixture past 47MB would make every CI topology and server version pay for it, and the size-based flush is already covered by unit tests in `common/db/buffered_bulk_test.go` (the "byte limit 1" and "full buffer with max-length namespace" cases), so this test keeps only the assertion it can honestly make. - `test/qa-tests/jstests/restore/duplicate_keys.js` -> `TestRestoreDuplicateKeys` (`integration/dumprestore/bulk_test.go`) - restoring a dump over a collection that still holds most of the dumped documents inserts the missing ones while the rest produce duplicate key errors, which mongorestore reports as failures without failing the restore. This is the first coverage of `--batchSize`. The JS asserted only the exit code and the final document count, which cannot tell "the ten missing documents were reinserted" apart from "all fifty were overwritten"; this marks the surviving documents after the dump is taken, so the marker exists nowhere in the dump and would disappear if a document were overwritten, and it asserts the exact split of inserts to duplicate-key failures. The JS looped `--batchSize` from 1 to 99, which is 99 dump and restore cycles for one behavior; this uses the default, 1, 7 so that the final batch is partial rather than full, and 51 so that every document lands in a single batch.
autarch
force-pushed
the
08-06-tools-4263_convert_mongorestore_bulk_sizing_tests_to_go
branch
from
August 12, 2026 18:43
e4aa6bb to
bd5eab2
Compare
autarch
changed the base branch from
08-06-tools-4263_convert_mongorestore_index_round-trip_tests_to_go
to
graphite-base/1091
August 13, 2026 20:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Adds
integration/dumprestore/bulk_test.go: twoDumpRestoreSuitemethods, five cases in total, covering how mongorestore batches documents when writing them back.JS -> Go mapping (all Go tests live in
integration/dumprestore/bulk_test.go):test/qa-tests/jstests/restore/large_bulk.js->TestRestoreLargeBulk(integration/dumprestore/bulk_test.go) - multi-megabyte documents survive a dump and restore byte for byte, batched together rather than written one at a time. Worth recording what this test does not cover: the JS was added for TOOLS-939, where mongorestore batched by document count alone and could build a write command larger than the server would accept, but it never actually reached that threshold. The buffered bulk inserter flushes atMAX_MESSAGE_SIZE_BYTESminus 1MB, roughly 47MB (common/db/buffered_bulk.go), not at the 16MBMaxBSONSizethat limits a single document, and the JS fixture totaled only about 35MB. Growing the fixture past 47MB would make every CI topology and server version pay for it, and the size-based flush is already covered by unit tests incommon/db/buffered_bulk_test.go(the "byte limit 1" and "full buffer with max-length namespace" cases), so this test keeps only the assertion it can honestly make.test/qa-tests/jstests/restore/duplicate_keys.js->TestRestoreDuplicateKeys(integration/dumprestore/bulk_test.go) - restoring a dump over a collection that still holds most of the dumped documents inserts the missing ones while the rest produce duplicate key errors, which mongorestore reports as failures without failing the restore. This is the first coverage of--batchSize. The JS asserted only the exit code and the final document count, which cannot tell "the ten missing documents were reinserted" apart from "all fifty were overwritten"; this marks the surviving documents after the dump is taken, so the marker exists nowhere in the dump and would disappear if a document were overwritten, and it asserts the exact split of inserts to duplicate-key failures. The JS looped--batchSizefrom 1 to 99, which is 99 dump and restore cycles for one behavior; this uses the default, 1, 7 so that the final batch is partial rather than full, and 51 so that every document lands in a single batch.