feat: support distributed bitmap index builds#6429
Open
butnaruandrei wants to merge 2 commits intolance-format:mainfrom
Open
feat: support distributed bitmap index builds#6429butnaruandrei wants to merge 2 commits intolance-format:mainfrom
butnaruandrei wants to merge 2 commits intolance-format:mainfrom
Conversation
…omponents - Updated `LanceDataset` to merge Bitmap index files alongside existing index types. - Introduced `BITMAP` as a supported index type in `SupportedDistributedIndices`. - Added a new test for building and merging Bitmap indices, ensuring compatibility with existing data. - Enhanced Rust implementation to handle Bitmap index creation and merging. This update expands the indexing capabilities of the Lance framework, allowing for more efficient data retrieval and management.
butnaruandrei
changed the title
butnaruandrei
changed the title
Contributor
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
butnaruandrei
changed the title
butnaruandrei
changed the title
Xuanwo
requested changes
Apr 8, 2026
Collaborator
Xuanwo
left a comment
Xuanwo
left a comment
There was a problem hiding this comment.
Thank you for working on this! However we have designed a new general distributed API and pipeline as showed up here: https://lance.org/guide/distributed_indexing/
Whenever possible, please adapt the index build API to this one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allows bitmap indexes to be built in a distributed/fragment-scoped manner,
consistent with the existing BTree and FTS distributed build patterns.
BitmapIndexPlugin::train_indexnow acceptsfragment_ids, writingper-shard
part_{partition_id}_bitmap_page_lookup.lancefiles insteadof erroring when
fragment_idsis setmerge_index_filesperforms a K-way sorted merge of shard files,unioning
RowAddrTreeMaps for matching keys, then writes the canonicalbitmap_page_lookup.lanceand deletes the shardsDataset::merge_index_metadatadispatchesIndexType::Bitmapto thenew merger
SupportedDistributedIndicesgains aBITMAPvariantNon-distributed builds (no
fragment_ids) are unchanged. After merge theon-disk layout is identical to a monolithic build.