Skip to content

feat: add runtime per-base store params for dataset reads#6422

Open
Xuanwo wants to merge 1 commit intomainfrom
xuanwo/per-base-runtime-store-params
Open

feat: add runtime per-base store params for dataset reads#6422
Xuanwo wants to merge 1 commit intomainfrom
xuanwo/per-base-runtime-store-params

Conversation

@Xuanwo
Copy link
Copy Markdown
Collaborator

@Xuanwo Xuanwo commented Apr 7, 2026

This adds runtime-only per-base object store params for dataset reads, keyed by BasePath.path, so callers can open different registered bases with different storage options without extending BasePath in the manifest format. The builder now normalizes per-base storage options into ObjectStoreParams once at load time, and Dataset::object_store_for_base uses those params when resolving stores.

This follows the current direction from discussion #6307 to keep provider-specific configuration in runtime state instead of the table format.

@github-actions github-actions bot added the enhancement New feature or request label Apr 7, 2026
@Xuanwo Xuanwo marked this pull request as ready for review April 7, 2026 08:51
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 7, 2026

Codecov Report

❌ Patch coverage is 85.71429% with 11 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance/src/dataset/builder.rs 80.00% 10 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Copy link
Copy Markdown
Member

@westonpace westonpace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be ok as is but I have a few questions.

Is it possible to set an option for the default base? Or would you need to override that option in every base?
Can you mask an option to hide it from the bases? For example, maybe the default storage options has a proxy_url but you don't want to use that in the non-default bases?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants