From 9fafc0a82cf21350af13b0afad8741eb9cc74322 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Tue, 7 Jul 2026 19:57:41 +0100 Subject: [PATCH] Dask: show streaming read/write in bucket integrations The Dask section showed only a single-file read. Add the streaming write (the pandas section shows both), use directory paths for Dask's partition-per-file model, and link the canonical Dask guide. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_019Fm1TL6E81bEVxEdMEkML9 --- docs/hub/storage-buckets-integrations.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/hub/storage-buckets-integrations.md b/docs/hub/storage-buckets-integrations.md index 4a26c8a49..6014395dc 100644 --- a/docs/hub/storage-buckets-integrations.md +++ b/docs/hub/storage-buckets-integrations.md @@ -15,12 +15,17 @@ df.to_parquet("hf://buckets/username/my-bucket/output.parquet") ## Dask +Dask reads and writes lazily, one partition per file, so you can process datasets larger than memory straight from a bucket: + ```python import dask.dataframe as dd -df = dd.read_parquet("hf://buckets/username/my-bucket/data.parquet") +df = dd.read_parquet("hf://buckets/username/my-bucket/data/") +df.to_parquet("hf://buckets/username/my-bucket/output/") ``` +See [Dask on the Hub](./datasets-dask) for more. + ## Daft Daft supports `hf://buckets/` paths natively, with [Xet-accelerated reads](https://docs.daft.ai/en/stable/connectors/huggingface/) enabled by default: