diff --git a/docs/hub/storage-buckets-integrations.md b/docs/hub/storage-buckets-integrations.md index 4a26c8a49..6014395dc 100644 --- a/docs/hub/storage-buckets-integrations.md +++ b/docs/hub/storage-buckets-integrations.md @@ -15,12 +15,17 @@ df.to_parquet("hf://buckets/username/my-bucket/output.parquet") ## Dask +Dask reads and writes lazily, one partition per file, so you can process datasets larger than memory straight from a bucket: + ```python import dask.dataframe as dd -df = dd.read_parquet("hf://buckets/username/my-bucket/data.parquet") +df = dd.read_parquet("hf://buckets/username/my-bucket/data/") +df.to_parquet("hf://buckets/username/my-bucket/output/") ``` +See [Dask on the Hub](./datasets-dask) for more. + ## Daft Daft supports `hf://buckets/` paths natively, with [Xet-accelerated reads](https://docs.daft.ai/en/stable/connectors/huggingface/) enabled by default: