[Dataset hub] more than storage: streaming, editing, connectors #2052

lhoestq · 2025-11-13T18:39:08Z

streaming
editing (cc @cfahlgren1 )
connectors

I also touched the libraries page a bit cc @davanstrien

HuggingFaceDocBuilderDev · 2025-11-13T18:41:01Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

lhoestq · 2025-11-14T10:18:44Z

docs/hub/datasets-streaming.md

+<div class="flex justify-center">
+<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage.png"/>
+<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage-dark.png"/>
+</div>


maybe we should have a toggle or something to show the Streaming code in addition to the Download code ? wdyt @cfahlgren1 ?

or if not a toggle, at least both snippets one above the other?

Good idea. We already show one snippet per subset (up to N) if there are multiple subsets. But instead we can show only one subset with something like this and a second snippet for streaming

subset = "first_subset" # One of "first_subset", "second_subset", etc.

julien-c

this is great 🔥 important stuff

julien-c · 2025-11-14T18:02:14Z

docs/hub/datasets-downloading.md

 ## Integrated libraries

-If a dataset on the Hub is tied to a [supported library](./datasets-libraries), loading the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/Samsung/samsum?library=datasets) shows how to do so with 🤗 Datasets below.
+If a dataset on the Hub is tied to a [supported library](./datasets-libraries), loading the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with 🤗 Datasets below.


julien-c · 2025-11-14T18:06:30Z

docs/hub/datasets-streaming.md

+
+## Integrated libraries
+
+If a dataset on the Hub is tied to a [supported library](./datasets-libraries) that allows streaming from Hugging Face, streaming the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with 🤗 Datasets below.


Suggested change

If a dataset on the Hub is tied to a [supported library](./datasets-libraries) that allows streaming from Hugging Face, streaming the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with 🤗 Datasets below.

If a dataset on the Hub is tied to a [supported library](./datasets-libraries) that allows streaming from Hugging Face, streaming the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`knkarthick/samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with 🤗 Datasets below.

docs/hub/datasets-streaming.md

julien-c · 2025-11-14T18:09:25Z

docs/hub/datasets-streaming.md

+from huggingface_hub import HfFileSystem
+
+fs = HfFileSystem()
+
+repo_id = "allenai/c4"
+path_in_repo = "en/c4-train.00000-of-01024.json.gz"
+
+# Stream the file
+with fs.open(f"datasets/{repo_id}/{path_in_repo}", "r", compression="gzip") as f:
+    print(f.readline())  # read only the first line
+    # {"text":"Beginners BBQ Class Taking Place in Missoula!...}


(nit, DX) any way to make the snippet even more simpler/more compact? for instance do we need to instantiate a HfFileSystem or we could have syntactic sugar maybe?

like a from huggingface_hub import fs you mean?

maybe this ?

import huggingface_hub with huggingface_hub.open(...) as f: ...

yes, one of those two could be very nice

from huggingface_hub import hffs maybe if we go w/ @Wauplin proposition

added from huggingface_hub import hffs here: huggingface/huggingface_hub#3556

julien-c · 2025-11-14T18:11:28Z

docs/hub/datasets-streaming.md

+
+Parquet is a great format for AI datasets. It offers good compression, a columnar structure for efficient processing and projections, and multi-level metadata for fast filtering, and is suitable for datasets of all sizes.
+
+Parquet files are divided in row groups that are often around 100MB each. This lets data loaders and data processing frameworks stream data progressively, iterating on row groups.


we do need a doc page about CDC Parquet btw!! cc @kszucs too

julien-c · 2025-11-14T18:12:58Z

docs/hub/datasets-streaming.md

+
+### Efficient random access
+
+Row groups are further divied into columns, and columns into pages. Pages are often around 1MB and are the smallest unit of data in Parquet, since this is where compression is applied. Accessing pages enables loading specific rows without having to load a full row group, and is possible if the Parquet file has a page index. However not every Parquet frameworks supports reading at the page level. PyArrow doesn't for example, but the `parquet` crate in Rust does:


add a good visual of Parquet file format, and btw we should have a /docs/hub/parquet page probably...

internal issue for discussion

julien-c · 2025-11-14T18:15:00Z

docs/hub/datasets-libraries.md

+
+### Optimized Parquet files
+
+Parquet files on Hugging Face are optimized to improve storage efficiency, accelerate downloads and uploads, and enable efficient dataset streaming and editing:


add a visual of how those files or datasets are marked on the Hub 😁

hint hint @lhoestq =)

internal issue for discussion

julien-c · 2025-11-14T18:18:23Z

docs/hub/datasets-editing.md

+
+## Using the `huggingface_hub` client library
+
+The rich features set in the `huggingface_hub` library allows you to manage repositories, including editing dataset files on the Hub. Visit [the client library's documentation](/docs/huggingface_hub/index) to learn more.


should embed at least an example of two (same in the "downloading" doc btw)

julien-c · 2025-11-14T18:19:47Z

docs/hub/datasets-editing.md

+Therefore the amount of data to reupload depends on the edits and the file structure.
+
+The Parquet format is columnar and compressed at the page level (pages are around ~1MB).
+We optimized Parquet for Xet with [Parquet Content Defined Chunking](https://huggingface.co/blog/parquet-cdc), which ensures unchanged data generally result in unchanged pages.


add a visual of how those files and/or datasets are marked on the Hub 😁

davanstrien

Looks very nice! Can take another look, but added some small language suggestions already.

docs/hub/datasets-editing.md

davanstrien · 2025-11-17T10:10:15Z

docs/hub/datasets-libraries.md

+
+_+s: Supports Streaming_
+_+p: Writes optimized Parquet files_
+_+p*: Requires passing extra arguments to write optimized Parquet files_


docs/hub/datasets-libraries.md

docs/hub/datasets-streaming.md

hanouticelina

looks great! i left some minor comments to improve readability

docs/hub/datasets-streaming.md

docs/hub/datasets-editing.md

docs/hub/datasets-streaming.md

Wauplin

Very nice additions! 🔥

docs/hub/datasets-downloading.md

docs/hub/datasets-editing.md

Wauplin · 2025-11-17T15:45:03Z

docs/hub/datasets-editing.md

+Click on **Toggle edit mode** to enable dataset editing.
+
+<div class="flex justify-center">
+<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/toggle_edit_button-min.png"/>


~~I'm getting a 404 on https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-edit/toggle_edit_button-min.png~~

EDIT: it was a file name issue https://huggingface.co/datasets/huggingface/documentation-images/commit/64bd5b3ceebe8f879aa6385bfc0930ae76cb8a75

docs/hub/datasets-editing.md

docs/hub/datasets-streaming.md

Wauplin · 2025-11-17T16:17:41Z

docs/hub/datasets-streaming.md

+<div class="flex justify-center">
+<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage.png"/>
+<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/datasets-usage-dark.png"/>
+</div>


or if not a toggle, at least both snippets one above the other?

Wauplin · 2025-11-17T16:19:28Z

docs/hub/datasets-streaming.md

+from huggingface_hub import HfFileSystem
+
+fs = HfFileSystem()
+
+repo_id = "allenai/c4"
+path_in_repo = "en/c4-train.00000-of-01024.json.gz"
+
+# Stream the file
+with fs.open(f"datasets/{repo_id}/{path_in_repo}", "r", compression="gzip") as f:
+    print(f.readline())  # read only the first line
+    # {"text":"Beginners BBQ Class Taking Place in Missoula!...}


like a from huggingface_hub import fs you mean?

docs/hub/datasets-streaming.md

Wauplin · 2025-11-17T16:23:38Z

docs/hub/datasets-streaming.md

+
+Parquet is a great format for AI datasets. It offers good compression, a columnar structure for efficient processing and projections, and multi-level metadata for fast filtering, and is suitable for datasets of all sizes.
+
+Parquet files are divided in row groups that are often around 100MB each. This lets data loaders and data processing frameworks stream data progressively, iterating on row groups.


Parquet files are divided in row groups that are often around 100MB each.

Above you mentioned

The Parquet format is columnar and compressed at the page level (pages are around ~1MB).

Are row groups and parquet pages the same or not? (and if yes, which value is the good one?).

Parquet files are made of row groups,
which are made of columns
which are made of pages

:)

happy to explain this more in the paragraph mentioning pages

Co-authored-by: Lucain <lucain@huggingface.co> Co-authored-by: célina <hanouticelina@gmail.com> Co-authored-by: Daniel van Strien <davanstrien@users.noreply.github.com> Co-authored-by: Julien Chaumond <julien@huggingface.co>

cfahlgren1 · 2025-11-17T22:52:59Z

docs/hub/datasets-editing.md

+> [!WARNING]
+> This feature is only available for CSV datasets for now.


Nice! Will keep in mind when expanding in future

docs/hub/datasets-editing.md

cfahlgren1

some small suggestions, but looks good! nice work!

burtenshaw

Very cool features! I left a few small nits mainly just readability.

docs/hub/datasets-editing.md

docs/hub/datasets-libraries.md

docs/hub/datasets-streaming.md

Co-authored-by: Caleb Fahlgren <cfahlgren1@gmail.com> Co-authored-by: burtenshaw <ben.burtenshaw@gmail.com>

lhoestq added 4 commits November 13, 2025 19:37

index

637c099

wip

9a13dab

update libraries page

a607ad2

datasets streaming

97fe466

lhoestq commented Nov 14, 2025

View reviewed changes

datasets editing

fd043a2

julien-c reviewed Nov 14, 2025

View reviewed changes

davanstrien reviewed Nov 17, 2025

View reviewed changes

hanouticelina reviewed Nov 17, 2025

View reviewed changes

Wauplin reviewed Nov 17, 2025

View reviewed changes

Apply suggestions from code review

adff916

Co-authored-by: Lucain <lucain@huggingface.co> Co-authored-by: célina <hanouticelina@gmail.com> Co-authored-by: Daniel van Strien <davanstrien@users.noreply.github.com> Co-authored-by: Julien Chaumond <julien@huggingface.co>

cfahlgren1 reviewed Nov 17, 2025

View reviewed changes

docs/hub/datasets-editing.md Outdated Show resolved Hide resolved

cfahlgren1 reviewed Nov 17, 2025

View reviewed changes

docs/hub/datasets-editing.md Outdated Show resolved Hide resolved

cfahlgren1 approved these changes Nov 17, 2025

View reviewed changes

burtenshaw reviewed Nov 18, 2025

View reviewed changes

Apply suggestions from code review

babca3f

Co-authored-by: Caleb Fahlgren <cfahlgren1@gmail.com> Co-authored-by: burtenshaw <ben.burtenshaw@gmail.com>


		## Integrated libraries

		If a dataset on the Hub is tied to a [supported library](./datasets-libraries) that allows streaming from Hugging Face, streaming the dataset can be done in just a few lines. For information on accessing the dataset, you can click on the "Use this dataset" button on the dataset page to see how to do so. For example, [`samsum`](https://huggingface.co/datasets/knkarthick/samsum?library=datasets) shows how to do so with 🤗 Datasets below.


		Parquet is a great format for AI datasets. It offers good compression, a columnar structure for efficient processing and projections, and multi-level metadata for fast filtering, and is suitable for datasets of all sizes.

		Parquet files are divided in row groups that are often around 100MB each. This lets data loaders and data processing frameworks stream data progressively, iterating on row groups.


		### Efficient random access

		Row groups are further divied into columns, and columns into pages. Pages are often around 1MB and are the smallest unit of data in Parquet, since this is where compression is applied. Accessing pages enables loading specific rows without having to load a full row group, and is possible if the Parquet file has a page index. However not every Parquet frameworks supports reading at the page level. PyArrow doesn't for example, but the `parquet` crate in Rust does:


		### Optimized Parquet files

		Parquet files on Hugging Face are optimized to improve storage efficiency, accelerate downloads and uploads, and enable efficient dataset streaming and editing:


		## Using the `huggingface_hub` client library

		The rich features set in the `huggingface_hub` library allows you to manage repositories, including editing dataset files on the Hub. Visit [the client library's documentation](/docs/huggingface_hub/index) to learn more.

		> [!WARNING]
		> This feature is only available for CSV datasets for now.

[Dataset hub] more than storage: streaming, editing, connectors #2052

Are you sure you want to change the base?

[Dataset hub] more than storage: streaming, editing, connectors #2052

Conversation

lhoestq commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Nov 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

julien-c left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davanstrien left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hanouticelina left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Wauplin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lhoestq commented Nov 13, 2025 •

edited

Loading

burtenshaw left a comment •

edited

Loading