Add load dataset #253
Merged
Add load dataset #253
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cevian
force-pushed
the
add_load_dataset
branch
from
4be25d7 to
9b0a932
Compare
jgpruitt
approved these changes
Dec 4, 2024
jgpruitt
approved these changes
Dec 9, 2024
| # Check if table exists | ||
| result = plpy.execute( | ||
| f""" | ||
| SELECT pg_catalog.to_regclass('{qualified_table}')::text as friendly_table_name |
There was a problem hiding this comment.
does this strip the schema if the search_path allows for it?
There was a problem hiding this comment.
yes it does. Is that a problem?
| , field_types jsonb default null | ||
| , batch_size int default 5000 | ||
| , max_batches int default null | ||
| , commit_every_n_batches int default null |
There was a problem hiding this comment.
are you sure you want commit_every_n_batches defaulted to null?
cevian
force-pushed
the
add_load_dataset
branch
2 times, most recently
from
30957d4 to
3df651f
Compare
This function allows you to create a table from any dataset on HuggingFace Hub. It's an easy way to load data for testing and experimentation. The hub contains 250k+ datasets. We use a streaming version of the API to allow for memory-efficient processing and to optimize the case where you want less than the entire dataset. On-disk caching should be minimal using streaming. We also add a multi-txn version of load_dataset.
cevian
force-pushed
the
add_load_dataset
branch
from
3df651f to
c7f0b68
Compare
JamesGuthrie
pushed a commit
that referenced
this pull request
Dec 10, 2024
This function allows you to create a table from any dataset on HuggingFace Hub. It's an easy way to load data for testing and experimentation. The hub contains 250k+ datasets. We use a streaming version of the API to allow for memory-efficient processing and to optimize the case where you want less than the entire dataset. On-disk caching should be minimal using streaming. We also add a multi-txn version of load_dataset.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This function allows you to create a table from any dataset on
HuggingFace Hub. It's an easy way to load data for testing and
experimentation. The hub contains 250k+ datasets.
We use a streaming version of the API to allow for memory-efficient
processing and to optimize the case where you want less than
the entire dataset. On-disk caching should be minimal using streaming.