Skip to content

Issues: Lightning-AI/litData

Beta
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

50 Open 153 Closed
50 Open 153 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Support for Pathlib in litdata.map enhancement New feature or request
#581 opened May 5, 2025 by SkafteNicki
1
There is still a risk of race condition when indexing an s3 bucket with multi node (Parquet datasets)
#578 opened May 2, 2025 by bhimrazy
litdata.optimize() function returns without raising error, prior to all processes finishing work bug Something isn't working help wanted Extra attention is needed waiting on author Waiting for user input or feedback.
#575 opened Apr 29, 2025 by JacobARose
1
Add example for hugging face dataset optimization and benchmarks reports documentation Improvements or additions to documentation
#558 opened Apr 17, 2025 by bhimrazy
@bhimrazy
Refactor Data Processor to Use a Shared Queue Across Workers enhancement New feature or request
#556 opened Apr 16, 2025 by bhimrazy
1
How to correctly mix multiple StreamingDataset to create new data?
#554 opened Apr 13, 2025 by philgzl
1
5
OutOfBoundsError when streaming parquet files with low_memory=True bug Something isn't working help wanted Extra attention is needed waiting on author Waiting for user input or feedback.
#553 opened Apr 13, 2025 by kyoungrok0517
9
Duplicate UserWarning Logs for lightning-sdk Version Check bug Something isn't working help wanted Extra attention is needed
#527 opened Mar 24, 2025 by deependujha
@deependujha
4
Cycle option for StreamingDataLoader enhancement New feature or request waiting on author Waiting for user input or feedback.
#524 opened Mar 24, 2025 by Aceticia
1
Local cache dir not fully clearing in DDP multi-node training. bug Something isn't working help wanted Extra attention is needed
#512 opened Mar 12, 2025 by JackUrb
11
Add batch sampler for StreamDataloader to enable flexiable training strategy. enhancement New feature or request
#509 opened Mar 11, 2025 by xinsir6
1
How to optimimize dataset for pretraining from HuggingFace bug Something isn't working question Further information is requested
#482 opened Feb 21, 2025 by TheLukaDragar
1
7
Add pytest fixture to limit max time a test can take bug Something isn't working help wanted Extra attention is needed
#475 opened Feb 17, 2025 by deependujha
2
CI error: All chunks should've been deleted keeps coming back bug Something isn't working help wanted Extra attention is needed won't fix
#437 opened Dec 20, 2024 by deependujha
1
Restart training with new data, mid-epoch enhancement New feature or request won't fix
#436 opened Dec 17, 2024 by schopra8
1
Question: Is there a list for publicly available s3 links of datasets of litdata.StreamingDataset format? question Further information is requested won't fix
#430 opened Dec 2, 2024 by 2catycm
3
Clear Examples of use with different dataset types and code changes. enhancement New feature or request won't fix
#409 opened Nov 4, 2024 by Woodr7
3
incorrect dataloader length when drop_last=False bug Something isn't working help wanted Extra attention is needed won't fix
#402 opened Oct 28, 2024 by grez72
2
Improve CombinedStreamingDataset to handle multiple subdatasets efficiently enhancement New feature or request
#386 opened Oct 2, 2024 by bhimrazy
1
The config isn't consistent between chunks bug Something isn't working help wanted Extra attention is needed waiting on author Waiting for user input or feedback.
#370 opened Sep 17, 2024 by gluonfield
6
How can I shut down automatically distributing data when using StreamingDataset? enhancement New feature or request question Further information is requested won't fix
#368 opened Sep 12, 2024 by ygtxr1997
4
StreamingDataset causes NCCL timeout when using multiple nodes bug Something isn't working help wanted Extra attention is needed
#340 opened Aug 26, 2024 by hubenjm
26
StreamingDataset intermittently fails due to lack of index.json bug Something isn't working help wanted Extra attention is needed won't fix
#337 opened Aug 20, 2024 by plra
7
Use different batch sizes in CombinedStreamingDataset enhancement New feature or request help wanted Extra attention is needed won't fix
#327 opened Aug 10, 2024 by schopra8
2
Add support for multi sample item in optimize and yielding from the _getitem_ of the StreamingDataset enhancement New feature or request help wanted Extra attention is needed won't fix
#317 opened Aug 8, 2024 by tchaton
1
ProTip! Add no:assignee to see everything that’s not assigned.