Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Raise DatasetTooSmall exception if canonical nodes is less than num samples #1518

Merged
merged 9 commits into from
Sep 12, 2024
Merged

Raise DatasetTooSmall exception if canonical nodes is less than num samples #1518

merged 9 commits into from
Sep 12, 2024

Conversation

irenedea
Copy link
Contributor

Raises a specialized error if you init a dataset that has too few samples.

@irenedea irenedea requested a review from a team as a code owner September 11, 2024 17:25
snarayan21
snarayan21 reviewed Sep 11, 2024
View reviewed changes
Copy link
Contributor

@snarayan21 snarayan21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unsure if these changes actually do what we want

llmfoundry/command_utils/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved
llmfoundry/command_utils/data_prep/convert_text_to_mds.py Show resolved Hide resolved
llmfoundry/data/finetuning/tasks.py Show resolved Hide resolved
llmfoundry/data/finetuning/tasks.py Show resolved Hide resolved
dakinggg
dakinggg approved these changes Sep 12, 2024
View reviewed changes
Copy link
Collaborator

@dakinggg dakinggg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets add a unit test for raising this error?

llmfoundry/command_utils/data_prep/convert_text_to_mds.py Outdated Show resolved Hide resolved
@irenedea irenedea merged commit 5465db4 into mosaicml:main Sep 12, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@snarayan21 snarayan21 snarayan21 left review comments

@dakinggg dakinggg dakinggg approved these changes

@jjanezhang jjanezhang Awaiting requested review from jjanezhang

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@irenedea @snarayan21 @jjanezhang @dakinggg