Skip to content

Conversation

@VijayVignesh1
Copy link

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

Fixes #317

PR review

Added support for multisample item.

Basically added a boolean parameter which creates a batch of sub samples for each sample, given a list of transform functions.

Sample code:

    def transform_fn_sq(x, *args, **kwargs):
        """A simple transform function that doubles the input."""
        return x * 2

    def transform_fn_add(x):
        """A simple transform function that adds 3 to the input."""
        return x + 3

    def transform_fn_identity(x):
        """A simple transform function that returns the input as is."""
        return x

    dataset = StreamingDataset(
        data_dir,
        cache_dir=str(cache_dir),
        shuffle=False,
        transform=[transform_fn_sq, transform_fn_add, transform_fn_identity],
        is_multisample=True,
    )

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@VijayVignesh1 VijayVignesh1 force-pushed the feature/add_multisample_support branch from 1b01b6f to 6a77302 Compare October 24, 2025 20:12
@VijayVignesh1
Copy link
Author

@tchaton @deependujha @bhimrazy Can you verify the approach once? I can then make changes to the README.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for multi sample item in optimize and yielding from the _getitem_ of the StreamingDataset

1 participant