Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assess continued use of TorchData #685

Open
atolopko-czi opened this issue Aug 3, 2023 · 0 comments
Open

Assess continued use of TorchData #685

atolopko-czi opened this issue Aug 3, 2023 · 0 comments
Labels
discovery Discovery work P1 Priority 1 - Improvement with wide impact, fix within 1 week python api Related to the API

Comments

@atolopko-czi
Copy link
Collaborator

atolopko-czi commented Aug 3, 2023

The TorchData library that the Census ExperimentDataPipe is implemented upon is no longer being actively developed: pytorch/data#1196.

Note that there are two separate IterDataPipe classes:

  • torch.utils.data.datapipes.datapipe.IterDataPipe
  • torchdata.datapipes.iter.IterDataPipe

Census API's ExperimentDataPipe uses the latter, currently. Even if we were to simply switch to usingtorch.utils.data.datapipes.datapipe.IterDataPipe, the larger issue is the expectation that TorchData DataPipe "operations" like RandomSplitter and Shuffler are available for use in client code. So we may want to provide equivalent, custom functionality in our ExperimentDataPipe, which would be unfortunate.

We can also keep our implementation "as is", since TorchData isn't being pulled away, it just won't see any improvements.

We certainly need to keep ExperimentDataPipe as an "experimental" Census API until we resolve this issue.

@pablo-gar pablo-gar added python api Related to the API P1 Priority 1 - Improvement with wide impact, fix within 1 week discovery Discovery work labels Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discovery Discovery work P1 Priority 1 - Improvement with wide impact, fix within 1 week python api Related to the API
Projects
None yet
Development

No branches or pull requests

2 participants