Assess continued use of TorchData #685
Labels
discovery
Discovery work
P1
Priority 1 - Improvement with wide impact, fix within 1 week
python api
Related to the API
The TorchData library that the Census ExperimentDataPipe is implemented upon is no longer being actively developed: pytorch/data#1196.
Note that there are two separate
IterDataPipe
classes:torch.utils.data.datapipes.datapipe.IterDataPipe
torchdata.datapipes.iter.IterDataPipe
Census API's
ExperimentDataPipe
uses the latter, currently. Even if we were to simply switch to usingtorch.utils.data.datapipes.datapipe.IterDataPipe
, the larger issue is the expectation that TorchData DataPipe "operations" likeRandomSplitter
andShuffler
are available for use in client code. So we may want to provide equivalent, custom functionality in ourExperimentDataPipe
, which would be unfortunate.We can also keep our implementation "as is", since TorchData isn't being pulled away, it just won't see any improvements.
We certainly need to keep
ExperimentDataPipe
as an "experimental" Census API until we resolve this issue.The text was updated successfully, but these errors were encountered: