Migrating download utils to torchdata

[biphasic]

Hello,
following Patrick's suggestion, I'm opening an issue here to discuss if the torchvision maintainers would be open to move the download utils upstream to torchdata. I'm the maintainer of a similar package which is modeled after torchvision to a large extent and would benefit greatly from such a move, much like torchaudio/text I would assume.

cc @pmeier

[pmeier]

Thanks for the suggestion @biphasic.

be open to move the download utils upstream to torchdata.

We had an initial push for this in #6060. Unfortunately, we hit quite a few roadblocks that would need to be resolved before we could adapt that. In addition, the prototype datasets are currently on hold and thus we currently don't depend on torchdata. Meaning, we would need to add a dependency, albeit from a first-party library, just for this. Not sure if its worth it.

benefit greatly from such a move

Could you explain how exactly you would benefit as a downstream library from this? Our idea was to just switch the backend and thus the users would have no touching points with torchdata.

[biphasic]

prototype datasets are currently on hold

Why is that if I may ask? Is there a place where your team defines a roadmap for torchvision?

benefits

The benefit would be not having to rely on torchvision for download utils and torchdata for datapipes, but just torchdata for all of that functionality. I'm not sure I understand what you mean by saying that the users would have no touching points with torchdata, isn't the goal of torchvision/prototype to use datapipes?

[NicolasHug]

Hi @biphasic

Why is that if I may ask? Is there a place where your team defines a roadmap for torchvision?

We came to the conclusion that for now, there are more suitable opportunities to improve data-loading speed in torchvision. We might revisit in the future. Re roadmap: we used to disclose those more actively, but recent events have made that difficult. To give you an idea, the rough roadmap for now is to 1) release the transforms V2 work 2) improve data-loading speed via improvements to Resize() 3) revamp video decoding utilities.

Going back to the original goal of this issue: any additional dependency comes with significant maintenance cost, and there's no immediate plan to take a dependency on torchdata. So as @pmeier already mentioned, it's unlikely that we'll migrate our download functionality to torchdata (on top of the blockers @pmeier already noted).

Also @biphasic please note that while our download utils are effectively public, we really don't recommend that you rely on them in your library: those were introduced a long time ago as public but they should have been private from the beginning. We might deprecate and make them private in the future, so please migrate away from them if you can.

Regardless of whether torchvision relies on torchdata for download functionality, there shouldn't be anything preventing you from relying on torchdata on your side? The download functionalities over there are public, so perhaps this is a better alternative for you

[biphasic]

Thanks a lot for the clarification @NicolasHug . While I understand that datapipes might not bring a big speed improvement, for me there's a big plus in their API. Would you say then there's a risk of torchdata not being actively developed anymore? In that case I might hold off as well then for the time being

[NicolasHug]

Would you say then there's a risk of torchdata not being actively developed anymore

I don't think this is the case, but let me ping @ejguan and @NivekT who will be better suited to answer that

[vadimkantorov]

Related on need of a better / reliable download utility across PyTorch. I was arguing that the basic (at least http-based) such download-a-file utility should be included directly in core.

• Potential race conditions between multiple workers trying to download and cache the same file in torch.hub.load_state_dict_from_url and torch.hub.download_url_to_file <- duplicate dataset/model downloads across DDP workers pytorch#68320
• [feature request] torch.hub.load_state_dict_from_url to be replaced by a new good general download-a-file function and to also support local paths and google drive links / private github release links pytorch#73466
• Proxy/cache server option/hooks for downloading model checkpoints and dataset archive files in cloud environment pytorch#91965

@biphasic maintainers of core might be open to accept such contributions (see discussion in pytorch/pytorch#68320)

[NicolasHug]

@biphasic just making sure you saw it: https://github.com/pytorch/data/#torchdata-see-note-below-on-current-status

Since torchdata development is paused, we won't be migrating to torchdata utils anytime soon, so I'll close this issue. Feel free to provide feedback for torchdata on pytorch/data#1196

[biphasic]

@NicolasHug thanks a lot for letting me know!