Support iterable / webdataset

[richardrl]

What would be required to make this work for very large video sets? Would it be some integration with web dataset?

[RaivoKoot]

Depends how large. in general your dataset can be as big as you want. I've used the code with a video dataset where the videos totalled 1.5 terrabyte. However, those 1.5 terrabyte did fit on my local disk. Once your dataset is too big to fit on a single machine or if you want to do multi-node training you would have to make some changes to the code to support this somehow.