You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Though we have process_webdataset, we also need a simple generator that just returns File objects for each member of an archive similarly to the old-style UDF index_tar.
This would allow us to test tar file handling in datachain.lib (which we're not currently doing, as mentioned in #412), and I consider it a blocker for #438 and #367.
The text was updated successfully, but these errors were encountered:
Though we have
process_webdataset
, we also need a simple generator that just returnsFile
objects for each member of an archive similarly to the old-style UDFindex_tar
.Usage should look like:
This would allow us to test tar file handling in
datachain.lib
(which we're not currently doing, as mentioned in #412), and I consider it a blocker for #438 and #367.The text was updated successfully, but these errors were encountered: