Reading from a tf.data Dataset

[modernAlcibiades]

I have created a batch data iterator using tf.data api and tensorflow iterator but I do not know how to plug this into returnn. Is this functionality already implemented?

[albertz]

This is planned and partly implemented. (You find some outdated discussion in the documentation of TFDataPipeline.py. This was written when the tf.data API did not exist yet.)

In principle, when ExternData gets created, it should not create tf.placeholder but use the data iterator instead. This by itself is easy. But the handling for multiple epochs, etc, this can make it more tricky.

Anyway, this is on my TODO list. But not sure when I get time for it.

[modernAlcibiades]

thanks.

[albertz]

Btw, just for reference: The current way to implement a dataset is to derive from the RETURNN class Dataset (or maybe better CachedDataset2 or so; see many of the existing examples). This is totally independent from TF.

[modernAlcibiades]

Yeah, I have currently locally modified CachedDataset2 to take tf.data as input and return that as output to _collect_single_seqs, but I am having trouble replacing TFDataPipeline as direct eval and assignment to extern_data.data[key].placeholder causes shape issues in multiple layers for some reason I am unable to decipher.
This error occurs in 2018-asr-attention experiment in the s layer inside output rec layer.
InvalidArgumentError (see above for traceback): ConcatOp : Dimensions of inputs should match: shape[0] = [58,10,1000] vs. shape[1] = [57,10,1000]

[albertz]

@wd929 This is not related to this post at all, or is it? To answer your question: You can use any you like. E.g. OggZipDataset (see tool bliss-to-ogg-zip), or HDFDataset (see tool hdf_dump). But even the original LibriSpeechDataset (just prepare it accordingly). See the documentation.
Please don't ask further questions on this here in this issue, which is not related to that. Open a new issue if you think there is a bug in Returnn. If you have questions, please post it on StackOverflow and we can answer it there.

[albertz]

I'm closing this now in favor of #292.