Is there a way to convert a custom keras.utils.Sequence custom class to a tf.Data pipeline?

[Abhishaike]

When I was building up my data pipeline, the Tensorflow docs were very insistent that generators are unsafe for multiprocessing, and that the best way to build up a multiprocessing streaming pipeline is to extend tensorflow.keras.utils.Sequence into your own custom class. This is written here: https://www.tensorflow.org/api_docs/python/tf/keras/utils/Sequence

So I did that, but now Tensorflow is telling me that Sequence extensions are ALSO not ideal for multiprocessing through the warning message multiprocessing can interact badly with TensorFlow, causing nondeterministic deadlocks. For high performance data pipelines tf.data is recommended.. So now the recommendation is to use tf.Data. And, as it were, I keep running into deadlocks 4~ epochs into training now.

Is there no converter between an existing sequence class and a tf.Data pipeline? It seems bizarre that the EXACT thing the Sequence extension class is recommended for seems to no longer work, and now only a brand new type of data pipeline will do the multiprocessing job. At the very least, this should be updated in the Sequence docs.

[gowthamkpr]

Can you please explain more about your customs class. To understand the difference between keras.utils.Sequence and a tf.Data pipeline you can take a look at the following question.

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[gowthamkpr]

Closing this issue as it has been inactive for 2 weeks. Please add additional comments for us to open this issue again. Thanks!

[MaxSchambach]

I was facing exactly the same question. In particular when upgrading certain routines such as data generation from TF1 to TF2. The mentioned statement in the documentation seems confusing. I am also facing deadlocks irregularly and in a non-reproducible manner. When using generators, derived from keras.utils.Sequence and passed e.g. to tf.keras.Model.fit (as fit_generator is deprecated), is it the same internally as using tf.data.Dataset.from_generator?

If so, would that be a recommended way to switch over to tf.data without having to rewrite custom data processing. If not, what would be the recommended way?

I too think the documentation is very confusing at that point...