Add gfile api shim to remove tensorflow dependency for basic IO. #2586
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
chiamp
force-pushed
the
flax_io
branch
3 times, most recently
from
3dfa654 to
e05ded2
Compare
| io_mode = BackendMode.TF | ||
| else: | ||
| logging.warning("Tensorflow library not found, tensorflow.io.gfile " | ||
| "operations will use slower native shim calls.") |
There was a problem hiding this comment.
Are native operations really slower than tensorflow?
Maybe worth pointing out that any access to GCS (i.e. "gs://" paths) will fail. Being explicit about this failure mode might also be a good idea inside below functions (e.g. when a program runs fine in one environment, but then crashes in another environment, it might not be obvious to the user that the 2nd invocation has crashed because it was trying to access a "gs://" path in an environment that did not have Tensorflow installed).
There was a problem hiding this comment.
I'm unsure on how the speed of native operations compare to tensorflow; maybe @levskaya can jump in.
As for GCS, yes good point, I'll add a disclaimer in the warning message.
There was a problem hiding this comment.
Created a new issue #2624
This fixes up a small mistake in google#2586, where tensorflow is still being imported from `flax.training.checkpoints`. To avoid the tensorflow dependency, an alias `flax.io.NotFoundError` is added which can be either `tf_errors.NotFoundError` or `FileNotFoundError` depending on the backend mode.
Taking over #2073
Added a shim for tensorflow.io.gfile with io.py to remove tensorflow dependency.