Loading EfficientNet-Lite4 or EfficientNet-Lite2 Feature Vectors as KerasLayer() fails from tfhub.dev

[salernoa]

Problem Description

Both EfficientNet-Lite4 and EfficientNet-Lite2 feature-vector modules fail to download correctly when attempting to create a feature layer using tensorflow_hub.KerasLayer() with the tfhub.dev URLs.

The temporary directory is created, but none of the files are downloaded/saved.

Error only occurs for EfficientNet-Lite2 and EfficientNet-Lite4 versions. Additional testing indicates that EfficientNet-Lite0, EfficientNet-Lite1, and EfficientNet-Lite3 work properly.

Issue first logged 2020-05-03.
Was previously working fine as of 2020-04-30

System Info:

OS: Windows 10
Python: 3.7.7
Tensorflow: 2.1.0
Tensorflow-Hub: 0.8.0

Code Snippet:

import tensorflow_hub as hub
import tensorflow as tf

# EfficientNet-Lite4
MODULE_HANDLE = "https://tfhub.dev/tensorflow/efficientnet/lite4/feature-vector/2"
EN4_feature_layer = hub.KerasLayer(MODULE_HANDLE)

# EfficientNet-Lite2
MODULE_HANDLE = "https://tfhub.dev/tensorflow/efficientnet/lite2/feature-vector/2"
EN2_feature_layer = hub.KerasLayer(MODULE_HANDLE)

Error

OSError: SavedModel file does not exist at: C:\Users\salernoa\AppData\Local\Temp\tfhub_modules\a3b91f2744273e376e489b1c60d6bddddebdc084/{saved_model.pbtxt|saved_model.pb}

Directory exists but is empty.

Current Workaround

Manually download and extract the feature vector modules for EfficientNet-Lite4 or EfficientNet-Lite2 from tfhub.dev, then create the KerasLayer using the local path instead of the URL.

[salernoa]

This also affects anyone attempting to use those EfficientNet-Lite models with TFLite Model Maker from tensorflow-examples

[gowthamkpr]

I have ran the same example on ubuntu 18.04 and I am not facing any issues. I think this issue is specific to windows.

[salernoa]

As of yesterday, the example runs successfully on Windows. I didn't change anything on my system, so I'm not sure what the issue was, but I'm glad it works again. Thank you for looking into it.

[djdongjin]

@gowthamkpr Hey I do think there might be some issues here. I ran the same issue on Ubuntu. I used mobilenet_v2 layer, and succeeded. But when later I ran the same command, it failed and the issue seems like TF-Hub cannot find the cached model. The reason I guess is because the cached model is deleted as it is stored in /tmp folders, but TF-Hub cannot detect this deletion.

I am using the make_image_classifier tool, but the problem shouldn't relevant to that, as the error happens when loading hub layers.

The command I used is

python make_image_classifier.py --image_dir /home/ml/users/jdong25/data/cifar10/cifar10/train --summaries_dir $HOME/summary --saved_model_dir $HOME/model --strategy default --cache true

I am making some changes to this tool so I build TF-Hub from source. But it shouldn't be relevant to the error as well, since I ran the same command multiple times and all succeeded.

The error message is attached:

2020-05-29 00:47:00.149271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0, 1
I0529 00:47:00.154991 140169536722688 resolver.py:79] Using /tmp/tfhub_modules to cache modules.
Traceback (most recent call last):
  File "make_image_classifier.py", line 307, in <module>
    run_main()
  File "make_image_classifier.py", line 303, in run_main
    app.run(main)
  File "/home/ml/users/jdong25/.conda/envs/tf/lib/python3.7/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/ml/users/jdong25/.conda/envs/tf/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "make_image_classifier.py", line 260, in main
    FLAGS.summaries_dir)
  File "/home/ml/users/jdong25/.conda/envs/tf/lib/python3.7/site-packages/tensorflow_hub/tools/make_image_classifier/make_image_classifier_lib.py", line 366, in make_image_classifier
    module_layer = hub.KerasLayer(tfhub_module, trainable=hparams.do_fine_tuning)
  File "/home/ml/users/jdong25/.conda/envs/tf/lib/python3.7/site-packages/tensorflow_hub/keras_layer.py", line 146, in __init__
    self._func = load_module(handle, tags)
  File "/home/ml/users/jdong25/.conda/envs/tf/lib/python3.7/site-packages/tensorflow_hub/keras_layer.py", line 398, in load_module
    return module_v2.load(handle, tags=tags)
  File "/home/ml/users/jdong25/.conda/envs/tf/lib/python3.7/site-packages/tensorflow_hub/module_v2.py", line 102, in load
    obj = tf_v1.saved_model.load_v2(module_path, tags=tags)
  File "/home/ml/users/jdong25/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 581, in load
    return load_internal(export_dir, tags)
  File "/home/ml/users/jdong25/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 591, in load_internal
    loader_impl.parse_saved_model_with_debug_info(export_dir))
  File "/home/ml/users/jdong25/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/saved_model/loader_impl.py", line 56, in parse_saved_model_with_debug_info
    saved_model = _parse_saved_model(export_dir)
  File "/home/ml/users/jdong25/.conda/envs/tf/lib/python3.7/site-packages/tensorflow/python/saved_model/loader_impl.py", line 113, in parse_saved_model
    constants.SAVED_MODEL_FILENAME_PB))
OSError: SavedModel file does not exist at: /tmp/tfhub_modules/145bb06ec3b59b08fb564ab752bd5aa222bfb50a/{saved_model.pbtxt|saved_model.pb}

And I checked the path /tmp/tfhub_modules/145bb06ec3b59b08fb564ab752bd5aa222bfb50a. The folder is still there but the model files are deleted.

[djdongjin]

One more followup. After deleting the folder, the command is runnable. So I guess the reason is that, when downloading a model, TF-Hub will generate a hash code for different models. But the hash code for each model is the same, such that TF Hub can detect that the model has been downloaded.

The issue is that sometimes, especially when the model is cached in a /tmp folder, the cached folder might be still there but the model files are deleted.

So an optimization might be: TF-Hub first check if the model exists using the hashcode, and if the hashcode folder exists, then it check if the model files exists. If so, then it can load the cached model; if not, it should re-download the model instead of raising errors.

[djdongjin]

Okay, I guess I target the issue. I think the bug is in this line.

When loading a layer with a TF-Hub link. The url string is mapped to a directory. But the resolver only checks if the directory exists but not check if model files exist. The situation is quite possible, as you don't know when the model files are deleted and when the folder is deleted.

I think we need to make sure both exist and only then we can simply return the module_dir. I will correct this if you think it's reasonable. @gowthamkpr