TF models save_pretrained() failed when saved_model=True

[ydshieh]

Environment info

• transformers version: 4.13.0.dev0
• Platform: Windows-10-10.0.19042-SP0
• Python version: 3.9.5
• PyTorch version (GPU?): 1.9.0+cpu (False)
• Tensorflow version (GPU?): 2.6.0 (False)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: None
• Using distributed or parallel set-up in script?: None

Who can help

TensorFlow: @Rocketknight1

To reproduce

from transformers import TFBertModel
import tensorflow as tf
from PIL import Image
import requests

model = TFBertModel.from_pretrained("bert-base-uncased")
model.save_pretrained("tmp", saved_model=True)

# this also failed
for x in model.config.items():
    print(x)

Error messages:

Traceback (most recent call last):
  File "C:\Users\33611\Desktop\Projects\transformers-dev-2\transformers\del.py", line 7, in <module>
    model.save_pretrained("tmp", saved_model=True)
  File "C:\Users\33611\Desktop\Projects\transformers-dev-2\transformers\src\transformers\modeling_tf_utils.py", line 1227, in save_pretrained
    self.save(saved_model_dir, include_optimizer=False, signatures=self.serving)
  File "C:\Users\33611\miniconda3\envs\py39\lib\site-packages\keras\engine\training.py", line 2145, in save
    save.save_model(self, filepath, overwrite, include_optimizer, save_format,
  File "C:\Users\33611\miniconda3\envs\py39\lib\site-packages\keras\saving\save.py", line 149, in save_model
    saved_model_save.save(model, filepath, overwrite, include_optimizer,
  File "C:\Users\33611\miniconda3\envs\py39\lib\site-packages\keras\saving\saved_model\save.py", line 94, in save
    metadata = generate_keras_metadata(saved_nodes, node_paths)
  File "C:\Users\33611\miniconda3\envs\py39\lib\site-packages\keras\saving\saved_model\save.py", line 123, in generate_keras_metadata
    metadata=node._tracking_metadata)  # pylint: disable=protected-access
  File "C:\Users\33611\miniconda3\envs\py39\lib\site-packages\keras\engine\base_layer.py", line 3078, in _tracking_metadata
    return self._trackable_saved_model_saver.tracking_metadata
  File "C:\Users\33611\miniconda3\envs\py39\lib\site-packages\keras\saving\saved_model\base_serialization.py", line 54, in tracking_metadata
    return json_utils.Encoder().encode(self.python_properties)
  File "C:\Users\33611\miniconda3\envs\py39\lib\site-packages\keras\saving\saved_model\layer_serialization.py", line 37, in python_properties
    return self._python_properties_internal()
  File "C:\Users\33611\miniconda3\envs\py39\lib\site-packages\keras\saving\saved_model\model_serialization.py", line 31, in _python_properties_internal
    metadata = super(ModelSavedModelSaver, self)._python_properties_internal()
  File "C:\Users\33611\miniconda3\envs\py39\lib\site-packages\keras\saving\saved_model\layer_serialization.py", line 54, in _python_properties_internal
    metadata.update(get_serialized(self.obj))
  File "C:\Users\33611\miniconda3\envs\py39\lib\site-packages\keras\saving\saved_model\layer_serialization.py", line 113, in get_serialized
    return generic_utils.serialize_keras_object(obj)
  File "C:\Users\33611\miniconda3\envs\py39\lib\site-packages\keras\utils\generic_utils.py", line 510, in serialize_keras_object
    for key, item in config.items():
  File "C:\Users\33611\Desktop\Projects\transformers-dev-2\transformers\src\transformers\configuration_utils.py", line 237, in __getattribute__
    return super().__getattribute__(key)
AttributeError: 'BertConfig' object has no attribute 'items'

Expected behavior

model.save_pretrained(..., saved_model=True) should work, because it is used in test_saved_model_creation_extended() in test_modeling_tf_common.py.

[LysandreJik]

This should have been caught by these tests: https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_common.py#L239-L306

But these tests are deactivated as they're too slow, unfortunartely.

[shabie]

Why are they there then? :D

[LysandreJik]

Because they were not deactivated at some point, but ended up taking 6+ hours as the number of architectures grew, which we can't do.

We need to refactor them to be simpler and faster. Want to take a stab at it? :)

[shabie]

If I won't ultimately be stabbing myself, yes!

[LysandreJik]

You can start by running the tests above for a single model, and see if they can't be refactored in a single test, or if some of the time spent cannot be reduced. Also pinging @Rocketknight1 as it might be of interest to him!

[shabie]

You can start by running the tests above for a single model, and see if they can't be refactored in a single test, or if some of the time spent cannot be reduced. Also pinging @Rocketknight1 as it might be of interest to him!

Alright. Thanks for the tip!

[LysandreJik]

And thank you for offering to help :)

[Rocketknight1]

I feel like this definitely worked in the past. I confirmed that no models are saving correctly with saved_model=True, and the problem is occurring when we call model.save() in the save_pretrained() function. Calling model.save() alone also causes this bug. I believe the underlying issue is that Keras is attempting to serialize all of the Model object's attributes, and doesn't know what to do with a BertConfig object.

[Rocketknight1]

Since this worked before (presumably that test passed at -some- point!), most likely something has changed regarding TensorFlow's save logic in 2.6, which means we might have to reassess how we use decorators like @keras_serializable. I'll put this on the list to investigate, but if anyone else has any insight, let me know!

[Rocketknight1]

Alternatively, this may be caused by my PR here, which made changes to the saving/loading of TF models. I'll try a version of Transformers before that to see if the issue is still there.

EDIT: Still happens before my PR, so that's not the problem.

[Rocketknight1]

Update: Bug still occurs in TF 2.5.

[Rocketknight1]

@shabie Just to let you know, we refactored a lot of those tests quite urgently when we realized that lack of coverage was causing serious problems! This issue should hopefully be resolved now, but if people encounter further difficulties with saving TF models, please comment or file a new issue.