'make_layer_stack' doesn't have a parameter named 'use_universal_transformer'.

[danyaljj]

Getting this error while exporting models:

export PROJECT=...
export ZONE=....
export MODEL_DIR=gs://...
t5_mesh_transformer \
  --gcp_project="${PROJECT}" \
  --tpu_zone="${ZONE}" \
  --model_dir="${MODEL_DIR}" \
  --checkpoint_steps=1100500 \
  --use_model_api \
  --mode="export" \
  --export_dir="${MODEL_DIR}/export" \
  --tpu="danielk-tpu-europe-west4-a-v3-8-no5"

WARNING:tensorflow:From /home/danielk/ENTER/lib/python3.7/site-packages/tensorflow_core/python/compat/v2_compat.py:68: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
I0420 19:29:35.249601 140589403665792 mesh_transformer_main.py:141] No write access to model directory. Skipping command logging.
Traceback (most recent call last):
  File "/home/danielk/ENTER/bin/t5_mesh_transformer", line 8, in <module>
    sys.exit(console_entry_point())
  File "/home/danielk/ENTER/lib/python3.7/site-packages/t5/models/mesh_transformer_main.py", line 222, in console_entry_point
    app.run(main)
  File "/home/danielk/ENTER/lib/python3.7/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/danielk/ENTER/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/home/danielk/ENTER/lib/python3.7/site-packages/t5/models/mesh_transformer_main.py", line 200, in main
    checkpoint_step=checkpoint_steps)
  File "/home/danielk/ENTER/lib/python3.7/site-packages/t5/models/mtf_model.py", line 360, in export
    gin.parse_config_file(_operative_config_path(self._model_dir))
  File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/config.py", line 1702, in parse_config_file
    parse_config(f, skip_unknown=skip_unknown)
  File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/config.py", line 1611, in parse_config
    bind_parameter((scope, selector, arg_name), value)
  File "/home/danielk/ENTER/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/utils.py", line 68, in try_with_location
    augment_exception_message_and_reraise(exception, _format_location(location))
  File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise
    six.raise_from(proxy.with_traceback(exception.__traceback__), None)
  File "<string>", line 3, in raise_from
  File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/utils.py", line 66, in try_with_location
    yield
  File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/config.py", line 1611, in parse_config
    bind_parameter((scope, selector, arg_name), value)
  File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/config.py", line 648, in bind_parameter
    pbk = ParsedBindingKey(binding_key)
  File "/home/danielk/ENTER/lib/python3.7/site-packages/gin/config.py", line 517, in __new__
    raise ValueError(err_str.format(selector, arg_name))
ValueError: Configurable 'make_layer_stack' doesn't have a parameter named 'use_universal_transformer'.
  In file "gs://danielk-files/t5-models/union_mixture/3B/operative_config.gin", line 83
    decoder/make_layer_stack.use_universal_transformer = False

Versions for completeness:

t5                                 0.5.0           
tensorboard                        1.15.0               
tensorflow                         1.15.2               
tensorflow-datasets                3.0.0                
tensorflow-estimator               1.15.1               
tensorflow-metadata                0.21.2               
tensorflow-text                    1.15.0     
torch                              1.4.0  

I used the same command to export models a couple of weeks ago. So I suspect a recent change could be causing this error.

[danyaljj]

Update: I think using a slightly older version of tensorflow-mesh (https://pypi.org/project/mesh-tensorflow/0.1.12/) the error goes away.

[craffel]

Hi Daniel, yes, unfortunately there was a change introduced in the Mesh TF Transformer that will break any operative config saved during a short period of time. It should be fine to just remove that line from the offending operative config and use the latest MTF.