About keras mask layers

[xiaohua6689]

Body

model buiding

a = Input(shape=[5])
b = Input(shape=[10])
emb_a = Embedding(8, 5, mask_zero=True)(a)
emb_b = Embedding(20, 5, mask_zero=True)(b)
cat = Concatenate(axis=1)([emb_a, emb_b])
lstm = LSTM(16)(cat)
dense = Dense(1)(lstm)
model = Model(inputs=[a, b], outputs=[dense])

#Error information
ValueError: Exception encountered when calling BroadcastTo.call().

Attempt to convert a value (None) with an unsupported type (<class 'NoneType'>) to a Tensor.

Arguments received by BroadcastTo.call():
• x=tf.Tensor(shape=(1, 5, 1), dtype=bool)

Guidelines

• I have read and understood this category's guidelines before making this post.

[ravindudil5han]

The error you’re seeing, ValueError: Exception encountered when calling BroadcastTo.call(), usually happens because there’s a mismatch in shapes or an issue with how Keras layers handle masking. The key issue is in using mask_zero=True in the Embedding layer. This mask_zero=True argument is supposed to mask out padding tokens, but it sometimes doesn't work well when passed through certain layers like Concatenate and LSTM.

Here’s the solution and reasoning to address this problem clearly:

Solution

1. Remove mask_zero=True from the Embedding layers.
   Masking with mask_zero=True in Embedding layers can lead to issues when concatenating embeddings or passing them to certain layers that don't support masking directly. In this case, you can apply masking separately.

2. Add an explicit Masking layer after concatenating.
   After concatenating the embeddings, you can add a Masking layer if masking is still necessary, or handle masking in a different way.

Here’s how the corrected model code would look:

from tensorflow.keras.layers import Input, Embedding, Concatenate, LSTM, Dense, Masking
from tensorflow.keras.models import Model

# Define inputs
a = Input(shape=(5,))
b = Input(shape=(10,))

# Embedding layers without mask_zero=True
emb_a = Embedding(8, 5)(a)
emb_b = Embedding(20, 5)(b)

# Concatenate embeddings along the time-step axis
cat = Concatenate(axis=1)([emb_a, emb_b])

# Optional: Add a Masking layer to handle masking after concatenation
masked_cat = Masking()(cat)

# LSTM and Dense layers
lstm = LSTM(16)(masked_cat)
dense = Dense(1)(lstm)

# Model
model = Model(inputs=[a, b], outputs=[dense])

Explanation

1. Embedding Layers Without mask_zero=True: Removing mask_zero=True prevents the BroadcastTo error by avoiding the unsupported mask propagation.

2. Adding a Masking Layer (if necessary): By adding an explicit Masking layer, you manually control which time steps are ignored during LSTM processing without relying on mask_zero=True in the Embedding layer.

Additional Tips

If your model requires masking, another way is to add a preprocessing step that identifies padding tokens and masks them separately.

[xiaohua6689]

when i back my python to version 3.8, and tensorflow to 2.4，the code run well. so i think this issue maybe caused by version updating.