Inference for TFMarianMTModel (en to Romance language translation) is slow and inaccurate

[danielenricocahall]

System Info

System
macOS Monterey 12.2.1

transformers==4.20.1
tensorflow==2.9.1

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import TFMarianMTModel, MarianTokenizer
model_name = "Helsinki-NLP/opus-mt-en-ROMANCE"
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = TFMarianMTModel.from_pretrained(model_name)
text_in = ['>>fr<< hello']
batch = tokenizer(text_in, return_tensors='tf', padding=True)
translated = model.generate(**batch) 

Output:

- Qu'est-ce qu'il y a, là-bas, là-bas, là---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Expected behavior

I would expect similar performance to the PyTorch model.

Inference requires about 120s on my machine and outputs an incorrect translation. In contrast, the PyTorch model (replacing TFMarianMTModel with MarianMTModel and changing return_tensors to 'pt' in the code snippet) returns the correct translation ("Bonjour") and inference requires about 6s on my machine.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[LysandreJik]

Sorry for missing this! Could you take a look at this, @gante, @Rocketknight1, @ydshieh?

[ydshieh]

Let me take a look on the quality issue. And possibly @gante or @Rocketknight1 for the speed issue, let's discuss it :-)

[ydshieh]

Actually, the performance issue comes from the quality issue. The TF version didn't stop the generation until 512 tokens.

[[65000    25  2092     7   179    15   276   185     7   227    32     9
      2  2538    15  5716     2  2538    15  5716     2  2538    15    15
     15    15    15    15    15    15    15    15    15    15    15    15 ............
     0]], shape=(1, 512), dtype=int32)

[ydshieh]

I believe the current PT / TF checkpoints for "Helsinki-NLP/opus-mt-en-ROMANCE" doesn't contain the same weight.
As if I change from

model = TFMarianMTModel.from_pretrained(model_name)

to

model = TFMarianMTModel.from_pretrained(model_name, from_pt=True)

I could get

[[65000 21959     3     0 65000 65000 65 ....] (still `512` tokens)

while the PyTorch version gives

tensor([[65000, 21959,     3,     0]])

So:

• we probably need to check which checkpoint is the correct one, and uploaded the new checkpoint
• investigate why TFMarianMTModel doesn't stop earlier.

[ydshieh]

After a double check (see code below, where I use from_pt=True), I believe the current PT checkpoint is the correct one, but not the TF checkpoint.

@gante Would you like to have a look too, upload a new TF checkpoint, and see why TFMarianMTModel doesn't stop the generation earlier as MarianMTModel does?

from transformers import MarianMTModel, MarianTokenizer, TFMarianMTModel

model_name = "Helsinki-NLP/opus-mt-en-ROMANCE"
tokenizer = MarianTokenizer.from_pretrained(model_name)

# text_in = ['>>fr<< hello']
text_in = ['>>fr<< Hello, I am a student.']

model = MarianMTModel.from_pretrained(model_name)

batch = tokenizer(text_in, return_tensors='pt', padding=True)
translated = model.generate(**batch)
o = tokenizer.batch_decode(translated, skip_special_tokens=True)

print(translated)
print(o)

model = TFMarianMTModel.from_pretrained(model_name, from_pt=True)

batch = tokenizer(text_in, return_tensors='tf', padding=True)
translated = model.generate(**batch)
o = tokenizer.batch_decode(translated, skip_special_tokens=True)

print(translated)
print(o)


text_in = ['>>it<< I love dogs and cats.']


model = MarianMTModel.from_pretrained(model_name)

batch = tokenizer(text_in, return_tensors='pt', padding=True)
translated = model.generate(**batch)
o = tokenizer.batch_decode(translated, skip_special_tokens=True)

print(translated)
print(o)

model = TFMarianMTModel.from_pretrained(model_name, from_pt=True)

batch = tokenizer(text_in, return_tensors='tf', padding=True)
translated = model.generate(**batch)
o = tokenizer.batch_decode(translated, skip_special_tokens=True)

print(translated)
print(o)

[gante]

Hi there @ydshieh @danielenricocahall 👋

None of the Marian models can be successfully converted to TF -- they all fail when validating the hidden layers and outputs of the models. This is a shame since there are a ton of Marian models for translation :(

This means there is something wrong with either the model architecture or with weight cross-loading. I haven't looked into it, other than noticing the issue when attempting to convert the weights from Helsinki-NLP

[danielenricocahall]

Thank you for looking into it @ydshieh and @gante !!! This is great information.

[gante]

@danielenricocahall a fix was merged and new weights were pushed -- if you run from main, the translations should be much better now 🙌

[ydshieh]

cc @gante

We still have the generation issue

from transformers import MarianMTModel, MarianTokenizer, TFMarianMTModel

model_name = "Helsinki-NLP/opus-mt-en-ROMANCE"
tokenizer = MarianTokenizer.from_pretrained(model_name)
text_in = ['>>fr<< hello']

# PT generates a few tokens then stops early -> very fast 
model = MarianMTModel.from_pretrained(model_name)
batch = tokenizer(text_in, return_tensors='pt', padding=True)
translated = model.generate(**batch)
o = tokenizer.batch_decode(translated, skip_special_tokens=True)

print(translated)
print(o)

# TF generates 512 tokens, although the decoded version gives the same result as PT -> very slow
model = TFMarianMTModel.from_pretrained(model_name, from_pt=False)
batch = tokenizer(text_in, return_tensors='tf', padding=True)
translated = model.generate(**batch)
o = tokenizer.batch_decode(translated, skip_special_tokens=True)

print(translated)
print(o)

[jamie0725]

@ydshieh Hi, I am experiencing the same issue. Expected the TF version would be faster than the PT version.