DataCollatorForWholeWordMask does not return attention_mask

[Aktsvigun]

Hi,

** DataCollatorForWholeWordMask** does not output attention_mask. According to the __call__ method:
return {"input_ids": inputs, "labels": labels}.

Is there a peculiar motivation behind it or is a small bug? From where I see, when we do the pre-training, most instances will not be of the same length and applying Attention for all the tokens (including the padding) may cause imprecise results.

[NielsRogge]

Not sure where you are seeing this. DataCollatorWithPadding can also pad attention masks, as it applies tokenizer.pad() on the inputs as can be seen here.

[Aktsvigun]

@NielsRogge am sorry for confusing, meant DataCollatorForWholeWordMask, will correct that now

[Aktsvigun]

There is also a bug with pad_to_multiple_of argument: it is not passed to _collate_batch inside __call__. Have made a pull request #12999 to add its usage.

[sgugger]

That class should never have been merged as it is, it was a mistake on our side. It contains multiple bugs and only works for BERT models. It need to be rewritten from scratch to be model agnostic.

[Aktsvigun]

For me it works perfectly for any arbitrary model (checked it is doing everything correct), except for attention_mask generation and pad_to_multiple_of usage (both can be corrected manually though, however, it is always better not to invent a bicycle).

[sgugger]

It cant' give good results on a tokenizer that is not like BERT, since it relies on the "##" to detect if something is inside a word.

Is there any progress on this issue? It seems that this issue still exists.

[Ivan3ol]

Still exists

[rjgleaton]

Would it be possible to get this issue reopened? I working on addressing it in a current fork.