LayoutXLMProcessor: Enforce using "return_overflowing_tokens" with "return_offsets_mapping"

[anthony2261]

What does this PR do?

Fixes #18726

Code to reproduce the error: https://colab.research.google.com/drive/1ETpz8UP42r7HjRg4VUkC7L8ou10qY3bQ?usp=sharing

Specific combination that caused the error: LayoutXLMProcessor with use_fast=False, return_overflowing_tokens=True and return_offsets_mapping=False

Who can review?

@NielsRogge

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[NielsRogge]

Thanks, although it doesn't include the same changes as #17092, could you add those?

[anthony2261]

Thanks, although it doesn't include the same changes as #17092, could you add those?

The additional changes in the other PR were already present, I added the missing ones, see the pictures below
This PR:

The other PR:

[NielsRogge]

Oh yeah, just realized I added that myself when adding LayoutLMv3. Thanks! I'll add another reviewer before merging

[anthony2261]

Btw, one issue I have with this is that if we're using the processor with use_fast=False, we won't be allowed to process data with return_overflowing_tokens=True, because this will force us to set return_offsets_mapping=True, and doing that will raise this error:

NotImplementedError: return_offset_mapping is not available when using Python tokenizers.
To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast.
More information on available tokenizers at https://github.com/huggingface/transformers/pull/2674

I was able to get 1-to-1 token mappings with the non-fast tokenizer but in a very hacky way. If I find a better way, I'll suggest doing that to support return_overflowing_tokens with regular tokenizers.

N.B: This is the same case with LMv2, not sure about LMv3

[LysandreJik]

Yes, this looks good to me! Thank you, @anthony2261!