LayoutLMv2Processor: ensure 1-to-1 mapping between images and samples in case of overflowing tokens

[garyhlai]

What does this PR do?

Fixes #13554

Problem re-summarized: when return_offsets_mapping is set to True, LayoutLMv2Processor would break up sequences that are too long into multiple input_ids sequences, causing a mismatch between input_ids (longer in length in the case of overflowing tokens) and images.

This fix would ensure the 1-to-1 mapping between the images and input_ids.

Reproducible Example: (The assertion at the end would fail without the fix, pass with the fix)

import transformers
from PIL import Image
from transformers import LayoutLMv2Processor
from datasets import Features, Sequence, ClassLabel, Value, Array2D, Array3D, load_dataset
import torch

datasets = load_dataset("nielsr/funsd")
labels = datasets['train'].features['ner_tags'].feature.names
id2label = {v: k for v, k in enumerate(labels)}
label2id = {k: v for v, k in enumerate(labels)}

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")

def preprocess_data(examples):
  images = [Image.open(path).convert("RGB") for path in examples['image_path']]
  words = examples['words']
  boxes = examples['bboxes']
  word_labels = examples['ner_tags']
  encoded_inputs = processor(images, words, boxes=boxes, word_labels=word_labels,
                             padding="max_length", truncation=True,
                             return_overflowing_tokens=True,
                             stride=50,
                             return_offsets_mapping=True,
                             return_tensors="pt")
  return encoded_inputs

train_data = preprocess_data(datasets["train"])

# this assert would fail without this PR fix. 
assert len(train_data["image"]) == len(train_data["input_ids"])

Required Input from Reviewers

Right now, the LayoutLMv2Processor would return a list for encoded_inputs["image"], regardless of the value of return_tensors. If we want it to return a torch tensor in the case return_tensors=="pt", we have to torch.stack the list (and do similar thing to support "np" and "tf").

Should I implement this in get_overflowing_images? Or should I just leave the return type as list and just print a warning?

Who can review?

@NielsRogge @sgugger @LysandreJik

P.S.

The test_processor_case_1 in test_processor_layoutlmv2.py fails before this PR. I'd be happy to look at it as well but it's unrelated to this PR.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

LGTM but let's wait for @NielsRogge to have a look too!

[sgugger]

No new assert in the code base, please use a test and raise a ValueError here.

[garyhlai]

Just added a test and changed the assert to ValueError (running make style led to quite a few formatting changes in test_processor_layoutlmv2.py for some reason).

The reason for no assert is because assert is mostly used only for debugging purpose (detecting programmer error), which can get muted during production, right?

[sgugger]

Yes, exactly!

[NielsRogge]

LGTM, thanks for improving!

[sgugger]

Are you sure you have the exact same version as black as is pinned in our setup? The CI check for style is passing on master, so none of the reformatting unrelated to the changes in your PR is necessary.
The easiest might be to revert your last commit once you have made sure of the version of black, as black doesn't undo the lines it adds.

[sgugger]

Can you split that message on several lines to respect the 119 char limit?

[garyhlai]

Done. Is there a reason that the CI can't catch cases where strings exceed the 119 char limit right now? (I know black doesn't enforce line-length limit on strings by default)

Fixed the formatting issue too (turns out that my VSCode uses a different version of black to autoformat on save, which running make style with the HF pinned black version afterwards can't undo).

[sgugger]

Thanks again for your contribution!

[timothyjlaurent]

Thanks for handling this @ghlai9665