Added missing code in exemplary notebook - custom datasets fine-tuning

[Pawloch247]

What does this PR do?

Added missing code in tokenize_and_align_labels function in the exemplary notebook on custom datasets - token classification.
The missing code concerns adding labels for all but the first token in a single word.
The added code was taken directly from huggingface official example - this colab notebook.

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

• maintained examples (not research project or legacy): @sgugger, @patil-suraj

[HuggingFaceDocBuilder]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

I don't understand what you mean, the notebook has the same code as the example: they are automatically synced at each merge in the Transformers repo.

[Pawloch247]

You're right. I pasted the wrong link so the comparison made no sense. The link should have been to the colab notebook:
https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb

In the colab notebook tokenize_and_align_labels has the else clause which is missing in the notebook in Github and hence is missing here: https://huggingface.co/docs/transformers/custom_datasets#token-classification-with-wnut-emerging-entities

[sgugger]

Those are two different tutorials, it's normal they have different code. The one in the main documentation is left as simple as possible on purpose.

[Pawloch247]

In the main documentation, there is a mention about "Only labeling the first token of a given word. Assign -100 to the other subtokens from the same word.". However, it is not the case with the code below, the tokenize_and_align_labels function, where not only are the other subtokens assigned with the true labels but also previous_word_idx is not being updated. Such contradiction was ambiguous for me. Only after having dug deeper (the colab notebook), I understood that this part of code was missing from the official documentation. I do not think that these few lines were omitted on purpose. If you don't think it makes such a difference, close this pr (or let me know if I should be the one to close it).

[sgugger]

Aaaah, now I understand. Thank you for taking the time to explain the problem. I left a couple of comments to adjust the PR accordingly, so the code matches the text.

[sgugger]

No need for this to keep things simple.

[sgugger]

Suggested change

	label_ids.append(label[word_idx] if label_all_tokens else -100)
	label_ids.append(-100)

[Pawloch247]

I adjusted your comments. I guess I could have been more precise from the beginning. Thanks.

[sgugger]

Thanks for amending your PR!