Add doctests to Perceiver examples

[stevenmanton]

What does this PR do?

Related to #16292

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Taken from #16292 : @patrickvonplaten @ydshieh @patil-suraj

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ydshieh]

Hi @stevenmanton

Thank you a lot for the PR! Currently the test will fail for the following

FAILED src/transformers/models/perceiver/modeling_perceiver.py::transformers.models.perceiver.modeling_perceiver.PerceiverForImageClassificationConvProcessing.forward
FAILED src/transformers/models/perceiver/modeling_perceiver.py::transformers.models.perceiver.modeling_perceiver.PerceiverForImageClassificationFourier.forward
FAILED src/transformers/models/perceiver/modeling_perceiver.py::transformers.models.perceiver.modeling_perceiver.PerceiverForImageClassificationLearned.forward

We should add the expected outputs in the doc examples.
For other doc examples in this model, the tests pass, but they don't have outputs to test.
For example, PerceiverForOpticalFlow has

>>> logits = outputs.logits

without output and the expected value. We can have something like

>>> list(logits .shape)
expected shapes

Would you like to fix and enhance those doc examples?

Once you have a change (staged or commited), you can run the test like

python utils/prepare_for_doc_test.py src/transformers/models/perceiver/modeling_perceiver.py
pytest --doctest-modules src/transformers/models/perceiver/modeling_perceiver.py -sv --doctest-continue-on-failure

Once the test is run, you can clean up the git status before further changes or push.

Thanks!

[stevenmanton]

@ydshieh thanks for the quick feedback! Yes, I noticed those tests were failing, but I thought it might be something about my local environment and the extra newline stuff. I just pushed a fix, which passes locally.

By the way, how did you know those tests were failing? The CI pipelines all seemed to be passing. Did you have to checkout my branch and run it locally?

[ydshieh]

@stevenmanton Thanks for the push. However, instead of changing to

>>> predicted_class = model.config.id2label[predicted_class_idx]

we should change it to

>>> print("Predicted class:", model.config.id2label[predicted_class_idx])
add some output here - so the doctest will test again it

You can find similar work is done

transformers/src/transformers/models/deit/modeling_deit.py

Lines 735 to 736 in d5848a5

	>>> print("Predicted class:", model.config.id2label[predicted_class_idx])
	Predicted class: maillot

Otherwise, the example has nothing to be tested.

By the way, how did you know those tests were failing? The CI pipelines all seemed to be passing. Did you have to checkout my branch and run it locally?

Yes. The PR CI on CircleCI does not run doctest :-). It is run after the PR being merged.

[stevenmanton]

@ydshieh Thanks for your feedback and patience. I believe I've corrected it. The extra newline stuff is confusing, but if you run prepare_for_doc_test.py (which I think just adds a newline to all the docstrings) on the last commit, all tests pass for me locally.

[ydshieh]

Thanks! As we are going to run the doctest for this model, would you like to add some expected outputs at the following places?

transformers/src/transformers/models/perceiver/modeling_perceiver.py

Line 1921 in bebcb95

>>> logits = outputs.logits

transformers/src/transformers/models/perceiver/modeling_perceiver.py

Line 1695 in bebcb95

>>> logits = outputs.logits

transformers/src/transformers/models/perceiver/modeling_perceiver.py

Line 1131 in bebcb95

>>> logits = outputs.logits

transformers/src/transformers/models/perceiver/modeling_perceiver.py

Line 1033 in bebcb95

>>> logits = outputs.logits

transformers/src/transformers/models/perceiver/modeling_perceiver.py

Line 1021 in bebcb95

>>> logits = outputs.logits

And a few places in this doc example (for logits and loss)

transformers/src/transformers/models/perceiver/modeling_perceiver.py

Line 1695 in bebcb95

>>> logits = outputs.logits

For logits, it would look like to add (the values should be collected from the run)

        >>> list(logits.shape)
        [1, 196, 8192]

[ydshieh]

When running prepare_for_doc_test.py, it will add some empty lines - to make doctest pass. That is why we should stage our change, run that script, run doctest, and discard the change before commit or further changes :-)

[stevenmanton]

@ydshieh Ok, I added some more checks for the sizes of logits. They all pass for me locally.

[ydshieh]

Hi @stevenmanton Thanks a lot of this PR and the follow-up effort :-). It's great, and I just added an extra expected loss value for PerceiverForMaskedLM.

@sgugger As I am not sure if this PR is classified as a PR that I can merge myself, tagging you for a review or let me know if I can merge the doctest PR in the future. Thanks.

[sgugger]

Thanks a lot for this PR!

@ydshieh Always good to have another pair of eyes :-)
Don't hesitate to merge such PRs without my approval in the future if I take too long to review, but I'd appreciate the opportunity to review!