add targets arg to fill-mask pipeline

[joeddav]

Proposal to add a targets arg when calling FillMaskPipeline, allowing a user to compare different target tokens in addition to getting the top k predictions. This could be useful in a number of areas, such as in probing model behavior:

nlp = pipeline('fill-mask', topk=2)
nlp("<mask> should be at home and take care of their children.")
> [{'sequence': '<s>Parents should be at home and take care of their children.</s>',
    'score': 0.32749035954475403,
    'token': 35835,
    'token_str': 'Parents'},
   {'sequence': '<s> parents should be at home and take care of their children.</s>',
    'score': 0.12840420007705688,
    'token': 1041,
    'token_str': 'Ġparents'}]

nlp("<mask> should be at home and take care of their children.", targets=["Men", "Women"])
> [{'sequence': '<s>Women should be at home and take care of their children.</s>',
    'score': 0.04439367726445198,
    'token': 19814,
    'token_str': 'Women'},
   {'sequence': '<s>Men should be at home and take care of their children.</s>',
    'score': 0.01326736994087696,
    'token': 17762,
    'token_str': 'Men'}]

This could also prove useful in the setting of using MLMs for cloze tasks and few/zero-shot prediction:

nlp("The acting was believable and the action was outstanding. The sentiment of this review is <mask>.")
> [{'sequence': '<s>The acting was believable and the action was outstanding. The sentiment of this review is clear.</s>',
    'score': 0.10008594393730164,
    'token': 699,
    'token_str': 'Ġclear'},
   {'sequence': '<s>The acting was believable and the action was outstanding. The sentiment of this review is undeniable.</s>',
    'score': 0.05471134930849075,
    'token': 29941,
    'token_str': 'Ġundeniable'}]

nlp("The acting was believable and the action was outstanding. The sentiment of this review is <mask>.",
    targets=[' positive', ' negative'])
> [{'sequence': '<s>The acting was believable and the action was outstanding. The sentiment of this review is positive.</s>',
    'score': 0.04867269843816757,
    'token': 1313,
    'token_str': 'Ġpositive'},
   {'sequence': '<s>The acting was believable and the action was outstanding. The sentiment of this review is negative.</s>',
    'score': 0.0009897189447656274,
    'token': 2430,
    'token_str': 'Ġnegative'}]

Notes:

• Passed targets must be in model vocab. If they're not, the target word is tokenized and the first token is used (with a user warning)
• Could possibly also be done with text-generation, but that might go against the spirit of that pipeline
• Current implem. can't do multi-input with different targets for each instance. Adding this would be a little less clean because you'd have to handle different output shapes for each instance, but I can add it in if that is important.

[LysandreJik]

I think that's a very cool feature! Maybe also print a warning if one of the tokens gets tokenized to an unknown token?

I think single input is okay for now. We should look at handling multi-input in the fill-mask pipeline as a whole, so if you want to take a stab at that, please feel free to do so :).

[codecov]

Codecov Report

Merging #6239 into master will increase coverage by 0.03%.
The diff coverage is 96.00%.

@@            Coverage Diff             @@
##           master    #6239      +/-   ##
==========================================
+ Coverage   79.61%   79.64%   +0.03%     
==========================================
  Files         146      146              
  Lines       26597    26618      +21     
==========================================
+ Hits        21175    21200      +25     
+ Misses       5422     5418       -4     

Impacted Files	Coverage Δ
src/transformers/pipelines.py	79.94% <96.00%> (+0.15%)	⬆️
src/transformers/tokenization_marian.py	68.14% <0.00%> (-25.67%)	⬇️
src/transformers/tokenization_xlm_roberta.py	84.52% <0.00%> (-10.72%)	⬇️
src/transformers/file_utils.py	80.30% <0.00%> (+0.25%)	⬆️
src/transformers/generation_tf_utils.py	86.46% <0.00%> (+4.76%)	⬆️
src/transformers/tokenization_bart.py	95.77% <0.00%> (+35.21%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0513f8d...dad4431. Read the comment docs.

[joeddav]

We should look at handling multi-input in the fill-mask pipeline as a whole

The fill-mask pipeline does, at least if we mean the same thing by "multi-input". It can take an arbitrarily long list of strings and return the top-k predictions for each.

[sshleifer]

Nice tests. Not sure whether this should be in a separate ScoreTargetsPipeline that inherits from FillMaskPipeline or some such. I don't have a strong opinion.

[joeddav]

Awesome. @LysandreJik am I set to merge?

[LysandreJik]

Great, LGTM