Pegasus for summarization !

[jpcorb20]

🌟 New model addition

Model description

https://ai.googleblog.com/2020/06/pegasus-state-of-art-model-for.html?m=1

https://arxiv.org/abs/1912.08777

Abstract
Recent work pre-training Transformers with self-supervised objectives on large text corpora has shown great success when fine-tuned on downstream NLP tasks including text summarization. However, pre-training objectives tailored for abstractive text summarization have not been explored. Furthermore there is a lack of systematic evaluation across diverse domains. In this work, we propose pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective. In PEGASUS, important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. We evaluated our best PEGASUS model on 12 downstream summarization tasks spanning news, science, stories, instructions, emails, patents, and legislative bills. Experiments demonstrate it achieves state-of-the-art performance on all 12 downstream datasets measured by ROUGE scores. Our model also shows surprising performance on low-resource summarization, surpassing previous state-of-the-art results on 6 datasets with only 1000 examples. Finally we validated our results using human evaluation and show that our model summaries achieve human performance on multiple datasets.

Open source status

• the model implementation is available: https://github.com/google-research/pegasus
• the model weights are available: https://github.com/google-research/pegasus
• who are the authors: Jingqing Zhang @JingqingZ, Yao Zhao @yaozhaogoogle, Mohammad Saleh and Peter J. Liu

[JingqingZ]

Thanks! The model checkpoints are available actually. Check here :)

[PingYu-iris]

Hope to provide a pytorch version code

[jpcorb20]

I might try the Huggingface's weight transfer code from tensorflow to pytorch in July if nobody's working on this post

[sshleifer]

Work has started on this, but we are still a few weeks out.

[chetanambi]

Just wanted to know when this model will be available

[sshleifer]

We're a little behind schedule. I'd say 60% by August 1, 90% by Sept 1.

[chrisdoyleIE]

this is awesome.

[MichaelJanz]

Very cool! Can it also be evaluated with Bert-Score?

[1337-Pete]

Can't wait for this...

[sshleifer]

Converted torch checkpoints are now available on master if you build from source.
Here is a list of available checkpoints.
PR: #6340

Usage:

from transformers import PegasusForConditionalGeneration, PegasusTokenizer
src_text = [
    """ PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow."""
]

model_name = 'google/pegasus-xsum'
torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'
tokenizer = PegasusTokenizer.from_pretrained(model_name)
model = PegasusForConditionalGeneration.from_pretrained(model_name).to(torch_device)
batch = tokenizer.prepare_seq2seq_batch(src_text, truncation=True, padding='longest').to(torch_device)
translated = model.generate(**batch)
tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)
assert tgt_text[0] == "California's largest electricity provider has turned off power to tens of thousands of customers."

Please make a new issue if you encounter a bug with the torch checkpoints and assign @sshleifer .
For conceptual/how to questions, ask on discuss.huggingface.co, (you can also tag @sshleifer. )

Still TODO:

• Tensorflow 2.0 implementation.
• ROUGE score is slightly worse than the original paper because we don't implement length penalty the same way. If anyone wants to try it, see Experiment: ROUGE impact of using pegasus length-penalty implementation #6420 .
• fp16 doesn't work for generation or finetuning
• I have not tried finetuning yet, no guarantees on that working well or replicating the paper.

[1337-Pete]

I assume these checkpoints are based on Mixed & Stochastic models, as opposed to models trained exclusively on either C4 or HugeNews?

[sshleifer]

Yes!

[chetanambi]

@sshleifer I am trying this code on Colab but running into below error. Can you let me know what is the issue?

ImportError: cannot import name 'PegasusForConditionalGeneration'

[beni1864]

I'm having the same issue as @chetanambi

[sshleifer]

I think you need to install from source, it's not part of the latest release. (will be in the next release).

[yxyzzz]

@sshleifer :

for the following model:
model_name = 'google/pegasus-cnn_dailymail';

I encountered this error when running:
translated = model.generate(**batch)
'---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
in
1 batch = tokenizer.prepare_seq2seq_batch(src_text, truncation=True, padding='longest').to(torch_device)
----> 2 translated = model.generate(**batch)
3 tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
13 def decorate_context(*args, **kwargs):
14 with self:
---> 15 return func(*args, **kwargs)
16 return decorate_context
17

~/projects/transformers/src/transformers/generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id, use_cache, **model_specific_kwargs)
394 encoder = self.get_encoder()
395
--> 396 encoder_outputs: tuple = encoder(input_ids, attention_mask=attention_mask)
397
398 # Expand input ids if num_beams > 1 or num_return_sequences > 1

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
720 result = self._slow_forward(*input, **kwargs)
721 else:
--> 722 result = self.forward(*input, **kwargs)
723 for hook in itertools.chain(
724 _global_forward_hooks.values(),

~/projects/transformers/src/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, output_attentions, output_hidden_states, return_dict)
328
329 inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale
--> 330 embed_pos = self.embed_positions(input_ids)
331 x = inputs_embeds + embed_pos
332 x = self.layernorm_embedding(x)

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
720 result = self._slow_forward(*input, **kwargs)
721 else:
--> 722 result = self.forward(*input, **kwargs)
723 for hook in itertools.chain(
724 _global_forward_hooks.values(),

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
13 def decorate_context(*args, **kwargs):
14 with self:
---> 15 return func(*args, **kwargs)
16 return decorate_context
17

~/projects/transformers/src/transformers/modeling_bart.py in forward(self, input_ids, use_cache)
1337 # starts at 0, ends at 1-seq_len
1338 positions = torch.arange(seq_len, dtype=torch.long, device=self.weight.device)
-> 1339 return super().forward(positions)

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/nn/modules/sparse.py in forward(self, input)
122
123 def forward(self, input: Tensor) -> Tensor:
--> 124 return F.embedding(
125 input, self.weight, self.padding_idx, self.max_norm,
126 self.norm_type, self.scale_grad_by_freq, self.sparse)

~/anaconda3/envs/abstractive_summarizer/lib/python3.8/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
1812 # remove once script supports set_grad_enabled
1813 no_grad_embedding_renorm(weight, input, max_norm, norm_type)
-> 1814 return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
1815
1816

IndexError: index out of range in self'

[sshleifer]

@yxyzzz can you make a new issue and follow the bug-report template. I can't reproduce based on what you've provided. Thanks!

[chetanambi]

I think you need to install from source, it's not part of the latest release. (will be in the next release).

Could you please let me know how to do this. Thanks!!

[arun-ghontale]

@chetanambi The instructions are provided here

[andrei-volkau]

@sshleifer
I installed transformers from the source using the current master branch.

I experience the following issue. 

>>> from transformers import PegasusForConditionalGeneration, PegasusTokenizer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/env5/lib/python3.6/site-packages/transformers/__init__.py", line 21, in <module>
    from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfig
  File "/home/ubuntu/env5/lib/python3.6/site-packages/transformers/configuration_albert.py", line 18, in <module>
    from .configuration_utils import PretrainedConfig
  File "/home/ubuntu/env5/lib/python3.6/site-packages/transformers/configuration_utils.py", line 24, in <module>
    from .file_utils import CONFIG_NAME, cached_path, hf_bucket_url, is_remote_url
  File "/home/ubuntu/env5/lib/python3.6/site-packages/transformers/file_utils.py", line 32, in <module>
    from .utils import logging
ModuleNotFoundError: No module named 'transformers.utils'

question It is the problem with the current master. How many commits do I need to rollback to sucsessuly run PEGASUS before September release?

Thank you in advance for the info!

[sshleifer]

master fixed by #6754 .

[andrei-volkau]

master fixed by #6754 .

@sshleifer

(1) I confirm that master is working now. So I was able to successfully run PEGASUS.

(2) Is there any way to control a length of a resulting summary made by PEGASUS? I would like to generate longer summaries.

[JingqingZ]

(2) Is there any way to control a length of a resulting summary made by PEGASUS? I would like to generate longer summaries.

@andrei-volkau

You can (1) fine-tune PEGASUS on a customised dataset which has longer summaries (2) tune the hyper-parameter beam_alpha which can lead to slightly longer/shorter summaries.

[sshleifer]

beam_alpha is called "length penalty" in this repo.

Be that length_penalty is named confusingly: (#4915)

• Increasing length_penalty will result in longer generations.
• Decreasing length_penalty will result in shorter generations.
• the formula differs slightly from the pegasus paper (Experiment: ROUGE impact of using pegasus length-penalty implementation #6420)

[mthielk]

Is there a short finetuning example somewhere?

[sshleifer]

Nothing short. Finetuning with examples/seq2seq/finetune.py https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune_pegasus_xsum.sh is almost ready (will be ready after #6654). To use that you should read the README.MD which covers how to format your data.

[chetanambi]

@chetanambi The instructions are provided here

I was able to run the models successfully. During the summarization I would like to run with different beam size. How can I do this?

Thanks!!

[umerhasan17]

Interesting, when I ran the example in the documentation (copied below).

I got the output: California's largest electricity provider has turned off power to hundreds of thousands of customers.

Whereas the assertion output was: California's largest electricity provider has turned off power to tens of thousands of customers.

Could someone shine a light on why this might be the case and which one is the 'correct' output? I'm certain I didn't change anything.

from transformers import PegasusForConditionalGeneration, PegasusTokenizer
import torch
src_text = [
    """ PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow."""
]

model_name = 'google/pegasus-xsum'
torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'
tokenizer = PegasusTokenizer.from_pretrained(model_name)
model = PegasusForConditionalGeneration.from_pretrained(model_name).to(torch_device)
batch = tokenizer.prepare_seq2seq_batch(src_text, truncation=True, padding='longest').to(torch_device)
translated = model.generate(**batch)
tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)
assert tgt_text[0] == "California's largest electricity provider has turned off power to tens of thousands of customers."

[sshleifer]

The docs are wrong, the code is right:
#6526 (merged since documentation was written) affected output (in a good way).
Update: I fixed the docs.

[chetanambi]

@sshleifer I am trying to implement this in a machine that is not connected to internet. So, I will have to download the model (ex: reddit-tifu) and pass the location to from_pretrained. Could you please suggest what all the files I need to download. Apperciate your help.

from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("google/pegasus-reddit_tifu")
model = AutoModelWithLMHead.from_pretrained("google/pegasus-reddit_tifu")

[sshleifer]

You can figure that out on your machine with internet by calling

from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("google/pegasus-reddit_tifu")
model = AutoModelWithLMHead.from_pretrained("google/pegasus-reddit_tifu")
model.save_pretrained('local_pegasus')
tokenizer.save_pretrained('local_pegasus')

Should contain ['config.json', 'pytorch_model.bin', 'tokenizer_config.json', 'special_tokens_map.json' 'spiece.model']

[chetanambi]

Thanks @sshleifer . I was able to figure it out by looking at the implementation for from_pretrained method. I have implemented it successfully now. Thanks !

[1337-Pete]

Thanks @sshleifer for all of your efforts on this. Your & HF's work is such a big win for the NLP community, I can't thank you enough.

Out of curiosity, any sense for when TF2.0 support may go live?

[sshleifer]

Thanks. I don't have a great guess, but it will be more than a few weeks. Feel free to tinker with #5411.
Our new tensorflow maven @jplu is trying to make some big API improvements, so I am waiting for those to settle before adding (Bart, Pegasus, Marian, mBART) TF support all in one go.