chapter 06 - summarization - processing the entire dataset

[amscosta]

Information

The question or comment is about chapter:

• Introduction
• Text Classification
• Transformer Anatomy
• Multilingual Named Entity Recognition
• Text Generation
• Summarization
• Question Answering
• Making Transformers Efficient in Production
• Dealing with Few to No Labels
• Training Transformers from Scratch
• Future Directions

Question or comment

Great book.
My question is very simple :
How can I extend the summarizing process for the entire dataset.
I.e. , from the first row:
sample_text = dataset["train"][1]["article"][:2000]
To all rows.
Apologies if sounds very silly.

[Ice-Citron]

sample_texts = [article[:2000] for article in dataset["train"]["article"]]

def shorten_article(example):
    example["article"] = example["article"][:2000]
    return example

dataset["train"] = dataset["train"].map(shorten_article)

[Ice-Citron]

not sure if these work. try em out