detail in input construction during pretraining

[hwijeen]

Hi,
I have a question on the input construction.

I get the basic idea from the clear figure in the paper:

But I guess the "original text" is not actually a single sentence but rather consecutive tokens, presumably crossing the sentence boundary. One option described in the RoBERTa paper is to pack input sequence with consecutive max_seq_len(say, 512) tokens, which can cross sentence or document boundary.

Could you explain how input is actually made or point to the relevant code? This self-contained library looks great, but it is hard for me to pinpoint where input construction happens.

[craffel]

We do concatenate across documents. The code is here:
https://github.com/google-research/text-to-text-transfer-transformer/blob/main/t5/data/preprocessors.py#L1864