[SFTTrainer] Fix non packed dataset

[younesbelkada]

What does this PR do?

This PR properly educates users on how to correctly use formatting_func method when someone uses a non-packed dataset.
Since the dataset processing calls dataset.map(xxx, batched=True) under the hood, it is important to return an array of processed texts to properly process all texts from the dataset example batch, otherwise it will lead to silent bugs that are hard to understand such as the one described in #439

from datasets import load_dataset
from trl import SFTTrainer
import transformers

dataset = load_dataset("tatsu-lab/alpaca", split="train")

model = transformers.AutoModelForCausalLM.from_pretrained("facebook/opt-350m")
tokenizer = transformers.AutoTokenizer.from_pretrained("facebook/opt-350m")

def formatting_prompts_func(examples):
    output_text = []
    for i in range(len(examples["instruction"])):
        instruction = examples["instruction"][i]
        input_text = examples["input"][i]
        response = examples["output"][i]

        if len(input_text) >= 2:
            text = f'''Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
            
            ### Instruction:
            {instruction}
            
            ### Input:
            {input_text}
            
            ### Response:
            {response}
            '''
        else:
            text = f'''Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
            
            ### Instruction:
            {instruction}
            
            ### Response:
            {response}
            '''
        output_text.append(text)

    return output_text

trainer = SFTTrainer(
    model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    formatting_func=formatting_prompts_func,
    max_seq_length=256,
    packing=False,
)

trainer.train()

The PR adds a sanity check when processing the dataset, and adds the argument padding=True, to always return a sequence of length max_seq_len and correctly appends the attention mask to the output dataset as well.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[vwxyzjn]

Oh interesting. So previously, we were dumping an entire dataset to the prompt?

[younesbelkada]

Exactly yes :D the previous examples on the documentation were wrong and we were dumping the entire mini-batches when processing the dataset .. :/

[PhilDakin]

@younesbelkada this code is still incorrect - consider the case where all samples in the dataset are less than max_seq_len. Each batch will be padded to the largest element in the batch, but no data will pass the if length == max_seq_len check below.

Perhaps:

                padding='max_length',

[younesbelkada]

Yes you are correct thanks a lot for flagging, do you want to open a PR for that? happy to do it otherwise

[ahmadmustafaanis]

Will this fine-tune the complete model end-to-end, or will this example fine-tune just a portion of it, like in Lora?

[lvwerra]

The above example will train the full model but there are also options to use LoRA.

[hy-chen]

Is this merged in main and then reverted? because padding is still False in _prepare_non_packed_dataloader in main (0.7.9)

[hy-chen]

Is this merged in main and then reverted? because padding is still False in _prepare_non_packed_dataloader in main (0.7.9)

Actually padding was turned off by this PR: #512

Now running SFT on alpaca gives
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (labelsin this case) have excessive nesting (inputs typelistwhere typeint is expected).

[haochuan-li]

Any update on this issue?