Update CPT documentation

[tsachiblau]

Currently, the CPT model lacks a code example.
In this pull request, I provide an explanation and a code example to address this.

Thanks,
Tsachi

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[BenjaminBossan]

Thanks for quickly following up with the example. Overall, this is a very nice notebook. I have some smaller comments, please check them out.

A tip: You should be able to run ruff on the notebook if you want to auto-format it: https://docs.astral.sh/ruff/faq/#does-ruff-support-jupyter-notebooks.

A question I had, is the code for CPTDataset, CPTDataCollatorForLanguageModeling etc. specifically for this example, or is there some reference code. If the latter, it would make sense to put a link to the reference code.

In addition, I had a bit of trouble getting this notebook to run due to OOM errors after I increased the dataset size. I tried a few common steps to mitigate this but nothing worked:

• Smaller model: bigscience/bloom-560m
• Reduced MAX_INPUT_LENGTH to 64
• Set per_device_eval_batch_size to 1
• Quantized the model with bitsandbytes (both 8 and 4 bit)

Even with all those steps combined, I couldn't train with 24GB of memory. Somewhat surprisingly, I had to reduce the number of train samples to 50 for training to run. I'd think that with a batch size of 1, it shouldn't matter that much for memory if I have 50 samples or 500. Do you have any idea what could be going on here?

[BenjaminBossan]

Hmm, I'm not sure if this link is going to work from the built docs. It's better if you link directly to the README, i.e. https://github.com/huggingface/peft/blob/main/examples/cpt_finetuning/README.md (of course, the link won't point anywhere right now, but after merging it will be valid).

[BenjaminBossan]

Remove?

[BenjaminBossan]

Same argument about the link.

[BenjaminBossan]

In this section, you use a lot of "we" and "our". Let's try to word it in a more neutral way, as for the reader it could appear like "we" refers to the PEFT maintainers :) So use "The approach" instead of "Our approach" etc.

[BenjaminBossan]

Suggested change

	Templates provide context-aware structure, ensuring the model does not overfit by utilizing structured input-output formats. Using cpt_tokens_type_mask, we gain fine-grained information about the roles of different tokens in the input-output structure. This enables the model to:
	Templates provide context-aware structure, ensuring the model does not overfit by utilizing structured input-output formats. Using `cpt_tokens_type_mask`, we gain fine-grained information about the roles of different tokens in the input-output structure. This enables the model to:

[BenjaminBossan]

It could be helpful to link the paper here.

[BenjaminBossan]

Let's not capitalize here: cpt_train_dataset

[BenjaminBossan]

Instead of hard-coding the model id here, can we re-use the tokenizer_name_or_path variable? Of course, it should be renamed in this case, e.g. to model_id.

[BenjaminBossan]

I assume you chose small subsets to make the notebook run fast. But maybe a little bit more would also be okay? Also, let's add a sentence here that for proper testing, users should use the whole dataset. Maybe there can even be a toggle that users can enable to use the full datasets.

[BenjaminBossan]

Can we delete these arguments?

[tsachiblau]

A tip: You should be able to run ruff on the notebook if you want to auto-format it: https://docs.astral.sh/ruff/faq/#does-ruff-support-jupyter-notebooks.

Done

A question I had, is the code for CPTDataset, CPTDataCollatorForLanguageModeling etc. specifically for this example, or is there some reference code. If the latter, it would make sense to put a link to the reference code.

Not specific to this example, but rather to our method.

Even with all those steps combined, I couldn't train with 24GB of memory. Somewhat surprisingly, I had to reduce the number of train samples to 50 for training to run. I'd think that with a batch size of 1, it shouldn't matter that much for memory if I have 50 samples or 500. Do you have any idea what could be going on here?

This is true. Our method can become computationally expensive as the number of examples grows, as we concatenate all examples to the input and utilize them all during optimization.

Hmm, I'm not sure if this link is going to work from the built docs. It's better if you link directly to the README, i.e. https://github.com/huggingface/peft/blob/main/examples/cpt_finetuning/README.md (of course, the link won't point anywhere right now, but after merging it will be valid).

Done

Remove?

Done

Same argument about the link.

Done

In this section, you use a lot of "we" and "our". Let's try to word it in a more neutral way, as for the reader it could appear like "we" refers to the PEFT maintainers :) So use "The approach" instead of "Our approach" etc.

Done

cpt_tokens_type_mask

Done

It could be helpful to link the paper here.

Done

Let's not capitalize here: cpt_train_dataset

Done

Instead of hard-coding the model id here, can we re-use the tokenizer_name_or_path variable? Of course, it should be renamed in this case, e.g. to model_id.

Done

I assume you chose small subsets to make the notebook run fast. But maybe a little bit more would also be okay? Also, let's add a sentence here that for proper testing, users should use the whole dataset. Maybe there can even be a toggle that users can enable to use the full datasets.

Done

Can we delete these arguments?

Done

[BenjaminBossan]

Thanks a lot for the updates.

This is true. Our method can become computationally expensive as the number of examples grows, as we concatenate all examples to the input and utilize them all during optimization.

Could you expand on this a little bit? I wonder if we could optimize this (e.g. via batching), since PEFT is intended for use with limited memory and ideally, an example as this one should be able to run with 24GB VRAM.

Moreover, for some reason, the updated notebook does not render, I get:

'execution_count' is a required property
Using nbformat v5.10.4 and nbconvert v7.16.1

Could you please check that everything is correct in the notebook? Perhaps, rerunning it from the start with an up-to-date jupyter version is enough.

[tsachiblau]

Could you expand on this a little bit? I wonder if we could optimize this (e.g. via batching), since PEFT is intended for use with limited memory and ideally, an example as this one should be able to run with 24GB VRAM.

CPT is designed as a solution for few-shot scenarios, typically involving up to tens of examples and not exceeding this range. However, the high memory usage you experienced can be attributed to the following key factors:

Self-Attention Mechanism:
The self-attention mechanism in transformers has a quadratic memory requirement with respect to the input sequence length. Concatenating multiple examples into the input significantly increases the sequence length, leading to a dramatic rise in memory usage during both forward and backward passes. This is a fundamental limitation of the self-attention mechanism in large transformer models and directly impacts the feasibility of handling very long inputs.

Gradient Storage:
During optimization, memory consumption scales with the number of tokens involved in the loss calculation. In CPT, we utilize more tokens for the loss by including not only the target examples but also tokens from the context. This ensures that the optimization process benefits from the additional information available in the context, but it also increases memory requirements. For each token used in the loss, gradients and activations need to be stored, leading to a proportional increase in memory consumption as more tokens are included.

Could you please check that everything is correct in the notebook? Perhaps, rerunning it from the start with an up-to-date jupyter version is enough.

Does it works now? I checked it on Colab.

[BenjaminBossan]

typically involving up to tens of examples and not exceeding this range

So this means that as a user, I should not use more than a few tens of examples? If so, let's make this very clear in the docs.

If I use a dataset of 1000 samples, all of them would be used as few shot examples? Would it make sense to add an argument to limit this number?

Does it works now? I checked it on Colab.

Yes, this works now, thanks for the update.

[tsachiblau]

Thank you for the suggestion, and I will ensure this limitation is clearly stated in the documentation.

Similar to In-Context Learning (ICL), there is indeed a practical limit to the number of examples that can be used in the context due to memory constraints. However, users can choose to select a subset of examples for the context while using the remaining examples purely for optimization. While this is not the original design of our method, it could serve as an adaptation to scale the approach for larger datasets with more examples.

Here’s a suggestion to integrate the functionality of selecting a subset of examples directly into your code. This adjustment allows users to limit the number of examples chosen for the context dynamically:

# Iterate through the CPT training dataset
for i in range(len(cpt_train_dataset)):
    # Add input IDs to the context
    context_ids += cpt_train_dataset[i]['input_ids']

    # Add attention mask to the context
    context_attention_mask += cpt_train_dataset[i]['attention_mask']

    # Adjust and add the input type mask to the context
    context_input_type_mask += [
        i + first_type_mask if i > 0 else 0 # Increment type indices dynamically
        for i in cpt_train_dataset[i]['input_type_mask']
        ]

    # Increment the type mask offset after processing the sample
    first_type_mask += 4

[BenjaminBossan]

Thanks for the update to the docs and explaining further.

Here’s a suggestion to integrate the functionality of selecting a subset of examples directly into your code. This adjustment allows users to limit the number of examples chosen for the context dynamically:

To clarify, is the selection of the subset already part of this snippet (which AFAICT is identical to the code in the notebook)? It looks like it iterates through the whole dataset. My guess is that if I wanted to set a limit, I would break out of the loop after a couple of iterations.

Alternatively, my suggestion would be to define a constant like MAX_ICL_SAMPLES = 10 in the notebook and then define a separate icl_dataset with MAX_ICL_SAMPLES samples, which is distinct from the train_dataset (as I don't think it really helps if the few shot samples and the train dataset overlap). Then the first 10 samples would be used for ICL and the train dataset would start after sample 10 and could be much bigger without blowing up memory. WDYT?

[tsachiblau]

Alternatively, my suggestion would be to define a constant like MAX_ICL_SAMPLES = 10 in the notebook and then define a separate icl_dataset with MAX_ICL_SAMPLES samples, which is distinct from the train_dataset (as I don't think it really helps if the few shot samples and the train dataset overlap). Then the first 10 samples would be used for ICL and the train dataset would start after sample 10 and could be much bigger without blowing up memory. WDYT?

Done.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Thanks for adding the documentation and example for CPT. Also nice addition of the ICL limit, I could now successfully run the notebook even with 1000 samples and had a lot of memory to spare.

[tsachiblau]

Thanks so much again! :)

Currently, the link to the example page is: https://github.com/huggingface/peft/blob/main/examples/cpt_finetuning/README.md, which points to the GitHub repository. Could we instead have a link on the official website, similar to this one: https://huggingface.co/docs/peft/main/en/package_reference/cpt?

[BenjaminBossan]

It's a pleasure.

We can always make changes like that, but I'm not sure which link exactly you want to change. The links to https://github.com/huggingface/peft/blob/main/examples/cpt_finetuning/README.md are specifically for examples, whereas https://huggingface.co/docs/peft/main/en/package_reference/cpt is the more technical documentation and thus serves a different purpose. Thus, I don't think changing the link makes sense here. Maybe I'm missing something?

[tsachiblau]

No, it is fine 👍