Improve docs #612
Merged
Improve docs #612
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The documentation is not available anymore as the PR was closed or merged. |
vwxyzjn
approved these changes
Aug 4, 2023
docs/source/how_to_train.md
Outdated
Comment on lines
5
to
10
| When performing classical supervised fine-tuning of language models, the loss (especially the validation loss) serves as a good indicator of the training progress. However, in Reinforcement Learning (RL), the loss becomes less informative about the model's performance, and its value may fluctuate while the actual performance improves. | ||
|
|
||
| To address this, we recommend focusing on two key metrics: | ||
|
|
||
| **Mean Reward**: The primary goal is to maximize the reward achieved by the model during RL training. | ||
| **Objective KL Divergence**: KL divergence (Kullback-Leibler divergence) measures the dissimilarity between two probability distributions. In the context of RL training, we use it to quantify the difference between the current model and a reference model. Ideally, we want to keep the KL divergence between 0 and 10 to ensure the model's generated text remains close to what the reference model produces. |
There was a problem hiding this comment.
Maybe we can merge this and https://github.com/lvwerra/trl/blob/main/docs/source/logging.mdx? I think another objective is entropy, where we would want the model to be as chaotic as possible.
There was a problem hiding this comment.
I added a link to the logging page, what do you think?
|
|
||
| When training RL models, optimizing solely for reward may lead to unexpected behaviors, where the model exploits the environment in ways that don't align with good language generation. In the case of RLHF, we use a reward model trained to predict whether a generated text is highly ranked by humans. | ||
|
|
||
| However, the RL model being optimized against the reward model may learn patterns that yield high reward but do not represent good language. This can result in extreme cases where the model generates texts with excessive exclamation marks or emojis to maximize the reward. In some worst-case scenarios, the model may generate patterns completely unrelated to natural language yet receive high rewards, similar to adversarial attacks. |
There was a problem hiding this comment.
We can probabaly give some examples/references. E.g., Table 10 from https://arxiv.org/pdf/1909.08593.pdf
Comment on lines
+51
to
+57
| Debugging the RL pipeline can be challenging due to its complexity. Here are some tips and suggestions to make the process easier: | ||
|
|
||
| - **Start from a working example**: Begin with a working example from the trl repository and gradually modify it to fit your specific use-case. Changing everything at once can make it difficult to identify the source of potential issues. For example, you can start by replacing the model in the example and once you figure out the best hyperparameters try to switch to your dataset and reward model. If you change everything at once you won't know where a potential problem comes from. | ||
| - **Start small, scale later**: Training large models can be very slow and take several hours or days until you see any improvement. For debugging this is not a convenient timescale so try to use small model variants during the development phase and scale up once that works. That being said you sometimes have to be careful as small models might not have the capacity to solve a complicated task either. | ||
| - **Start simple**: Try to start with a minimal example and build complexity from there. Your use-case might require for example a complicated reward function consisting of many different rewards - try to use one signal first and see if you can optimize that and then add more complexity after that. | ||
| - **Inspect the generations**: It's always a good idea to inspect what the model is generating. Maybe there is a big in your post-processing or your prompt. Due to bad settings you might cut-off generations too soon. These things are very hard to see on the metrics but very obvious if you look at the generations. | ||
| - **Inspect the reward model**: If you reward is not improving over time maybe there's an issue with the reward model. You can look at extreme cases to see if it does what it should: e.g. in the sentiment case you can check if simple positive and negative examples really get different rewards. And you can look at the distribution of your dataset. Finally, maybe the reward is dominated by the query which the model can't affect so you might need to normalize this (e.g. reward of query+response minus reward of the query). |
docs/source/use_model.md
Outdated
| ```python | ||
| from transformers import AutoTokenizer, AutoModelForCausalLM | ||
|
|
||
| model_name_or_path = "path/to/your/model/or/name/on/hub" |
There was a problem hiding this comment.
Maybe we can provide a working example and put # "path/to/your/model/or/name/on/hub"
There was a problem hiding this comment.
fixed, except for the adapter as i don't have a good example at hand :)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal of this PR is to improve the overall docs: