output management checkpoints and final model

docs/sagemaker/train.md

@@ -99,6 +99,13 @@ _Note that SageMaker doesn’t support argparse actions. For example, if you wan
 
 Look [here](https://github.com/huggingface/notebooks/blob/master/sagemaker/01_getting_started_pytorch/scripts/train.py) for a complete example of a 🤗 Transformers training script.
 
+## Training Output Management
+
+If `output_dir` in the `TrainingArguments` is set to '/opt/ml/model' the Trainer saves all training artifacts, including logs, checkpoints, and models. Amazon SageMaker archives the whole '/opt/ml/model' directory as `model.tar.gz` and uploads it at the end of the training job to Amazon S3. Depending on your Hyperparameters and `TrainingArguments` this could lead to a large artifact (> 5GB), which can slow down deployment for Amazon SageMaker Inference. 
+You can control how checkpoints, logs, and artifacts are saved by customization the [TrainingArguments](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transformers.TrainingArguments). For example by providing `save_total_limit` as `TrainingArgument` you can control the limit of the total amount of checkpoints. Deletes the older checkpoints in `output_dir` if new ones are saved and the maximum limit is reached.
+
+If you are using the HuggingFace framework estimator, you need to specify a checkpoint output path through hyperparameters. To enable checkpointing, set `output_dir` to `/opt/ml/checkpoints` in hyperparameters, and point `checkpoint_s3_uri` to an S3 location in your estimator ([see Use Checkpoints on Amazon SageMaker Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/model-checkpoints.html)).
+
 ## Create a Hugging Face Estimator
 
 Run 🤗 Transformers training scripts on SageMaker by creating a [Hugging Face Estimator](https://sagemaker.readthedocs.io/en/stable/frameworks/huggingface/sagemaker.huggingface.html#huggingface-estimator). The Estimator handles end-to-end SageMaker training. There are several parameters you should define in the Estimator:
@@ -339,4 +346,4 @@ huggingface_estimator = HuggingFace(
         hyperparameters = hyperparameters)
 ```
 
-📓 Open the [notebook](https://github.com/huggingface/notebooks/blob/master/sagemaker/06_sagemaker_metrics/sagemaker-notebook.ipynb) for an example of how to capture metrics in SageMaker.
+📓 Open the [notebook](https://github.com/huggingface/notebooks/blob/master/sagemaker/06_sagemaker_metrics/sagemaker-notebook.ipynb) for an example of how to capture metrics in SageMaker.