Skip to content

Releases: mosaicml/llm-foundry

v0.2.0

04 Jul 05:36
@vchiley vchiley
v0.2.0
d0efe55
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.2.0

🚀 LLM Foundry v0.2.0

LLM Foundry is an efficient codebase for training, evaluating, and deploying Large Language Models (LLM). LLM Foundry serves as the efficient training codebase for the MPT-7B and MPT-30B models. Our emphasis is on efficiency, scalability, and ease-of-use, to enable fast iteration and prototyping.

We are excited to share the release of v0.2.0, packed with support for new hardware, features, and tutorials.

📖 Tutorials

We have released new tutorial content and helper scripts for dataset preparation, pre-training, fine-tuning, and inference!

To start off, a basic walkthrough and answers to FAQs can be found in our Basic Tutorial.

Next, detailed guides for different workflows are linked below:

Training

  1. Part 1: LLM Pretraining
    1. Installation
    2. Dataset Preparation
    3. How to start single and multi-node pretraining
  2. Part 2: LLM Finetuning
    1. Using a dataset on the HuggingFace Hub
    2. Using a local dataset
    3. Using a StreamingDataset (MDS) formatted dataset locally or in an object store

In addition, for a more advanced and self-contained example of finetuning the MPT-7B model, see Finetune Example.

Inference

The inference tutorials cover several new features we've added that improve integration with HuggingFace and FasterTransformer libraries:

Major Features

LLM Foundry now uses Composer v0.15.0 and Streaming v0.5.1 as minimum requirements. For more details, see their release notes for Composer and Streaming for all the improvements.

⚠️ The new Streaming release includes a few API changes, see the Streaming v0.5 release notes for more details. Our API have also been changed to reflect these API modifications.

  1. 🆕 Torch 2.0 support

    LLM Foundry is now Torch 2.0 compatible!

    Note: we have not tested torch.compile, but do not expect significant performance improvements.

  2. ⚡ H100 Support

    We now support NVIDIA H100 systems! See our blog post on Benchmarking LLMs on H100 GPUs for initial performance and convergence details.

    To run LLM Foundry with NVIDIA H100 systems, be sure to use a docker images that has CUDA 11.8+ and PyTorch 2.0+ versions.

    For example, mosaicml/pytorch:2.0.1_cu118-python3.10-ubuntu20.04 from our dockerhub has been tested with NVIDIA H100 systems.

    No code changes should be required.

  3. 📈 AMD MI250 GPU Support

    With the release of PyTorch 2.0 and ROCm 5.4+, excited to share that LLM training now works out of the box on AMD Datacenter GPUs! Read our blog post on Training LLMs with AMD MI250 GPUs for more details.

    Running with our stack was straightforward: use the ROCm 5.4 docker image rocm/dev-ubuntu-20.04:5.4.3-complete; and install PyTorch for ROCm 5.4 and install Flash Attention.

    Modify your configuration settings:

    • attn_impl=flash instead of the default triton
      • Note: ALiBi is currently not supported with attn_impl=flash.
    • loss_fn=torch_crossentropy instead of the default fused_crossentropy.

  4. 🚧 LoRA finetuning (Preview)

    We have included a preview release of Low Rank Adaptation (LoRA) support for memory-efficient fine-tuning of LLMs (Shen et al, 2021).

    To use LoRA, follow the instructions found here.

    Note: This is a preview feature, please let us know any feedback! The API and support is subject to change.

  5. 🔎 Evaluation Refactor (#308)

    Our evaluation suite has been significantly refactored into our Model Gauntlet approach. This includes a number of breaking API changes to support multiple models:

    • Instead of model, use the models keyword and provide a list of models.
    • tokenizer is now model-specific.

    For example, to run the gauntlet of various eval tasks with mosaicml/mpt-7b:

    cd llm-foundry/scripts
composer eval/eval.py eval/yamls/hf_eval.yaml
    model_name_or_path=mosaicml/mpt-7b

    This release also makes evaluation deterministic even on different number of GPUs.

    For more details on all these changes, see #308

  6. ⏱️ Benchmarking Inference

    To better support the deployment of LLMs, we have included inference benchmarking suite and results across different hardware and other LLM models.

PR List

Read more

Contributors

patrickhwood, dwyatte, and 29 other contributors
Assets 2
Loading

v0.1.1

10 May 18:12
@mvpatel2000 mvpatel2000
v0.1.1
6c16a6e
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.1

What's New

LLM Foundry is now on PyPI!

What's Changed

New Contributors

Full Changelog: v0.1.0...v0.1.1

Contributors

vchiley, alextrott16, and 3 other contributors
Assets 2
Loading

Announcing LLM Foundry and the MPT foundation series

08 May 23:12
@dakinggg dakinggg
v0.1.0
67e61a2
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
Announcing LLM Foundry and the MPT foundation series

🚀 LLM Foundry v0.1.0

This is the first release of MosaicML's LLM Foundry!

Our efficient code for training, evaluating, and deploying LLMs outgrew our examples repository, so we've migrated to a brand new repository dedicated to everything LLMs. Keep watching this space and see the top-level README and our blog post for more details on this announcement!

Model releases

In addition to all the open-source code released here, we're releasing four open-source models that we hope will be useful to the community. All models were trained on the MosaicML platform, using Composer and Streaming. If you're interested in training your own models, or using these models with our optimized inference stack, please reach out!

  • mpt-7b: This is our base 7-billion parameter model, trained for 1 trillion tokens. This model is released with an Apache-2.0 (commercial use permitted) license.
  • mpt-7b-storywriter: All of the models use ALiBi to allow them to exrapolate to longer sequence lengths than they saw during training, but storywriter is our long context model, further pretrained on 65k-token excerpts of a fiction subset of the books3 corpus. This model is released with an Apache-2.0 (commercial use permitted) license.
  • mpt-7b-instruct: This model is instruction finetuned on a dataset we also release, derived from Databrick's Dolly-15k and Anthropic’s Helpful and Harmless datasets. This model is released with a CC-By-SA-3.0 (commercial use permitted) license.
  • mpt-7b-chat: This model is trained to be able to chat by further training on the ShareGPT-Vicuna, HC3, Alpaca, Helpful and Harmless, and Evol-Instruct datasets. This model is released with a CC-By-NC-SA-4.0 (non-commercial use only) license.

Features

Training

We release fully featured code for efficiently training any HuggingFace LLM (including our optimized MPT using FSDP, Composer, and Streaming. Seamlessly scale to multi-gpu and multi-node training, stream your data from one cloud, train on a different cloud, write checkpoints to a third cloud, send your training logs to Weights&Biases, and much more. See the README for more detailed instructions on getting started pretraining and finetuning!

Our MPT model is equipped with the latest advancements in training large transformers (e.g. ALiBi, the LION optimizer, FlashAttention), and is desgined to be easily hackable, configurable, and extendable!

Evaluation

Our evaluation framework, makes it easy to fully re-evaluate any HuggingFace model. We also include copies of the processed data for many popular benchmarks, to make it easy to replicate our evals, and perform your own! We welcome the addition of new benchmarks to our suite. In previous benchmarks, our setup is 8x faster than other eval frameworks on a single GPU and seamlessly achieves linear scaling with multiple GPUs. Built-in support for FSDP makes it possible to evaluate large models and use larger batch sizes for further acceleration.

Inference

MPT is designed to be fast, easy, and cheap to deploy for inference. To begin with, all MPT models are subclassed from the HuggingFace PretrainedModel base class, which means that they are fully compatible with the HuggingFace ecosystem. You can upload MPT models to the HuggingFace Hub, generate outputs with standard pipelines like model.generate(...), build HuggingFace Spaces (see some of ours here!), and more.

What about performance? With MPT’s optimized layers (including FlashAttention and low precision layernorm), the out-of-the-box performance of MPT-7B on GPUs when using model.generate(...) is 1.5x-2x faster than other 7B models like LLaMa-7B. This makes it easy to build fast and flexible inference pipelines with just HuggingFace and PyTorch.

Finally, for the best hosting experience, deploy your MPT models directly on MosaicML’s Inference service. Start with our managed endpoints for models like MPT-7B-Instruct, and/or deploy your own custom model endpoints for optimal cost and data privacy. Check out the Inference blog post for more details!

Assets 2
Loading