Update docs

togethercomputer · Apr 18, 2023 · 9a9d186 · 9a9d186
1 parent f83dd16
commit 9a9d186
Show file tree

Hide file tree

Showing 5 changed files with 121 additions and 15 deletions.
diff --git a/README.md b/README.md
@@ -10,6 +10,7 @@ In this repo, you'll find code for:
 
 # Contents
 
+- [Try it out](#try-it-out)
 - [Getting Started](#getting-started)
   * [Requirements](#requirements)
   * [Chatting with Pythia-Chat-Base-7B](#chatting-with-pythia-chat-base-7b)
@@ -28,9 +29,16 @@ In this repo, you'll find code for:
 - [Citing OpenChatKit](#citing-openchatkit)
 - [Acknowledgements](#acknowledgements)
 
+# Try it out
+- [OpenChatKit Feedback App](https://huggingface.co/spaces/togethercomputer/OpenChatKit)
+Feedback helps improve the bot and open-source AI research.
+
+- [Run it on Google Colab](inference/README.md#running-on-google-colab) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/togethercomputer/OpenChatKit/blob/main/inference/example/example.ipynb)
+- Continue reading to run it on your own!
+
 # Getting Started
 
-In this tutorial, you will download Pythia-Chat-Base-7B, an instruction-tuned language model, and run some some inference requests against it using a command-line tool.
+In this tutorial, you will download Pythia-Chat-Base-7B, an instruction-tuned language model, and run some inference requests against it using a command-line tool.
 
 Pythia-Chat-Base-7B is a 7B-parameter fine-tuned variant of Pythia-6.9B-deduped from Eleuther AI. Pre-trained weights for this model are available on Huggingface as [togethercomputer/Pythia-Chat-Base-7B](https://huggingface.co/togethercomputer/Pythia-Chat-Base-7B) under an Apache 2.0 license.
 
@@ -73,7 +81,7 @@ conda activate OpenChatKit
 
 ## Chatting with Pythia-Chat-Base-7B
 
-To help you try the model, [`inference/bot.py`](inference/bot.py) is a simple command-line test harness that provides a shell inferface enabling you to chat with the model. Simply enter text at the prompt and the model replies. The test harness also maintains conversation history to provide the model with context.
+To help you try the model, [`inference/bot.py`](inference/bot.py) is a simple command-line test harness that provides a shell interface enabling you to chat with the model. Simply enter text at the prompt and the model replies. The test harness also maintains conversation history to provide the model with context.
 
 
 Start the bot by calling `bot.py` from the root for the repo.

diff --git a/docs/GPT-NeoXT-Chat-Base-20B.md b/docs/GPT-NeoXT-Chat-Base-20B.md
@@ -184,7 +184,7 @@ Hello human.
 
 Commands are prefixed with a `/`, and the `/quit` command exits.
 
-Please see [the inference README](inference/README.md) for more details about arguments, running on multiple/specific GPUs, and running on consumer hardware.
+Please see [the inference README](GPT-NeoXT-Chat-Base-Inference.md) for more details about arguments, running on multiple/specific GPUs, and running on consumer hardware.
 
 # Monitoring
 

diff --git a/docs/GPT-NeoXT-Chat-Base-Inference.md b/docs/GPT-NeoXT-Chat-Base-Inference.md
@@ -0,0 +1,79 @@
+# GPT-NeoXT-Chat-Base-20B Inference
+
+## Contents
+
+- [Arguments](#arguments)
+- [Hardware requirements](#hardware-requirements-for-inference)
+- [Running on multiple GPUs](#running-on-multiple-gpus)
+- [Running on specific GPUs](#running-on-specific-gpus)
+- [Running on consumer hardware](#running-on-consumer-hardware)
+
+## Arguments
+- `--gpu-id`: primary GPU device to load inputs onto for inference. Default: `0`
+- `--model`: name/path of the model. Default = `../huggingface_models/GPT-NeoXT-Chat-Base-20B`
+- `--max-tokens`: the maximum number of tokens to generate. Default: `128`
+- `--sample`: indicates whether to sample. Default: `True`
+- `--temperature`: temperature for the LM. Default: `0.6`
+- `--top-k`: top-k for the LM. Default: `40`
+- `--retrieval`: augment queries with context from the retrieval index. Default `False`
+- `-g` `--gpu-vram`: GPU ID and VRAM to allocate to loading the model, separated by a `:` in the format `ID:RAM` where ID is the CUDA ID and RAM is in GiB. `gpu-id` must be present in this list to avoid errors. Accepts multiple values, for example, `-g ID_0:RAM_0 ID_1:RAM_1 ID_N:RAM_N`
+- `-r` `--cpu-ram`: CPU RAM overflow allocation for loading the model. Optional, and only used if the model does not fit onto the GPUs given.
+- `--load-in-8bit`: load model in 8-bit. Requires `pip install bitsandbytes`. No effect when used with `-g`. 
+
+## Hardware requirements for inference
+The GPT-NeoXT-Chat-Base-20B model requires at least 41GB of free VRAM. Used VRAM also goes up by ~100-200 MB per prompt. 
+
+- A **minimum of 80 GB is recommended** 
+
+- A **minimum of 48 GB in VRAM is recommended** for fast responses.
+
+If you'd like to run inference on a GPU with <48 GB VRAM, refer to this section on [running on consumer hardware](#running-on-consumer-hardware).
+
+By default, inference uses only CUDA Device 0.
+
+**NOTE: Inference currently requires at least 1x GPU.**
+
+## Running on multiple GPUs
+Add the argument 
+
+```-g ID0:MAX_VRAM ID1:MAX_VRAM ID2:MAX_VRAM ...``` 
+
+where IDx is the CUDA ID of the device and MAX_VRAM is the amount of VRAM you'd like to allocate to the device.
+
+For example, if you are running this on 4x 48 GB GPUs and want to distribute the model across all devices, add ```-g 0:10 1:12 2:12 3:12 4:12```. In this example, the first device gets loaded to a max of 10 GiB while the others are loaded with a max of 12 GiB.
+
+How it works: The model fills up the max available VRAM on the first device passed and then overflows into the next until the whole model is loaded.
+
+**IMPORTANT: This MAX_VRAM is only for loading the model. It does not account for the additional inputs that are added to the device. It is recommended to set the MAX_VRAM to be at least 1 or 2 GiB less than the max available VRAM on each device, and at least 3GiB less than the max available VRAM on the primary device (set by `gpu-id` default=0).**
+
+**Decrease MAX_VRAM if you run into CUDA OOM. This happens because each input takes up additional space on the device.**
+
+**NOTE: Total MAX_VRAM across all devices must be > size of the model in GB. If not, `bot.py` automatically offloads the rest of the model to RAM and disk. It will use up all available RAM. To allocate a specified amount of RAM: [refer to this section on running on consumer hardware](#running-on-consumer-hardware).**
+
+## Running on specific GPUs
+If you have multiple GPUs but would only like to use a specific device(s), [use the same steps as in this section on running on multiple devices](#running-on-multiple-gpus) and only specify the devices you'd like to use. 
+
+Also, if needed, add the argument `--gpu-id ID` where ID is the CUDA ID of the device you'd like to make the primary device. NOTE: The device specified in `--gpu-id` must be present as one of the ID in the argument `-g` to avoid errors.
+
+- **Example #1**: to run inference on devices 2 and 5 with a max of 25 GiB on each, and make device 5 the primary device, add: `--gpu-id 5 -g 2:25 5:25`. In this example, not adding `--gpu-id 5` will give you an error.
+- **Example #2**: to run inference on devices 0 and 3 with a max of 10GiB on 0 and 40GiB on 3, with device 0 as the primary device, add: `-g 0:10 3:40`. In this example, `--gpu-id` is not required because device 0 is specified in `-g`.
+- **Example #3**: to run inference only on device 1 with a max of 75 GiB, add: `--gpu-id 1 -g 1:75`
+
+
+## Running on consumer hardware
+If you have multiple GPUs, each <48 GB VRAM, [the steps mentioned in this section on running on multiple GPUs](#running-on-multiple-gpus) still apply, unless, any of these apply:
+- Running on just 1x GPU with <48 GB VRAM,
+- <48 GB VRAM combined across multiple GPUs
+- Running into Out-Of-Memory (OOM) issues
+
+In which case, add the flag `-r CPU_RAM` where CPU_RAM is the maximum amount of RAM you'd like to allocate to loading model. Note: This significantly reduces inference speeds. 
+
+The model will load without specifying `-r`, however, it is not recommended because it will allocate all available RAM to the model. To limit how much RAM the model can use, add `-r`.
+
+If the total VRAM + CPU_RAM < the size of the model in GiB, the rest of the model will be offloaded to a folder "offload" at the root of the directory. Note: This significantly reduces inference speeds.
+
+- Example: `-g 0:12 -r 20` will first load up to 12 GiB of the model into the CUDA device 0, then load up to 20 GiB into RAM, and load the rest into the "offload" directory.
+
+How it works: 
+- https://github.com/huggingface/blog/blob/main/accelerate-large-models.md
+- https://www.youtube.com/embed/MWCSGj9jEAo
diff --git a/inference/README.md b/inference/README.md
@@ -1,25 +1,37 @@
 # OpenChatKit Inference
 This directory contains code for OpenChatKit's inference.
 
+## Contents
+
+- [Arguments](#arguments)
+- [Hardware requirements](#hardware-requirements-for-inference)
+- [Running on multiple GPUs](#running-on-multiple-gpus)
+- [Running on specific GPUs](#running-on-specific-gpus)
+- [Running on consumer hardware](#running-on-consumer-hardware)
+- [Running on Google Colab](#running-on-google-colab) 
+
 ## Arguments
-- `--gpu-id`: Primary GPU device to load inputs onto for inference. Default: `0`
-- `--model`: name/path of the model. Default = `../huggingface_models/GPT-NeoXT-Chat-Base-20B`
+- `--gpu-id`: primary GPU device to load inputs onto for inference. Default: `0`
+- `--model`: name/path of the model. Default = `../huggingface_models/Pythia-Chat-Base-7B`
 - `--max-tokens`: the maximum number of tokens to generate. Default: `128`
 - `--sample`: indicates whether to sample. Default: `True`
 - `--temperature`: temperature for the LM. Default: `0.6`
 - `--top-k`: top-k for the LM. Default: `40`
 - `--retrieval`: augment queries with context from the retrieval index. Default `False`
 - `-g` `--gpu-vram`: GPU ID and VRAM to allocate to loading the model, separated by a `:` in the format `ID:RAM` where ID is the CUDA ID and RAM is in GiB. `gpu-id` must be present in this list to avoid errors. Accepts multiple values, for example, `-g ID_0:RAM_0 ID_1:RAM_1 ID_N:RAM_N`
 - `-r` `--cpu-ram`: CPU RAM overflow allocation for loading the model. Optional, and only used if the model does not fit onto the GPUs given.
+- `--load-in-8bit`: load model in 8-bit. Requires `pip install bitsandbytes`. No effect when used with `-g`. 
 
 ## Hardware requirements for inference
-The GPT-NeoXT-Chat-Base-20B model requires at least 41GB of free VRAM. Used VRAM also goes up by ~100-200 MB per prompt. 
+The Pythia-Chat-Base-7B model requires:
+
+- **18 GB of GPU memory for the base model**
 
-- A **minimum of 80 GB is recommended** 
+- **9 GB of GPU memory for the 8-bit quantized model**
 
-- A **minimum of 48 GB in VRAM is recommended** for fast responses.
+Used VRAM also goes up by ~100-200 MB per prompt. 
 
-If you'd like to run inference on a GPU with <48 GB VRAM, refer to this section on [running on consumer hardware](#running-on-consumer-hardware).
+If you'd like to run inference on a GPU with less VRAM than the size of the model, refer to this section on [running on consumer hardware](#running-on-consumer-hardware).
 
 By default, inference uses only CUDA Device 0.
 
@@ -32,7 +44,7 @@ Add the argument
 
 where IDx is the CUDA ID of the device and MAX_VRAM is the amount of VRAM you'd like to allocate to the device.
 
-For example, if you are running this on 4x 48 GB GPUs and want to distribute the model across all devices, add ```-g 0:10 1:12 2:12 3:12 4:12```. In this example, the first device gets loaded to a max of 10 GiB while the others are loaded with a max of 12 GiB.
+For example, if you are running this on 4x 8 GB GPUs and want to distribute the model across all devices, add ```-g 0:4 1:4 2:6 3:6```. In this example, the first two devices get loaded to a max of 4 GiB while the other two are loaded with a max of 6 GiB.
 
 How it works: The model fills up the max available VRAM on the first device passed and then overflows into the next until the whole model is loaded.
 
@@ -53,9 +65,9 @@ Also, if needed, add the argument `--gpu-id ID` where ID is the CUDA ID of the d
 
 
 ## Running on consumer hardware
-If you have multiple GPUs, each <48 GB VRAM, [the steps mentioned in this section on running on multiple GPUs](#running-on-multiple-gpus) still apply, unless, any of these apply:
-- Running on just 1x GPU with <48 GB VRAM,
-- <48 GB VRAM combined across multiple GPUs
+If you have multiple GPUs [the steps mentioned in this section on running on multiple GPUs](#running-on-multiple-gpus) still apply, unless, any of these apply:
+- Running on just 1x GPU with VRAM < size of the model,
+- Less combined VRAM across multiple GPUs than the size of the model,
 - Running into Out-Of-Memory (OOM) issues
 
 In which case, add the flag `-r CPU_RAM` where CPU_RAM is the maximum amount of RAM you'd like to allocate to loading model. Note: This significantly reduces inference speeds. 
@@ -64,8 +76,15 @@ The model will load without specifying `-r`, however, it is not recommended beca
 
 If the total VRAM + CPU_RAM < the size of the model in GiB, the rest of the model will be offloaded to a folder "offload" at the root of the directory. Note: This significantly reduces inference speeds.
 
-- Example: `-g 0:12 -r 20` will first load up to 12 GiB of the model into the CUDA device 0, then load up to 20 GiB into RAM, and load the rest into the "offload" directory.
+- Example: `-g 0:3 -r 4` will first load up to 3 GiB of the model into the CUDA device 0, then load up to 4 GiB into RAM, and load the rest into the "offload" directory.
 
 How it works: 
 - https://github.com/huggingface/blog/blob/main/accelerate-large-models.md
 - https://www.youtube.com/embed/MWCSGj9jEAo
+
+## Running on Google Colab
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/togethercomputer/OpenChatKit/blob/main/inference/example/example.ipynb)
+
+In the [example notebook](example/example.ipynb), you will find code to run the Pythia-Chat-Base-7B 8-bit quantized model. This is recommended for the free version of Colab. If you'd like to disable quantization, simple remove the `--load-in-8bit` flag from the last cell.
+
+Or, simple click on the "Open In Colab" badge to run the example notebook.
diff --git a/inference/example.ipynb → inference/example/example.ipynb b/inference/example.ipynb → inference/example/example.ipynb
@@ -25,7 +25,7 @@
         "colab_type": "text"
       },
       "source": [
-        "<a href=\"https://colab.research.google.com/github/togethercomputer/OpenChatKit/blob/main/inference/example.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
+        "<a href=\"https://colab.research.google.com/github/togethercomputer/OpenChatKit/blob/main/inference/example/example.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
       ]
     },
     {