VRAM requrements

[dp289m]

What are the GPU RAM requrements please?
Can it be ported to CPU maybe?

Congratulations to all of us. Cautiously curious what will come out of this...

[Konard]

I had created an issue about hardware requirements, maybe it will be possible to get answers there: #62

At the moment as it have to use ~300 GB model, looks like it is unlikely that it will work on CPU without changes to code and to the model, it also stated that it requires multi-GPU machine, and I guess more than 300 GB VRAM will be required.

[ChainsDW]

damn!! whatever, it's great!!

[yxc0915]

OMG，300G！！i only have 24gb vram......😭😭😭

[1511878618]

only 4060, ash.........

[AdaptiveStep]

I bet you could run it on the cpu, at a speed of 1token/day :D

[goeielewe]

https://aws.amazon.com/ec2/instance-types/p5/

[elysiumplain]

Sales reps crawling GH discussion pages now? \s

[goeielewe]

👀

[vsbn]

I know this is obvious, but I have tried this on my 48 GB modified VRAM 3090, and it did not have enough vram.

[Whytehorse2022]

Running a large language model (LLM) with the specifications you provided would require substantial hardware resources. Here's an overview of the hardware requirements you might need:

1. Graphics Processing Units (GPUs): LLMs of this scale typically require multiple high-end GPUs to handle the computational load. You would likely need at least 8-16 GPUs with a minimum of 32-40 GB of VRAM each, such as NVIDIA A100 or H100 GPUs. These GPUs would need to be interconnected using high-bandwidth interconnects like NVLink or NVSwitch.

2. CPU: You would need a powerful CPU or multiple CPUs to handle data preprocessing, input/output operations, and other auxiliary tasks. A modern server-grade CPU with at least 32 cores and 64 threads would be recommended, such as AMD EPYC or Intel Xeon Scalable processors.

3. Memory (RAM): With 314B parameters and a 6,144 embedding size, you would need a significant amount of RAM to store the model weights and intermediate activations. Depending on the specific implementation and optimization techniques, you might require at least 1-2 TB of high-performance DDR4 or DDR5 RAM.

4. Storage: You would need a high-speed storage system, such as NVMe SSDs or a distributed file system, to store the model checkpoints and data. The storage requirements would depend on the number of checkpoints you plan to store and the amount of data you need to process, but you should plan for at least several terabytes of storage.

5. Networking: If you plan to distribute the model across multiple GPUs or machines, you would need a high-speed network infrastructure, such as 100 Gbps Ethernet or InfiniBand, to facilitate communication between the devices.

6. Power and Cooling: Running multiple high-end GPUs and CPUs will generate a significant amount of heat, so you would need a robust power supply and cooling system to ensure stable operation.

It's worth noting that running an LLM of this scale is a significant undertaking and would likely require a dedicated high-performance computing (HPC) cluster or a cloud-based solution from providers like AWS, Google Cloud, or Microsoft Azure. Additionally, you might need to consider techniques like model parallelism, tensor parallelism, or pipeline parallelism to distribute the workload across multiple devices effectively.

[TechGym]

Really appreciate the details you provided.

[AdaptiveStep]

There has to be some smarter way to cache the calculations, so it won't go so slow.. Can't we just put a bunch of calculations on 100TB of nvme M2s? And thereby dilute the model into something more manageable?

[Whytehorse2022]

Yes you can take a large model and make it smaller but first you have to be able to run it. You could probably run it very slowly using virtual shared memory. It seems like lots of the bugs reported were people exceeding /dev/shm(shared memory). Maybe just add a ton of disk space to /dev/shm?

[CorniiDog]

At that rate, why not allow Grok-1 to cache onto VRAM while the rest uses dedicated RAM? That would significantly be more cost effective would it not?

[StefanOjanen]

You don't split between VRAM and RAM because you'll spend all of your compute time just waiting for data to move around. This (time it takes to update weights stored across all the GPUs) is the general bottleneck in multi-node GPU clusters in the first place. To run such a model on a "home PC", feasible setup would be like a Threadripper CPU with enough RAM to store the whole model there and forget about GPUs altogether.

[StefanOjanen]

grok-1 GPU memory requirements
VRAM for inference: 314 GB
VRAM for training: 41270 GB
VRAM for fine-tuning (LoRa): 39389 GB
VRAM for fine-tuning (QLoRa): 27573 GB

I calculated those using the following parameters:
Model Configuration (Grok-1)
Sequence Length [tokens] | 8192
Batch Size | 32
Hidden Size | 32768
Layers | 64
Attention Heads | 48
Tensorparallelism Degree (1 if not) | 1
Number of fp8 Parameters in 1M | 314000
Number of fp16 Parameters in 1M | 0
Number of fp32 Parameters in 1M | 0
Optimizer | 8-bit

Metrics Training | Memory in Gigabyte
Memory Consumption Training | 41270
Memory Using Full Activations Recomputation | 3612

Metrics Inference |  Memory in Gigabyte
Memory Consumption Inference | 314

Finetuning Configuration |  
r_lora | 64

Metrics Finetuning |  Memory in Gigabyte
LoRa | 39389
QLoRa | 27573

[PenutChen]

Are you inference in 8k sequence length?

[TechGym]

Sounds more reachable than what Why... said.

[StefanOjanen]

Grok-1 has 8k context window, only Grok-1.5 extended this to 128k

[dp289m]

So even RTX 4090 owners can not yet run GROK locally. Wonder how far can optimization go without considerable loss of quality...

[MrRaja23]

It will take one or more weeks before we get a locally inferable grok-1 (Assuming they Prune and Quantize it without to much loss which any loss could be recovered via Finetuning hopefully) overall wishful thinking :)