DeepSpeed integration for <10B models on strangeporks.

[TheRootOf3]

Includes #105.

[Willmish]

Setting up the environment for deepspeed training

1. Create a venv after sourcing source /share/apps/source_files/python/python-3.11.5_bzip2.source (this includes python developer headers, so there wont be issues with missing Python.h files)
2. install all requirements (accelerate, deepspeed, torch (should be 2.3.1>), transformers, and might be a few others depending on used model)
3. Configure deepspeed using accelerate config. See this pdf for ZeRO 2 config with RAM (CPU) parameter offloading:
   NOTE_SNLP_DEEPSPEED.pdf
4. Launch llm unlearning as before, but now source the venv with deepspeed, and use accelerate launch instead of python3 as the launcher.
5. There might be potenitally more issues if multiple deepspeed sessions running on the same node (may need to tweak --main_process_port https://huggingface.co/docs/accelerate/en/package_reference/cli ) and potenitally CUDA_VISIBLE_DEVICES environment var

If encountering an error: while Building extension module cpu_adam...

FAILED: cpu_adam.so
c++ cpu_adam.o cpu_adam_impl.o -shared -lcurand -L/home/sduchnie/strangepork_venv/lib/python3.11/site-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o cpu_adam.so
/usr/bin/ld: cannot find -lcurand
collect2: error: ld returned 1 exit status

It means the lib curand for CUDA is not visible by the linker (see microsoft/DeepSpeed#3929 ) and needs to be linked in manually. you can do so by:

1. Cd into: cd venv/lib/python3.11/site-packages/torch/lib (or whatever your lib is called)
2. create a symbolic link for the missing lib (should be located in: /usr/local/cuda/lib64/libcurand.so): ln -s /usr/local/cuda/lib64/libcurand.so .
3. Retry running the command

[TheRootOf3]

@Willmish Useful stuff, please convert to a markdown and add to a documentation directory :)