Error building extension 'fused_adam' with DeepSpeed==0.3.13

[saichandrapandraju]

Hi,

I upgraded DeepSpeed to 0.3.13 and Torch to 1.8.0 and while using DeepSpeed with HF (HuggingFace), I'm getting below error -
RuntimeError: Error building extension 'fused_adam' and here is the stacktrace -

[2021-03-23 07:03:49,374] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13, git-hash=unknown, git-branch=unknown
[2021-03-23 07:03:49,407] [INFO] [engine.py:77:_initialize_parameter_parallel_groups] data_parallel_size: 1, parameter_parallel_size: 1
Using /home/jovyan/.cache/torch_extensions as PyTorch extensions root...
Creating extension directory /home/jovyan/.cache/torch_extensions/fused_adam...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/jovyan/.cache/torch_extensions/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
~/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py in _run_ninja_build(build_directory, verbose, error_prefix)
   1672             check=True,
-> 1673             env=env)
   1674     except subprocess.CalledProcessError as e:

/usr/lib/python3.6/subprocess.py in run(input, timeout, check, *popenargs, **kwargs)
    437             raise CalledProcessError(retcode, process.args,
--> 438                                      output=stdout, stderr=stderr)
    439     return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
<ipython-input-24-3435b262f1ae> in <module>
----> 1 trainer.train()

~/.local/lib/python3.6/site-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, **kwargs)
    901         delay_optimizer_creation = self.sharded_ddp is not None and self.sharded_ddp != ShardedDDPOption.SIMPLE
    902         if self.args.deepspeed:
--> 903             model, optimizer, lr_scheduler = init_deepspeed(self, num_training_steps=max_steps)
    904             self.model = model.module
    905             self.model_wrapped = model  # will get further wrapped in DDP

~/.local/lib/python3.6/site-packages/transformers/integrations.py in init_deepspeed(trainer, num_training_steps)
    416         model=model,
    417         model_parameters=model_parameters,
--> 418         config_params=config,
    419     )
    420 

~/.local/lib/python3.6/site-packages/deepspeed/__init__.py in initialize(args, model, optimizer, model_parameters, training_data, lr_scheduler, mpu, dist_init_required, collate_fn, config_params)
    123                                  dist_init_required=dist_init_required,
    124                                  collate_fn=collate_fn,
--> 125                                  config_params=config_params)
    126     else:
    127         assert mpu is None, "mpu must be None with pipeline parallelism"

~/.local/lib/python3.6/site-packages/deepspeed/runtime/engine.py in __init__(self, args, model, optimizer, model_parameters, training_data, lr_scheduler, mpu, dist_init_required, collate_fn, config_params, dont_change_device)
    181         self.lr_scheduler = None
    182         if model_parameters or optimizer:
--> 183             self._configure_optimizer(optimizer, model_parameters)
    184             self._configure_lr_scheduler(lr_scheduler)
    185             self._report_progress(0)

~/.local/lib/python3.6/site-packages/deepspeed/runtime/engine.py in _configure_optimizer(self, client_optimizer, model_parameters)
    596                 logger.info('Using client Optimizer as basic optimizer')
    597         else:
--> 598             basic_optimizer = self._configure_basic_optimizer(model_parameters)
    599             if self.global_rank == 0:
    600                 logger.info(

~/.local/lib/python3.6/site-packages/deepspeed/runtime/engine.py in _configure_basic_optimizer(self, model_parameters)
    670                     optimizer = FusedAdam(model_parameters,
    671                                           **optimizer_parameters,
--> 672                                           adam_w_mode=effective_adam_w_mode)
    673 
    674         elif self.optimizer_name() == LAMB_OPTIMIZER:

~/.local/lib/python3.6/site-packages/deepspeed/ops/adam/fused_adam.py in __init__(self, params, lr, bias_correction, betas, eps, adam_w_mode, weight_decay, amsgrad, set_grad_none)
     70         self.set_grad_none = set_grad_none
     71 
---> 72         fused_adam_cuda = FusedAdamBuilder().load()
     73         # Skip buffer
     74         self._dummy_overflow_buf = torch.cuda.IntTensor([0])

~/.local/lib/python3.6/site-packages/deepspeed/ops/op_builder/builder.py in load(self, verbose)
    213             return importlib.import_module(self.absolute_name())
    214         else:
--> 215             return self.jit_load(verbose)
    216 
    217     def jit_load(self, verbose=True):

~/.local/lib/python3.6/site-packages/deepspeed/ops/op_builder/builder.py in jit_load(self, verbose)
    250             extra_cuda_cflags=self.nvcc_args(),
    251             extra_ldflags=self.extra_ldflags(),
--> 252             verbose=verbose)
    253         build_duration = time.time() - start_build
    254         if verbose:

~/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py in load(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
   1089         is_python_module,
   1090         is_standalone,
-> 1091         keep_intermediates=keep_intermediates)
   1092 
   1093 

~/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py in _jit_compile(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
   1300                         verbose=verbose,
   1301                         with_cuda=with_cuda,
-> 1302                         is_standalone=is_standalone)
   1303             finally:
   1304                 baton.release()

~/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py in _write_ninja_file_and_build_library(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_standalone)
   1405         build_directory,
   1406         verbose,
-> 1407         error_prefix=f"Error building extension '{name}'")
   1408 
   1409 

~/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py in _run_ninja_build(build_directory, verbose, error_prefix)
   1681         if hasattr(error, 'output') and error.output:  # type: ignore
   1682             message += f": {error.output.decode()}"  # type: ignore
-> 1683         raise RuntimeError(message) from e
   1684 
   1685 

RuntimeError: Error building extension 'fused_adam'

Versions that I'm using are -

Collecting environment information...
PyTorch version: 1.8.0+cu101
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: 
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB

Nvidia driver version: 450.51.06
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] kubeflow-pytorchjob==0.1.3
[pip3] numpy==1.18.5
[pip3] torch==1.8.0+cu101
[pip3] torchvision==0.8.1
[conda] Could not collect

transformers==4.4.2
DeepSpeed==0.3.13
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

But I was able to run DeepSpeed-0.3.10 with HuggingFace-4.3.2 and Torch-1.7.1+cu101 without any issue.

Plz suggest how to proceed further..

[saichandrapandraju]

Here is the config file that I'm using for DeepSpeed -

{
    "fp16": {
        "enabled": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "hysteresis": 2,
        "min_loss_scale": 1
    },

   "zero_optimization": {
       "stage": 2,
       "allgather_partitions": true,
       "allgather_bucket_size": 2e8,
       "overlap_comm": true,
       "reduce_scatter": true,
       "reduce_bucket_size": 2e8,
       "contiguous_gradients": true,
       "cpu_offload": false
   },

   "optimizer": {
     "type": "AdamW",
     "params": {
       "lr": 3e-5,
       "betas": [
         0.8,
         0.999
       ],
       "eps": 1e-8,
       "weight_decay": 3e-7
     }
   },

   "scheduler": {
     "type": "WarmupLR",
     "params": {
       "warmup_min_lr": 0,
       "warmup_max_lr": 3e-5,
       "warmup_num_steps": 500
     }
   },

    "steps_per_print": 2000,
    "wall_clock_breakdown": false
}

When I use "cpu_offload": true, getting error as - RuntimeError: Error building extension 'cpu_adam'.
Below is the full stacktrace -

[2021-03-23 07:21:31,906] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13, git-hash=unknown, git-branch=unknown
[2021-03-23 07:21:31,929] [INFO] [engine.py:77:_initialize_parameter_parallel_groups] data_parallel_size: 1, parameter_parallel_size: 1
Using /home/jovyan/.cache/torch_extensions as PyTorch extensions root...
Creating extension directory /home/jovyan/.cache/torch_extensions/cpu_adam...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/jovyan/.cache/torch_extensions/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
~/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py in _run_ninja_build(build_directory, verbose, error_prefix)
   1672             check=True,
-> 1673             env=env)
   1674     except subprocess.CalledProcessError as e:

/usr/lib/python3.6/subprocess.py in run(input, timeout, check, *popenargs, **kwargs)
    437             raise CalledProcessError(retcode, process.args,
--> 438                                      output=stdout, stderr=stderr)
    439     return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
<ipython-input-24-3435b262f1ae> in <module>
----> 1 trainer.train()

~/.local/lib/python3.6/site-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, **kwargs)
    901         delay_optimizer_creation = self.sharded_ddp is not None and self.sharded_ddp != ShardedDDPOption.SIMPLE
    902         if self.args.deepspeed:
--> 903             model, optimizer, lr_scheduler = init_deepspeed(self, num_training_steps=max_steps)
    904             self.model = model.module
    905             self.model_wrapped = model  # will get further wrapped in DDP

~/.local/lib/python3.6/site-packages/transformers/integrations.py in init_deepspeed(trainer, num_training_steps)
    416         model=model,
    417         model_parameters=model_parameters,
--> 418         config_params=config,
    419     )
    420 

~/.local/lib/python3.6/site-packages/deepspeed/__init__.py in initialize(args, model, optimizer, model_parameters, training_data, lr_scheduler, mpu, dist_init_required, collate_fn, config_params)
    123                                  dist_init_required=dist_init_required,
    124                                  collate_fn=collate_fn,
--> 125                                  config_params=config_params)
    126     else:
    127         assert mpu is None, "mpu must be None with pipeline parallelism"

~/.local/lib/python3.6/site-packages/deepspeed/runtime/engine.py in __init__(self, args, model, optimizer, model_parameters, training_data, lr_scheduler, mpu, dist_init_required, collate_fn, config_params, dont_change_device)
    181         self.lr_scheduler = None
    182         if model_parameters or optimizer:
--> 183             self._configure_optimizer(optimizer, model_parameters)
    184             self._configure_lr_scheduler(lr_scheduler)
    185             self._report_progress(0)

~/.local/lib/python3.6/site-packages/deepspeed/runtime/engine.py in _configure_optimizer(self, client_optimizer, model_parameters)
    596                 logger.info('Using client Optimizer as basic optimizer')
    597         else:
--> 598             basic_optimizer = self._configure_basic_optimizer(model_parameters)
    599             if self.global_rank == 0:
    600                 logger.info(

~/.local/lib/python3.6/site-packages/deepspeed/runtime/engine.py in _configure_basic_optimizer(self, model_parameters)
    665                     optimizer = DeepSpeedCPUAdam(model_parameters,
    666                                                  **optimizer_parameters,
--> 667                                                  adamw_mode=effective_adam_w_mode)
    668                 else:
    669                     from deepspeed.ops.adam import FusedAdam

~/.local/lib/python3.6/site-packages/deepspeed/ops/adam/cpu_adam.py in __init__(self, model_params, lr, bias_correction, betas, eps, weight_decay, amsgrad, adamw_mode)
     76         DeepSpeedCPUAdam.optimizer_id = DeepSpeedCPUAdam.optimizer_id + 1
     77         self.adam_w_mode = adamw_mode
---> 78         self.ds_opt_adam = CPUAdamBuilder().load()
     79 
     80         self.ds_opt_adam.create_adam(self.opt_id,

~/.local/lib/python3.6/site-packages/deepspeed/ops/op_builder/builder.py in load(self, verbose)
    213             return importlib.import_module(self.absolute_name())
    214         else:
--> 215             return self.jit_load(verbose)
    216 
    217     def jit_load(self, verbose=True):

~/.local/lib/python3.6/site-packages/deepspeed/ops/op_builder/builder.py in jit_load(self, verbose)
    250             extra_cuda_cflags=self.nvcc_args(),
    251             extra_ldflags=self.extra_ldflags(),
--> 252             verbose=verbose)
    253         build_duration = time.time() - start_build
    254         if verbose:

~/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py in load(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
   1089         is_python_module,
   1090         is_standalone,
-> 1091         keep_intermediates=keep_intermediates)
   1092 
   1093 

~/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py in _jit_compile(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
   1300                         verbose=verbose,
   1301                         with_cuda=with_cuda,
-> 1302                         is_standalone=is_standalone)
   1303             finally:
   1304                 baton.release()

~/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py in _write_ninja_file_and_build_library(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_standalone)
   1405         build_directory,
   1406         verbose,
-> 1407         error_prefix=f"Error building extension '{name}'")
   1408 
   1409 

~/.local/lib/python3.6/site-packages/torch/utils/cpp_extension.py in _run_ninja_build(build_directory, verbose, error_prefix)
   1681         if hasattr(error, 'output') and error.output:  # type: ignore
   1682             message += f": {error.output.decode()}"  # type: ignore
-> 1683         raise RuntimeError(message) from e
   1684 
   1685 

RuntimeError: Error building extension 'cpu_adam'

[stas00]

I created a colab notebook that took quite a lot of trial and error to figure out the right versions of everything to make DeepSpeed compile.

As you can see in the notebook I'm using torch==1.7.1+cu110 - have you tried running my notebook? It worked a week ago.

I will try to keep it up-to-date as colab changes its setup so ping me if it stops working (or file an issue) and requires a tune up.

I documented the critical components to successfully building deepspeed here.

[saichandrapandraju]

Hi @stas00 ,

I ran your notebook in colab and it gave this error -

[1/3] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/usr/local/lib/python3.7/dist-packages/deepspeed/ops/csrc/includes -I/usr/local/cuda/include -isystem /usr/local/lib/python3.7/dist-packages/torch/include -isystem /usr/local/lib/python3.7/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.7/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.7/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_75,code=sm_75 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_75,code=compute_75 -c /usr/local/lib/python3.7/dist-packages/deepspeed/ops/csrc/adam/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o 
[2/3] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/usr/local/lib/python3.7/dist-packages/deepspeed/ops/csrc/includes -I/usr/local/cuda/include -isystem /usr/local/lib/python3.7/dist-packages/torch/include -isystem /usr/local/lib/python3.7/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.7/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.7/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++14 -L/usr/local/cuda/lib64 -lcudart -lcublas -g -Wno-reorder -march=native -fopenmp -D__AVX256__ -c /usr/local/lib/python3.7/dist-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o 
[3/3] c++ cpu_adam.o custom_cuda_kernel.cuda.o -shared -L/usr/local/lib/python3.7/dist-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o cpu_adam.so
Loading extension module cpu_adam...
Time to load cpu_adam op: 26.246264457702637 seconds
Killing subprocess 468
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/deepspeed/launcher/launch.py", line 171, in <module>
    main()
  File "/usr/local/lib/python3.7/dist-packages/deepspeed/launcher/launch.py", line 161, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/usr/local/lib/python3.7/dist-packages/deepspeed/launcher/launch.py", line 139, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-u', 'examples/seq2seq/run_seq2seq.py', '--local_rank=0', '--model_name_or_path', 'google/mt5-small', '--output_dir', 'output_dir', '--adam_eps', '1e-06', '--evaluation_strategy=steps', '--do_train', '--label_smoothing', '0.1', '--learning_rate', '3e-5', '--logging_first_step', '--logging_steps', '1000', '--max_source_length', '128', '--max_target_length', '128', '--num_train_epochs', '1', '--overwrite_output_dir', '--per_device_train_batch_size', '16', '--predict_with_generate', '--sortish_sampler', '--val_max_target_length', '128', '--warmup_steps', '500', '--max_train_samples', '2000', '--max_val_samples', '500', '--task', 'translation_en_to_ro', '--dataset_name', 'wmt16', '--dataset_config', 'ro-en', '--source_prefix', 'translate English to Romanian: ', '--deepspeed', 'ds_config.json', '--fp16']' died with <Signals.SIGKILL: 9>.

Here is the full notebook with outputs.

All the critical components that you mentioned for building deepspeed are not valid in my case as I'm using system wide cuda version while installing torch. Also I don't have multiple cuda versions in my system and using gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

[stas00]

This is odd, since I have just re-run my notebook on the free version of colab and it didn't have any problems.

So you have may have noticed you made a progress, since you managed to now build the deepspeed extensions using this notebook. But then something killed the process immediately after it built the extension. so now you have a correct combination of the packages.

Try to re-run that last cell again - since the extension is now built and cached (that is if you're in the same session, if not start a new and re-run this cell second time if it dies again the first time).

In theory everybody gets mostly the same environment, but perhaps it's not so. Could you monitor that you disk space and RAM are not at 100% - perhaps the watchdog kills the process when resources are exhausted?

I'm curious what happens if you run the training cell the 2nd time.

[saichandrapandraju]

You're absolutely correct @stas00

RAM is reaching 100% and process is getting killed. I ran 2nd time by changing cpu_offload to false and training completed successfully. But I'm not sure why the same didn't happen for you in colab.

Also, I'm thinking how is this setup related to my issue. Are you suggesting me to upgrade my system-wide cuda to 11?

[stas00]

RAM is reaching 100% and process is getting killed. I ran 2nd time by changing cpu_offload to false and training completed successfully. But I'm not sure why the same didn't happen for you in colab.

Glad you figured it out!

We don't know if you get the same environment as I do. Actually it looks pretty random. I just tried 2 different notebooks and in the deepspeed one it gave me 25GB RAM and in another one only 12GB!

Run a cell with:

! free -h

I will add this to the notebook with a note, so others will know.

Also, I'm thinking how is this setup related to my issue. Are you suggesting me to upgrade my system-wide cuda to 11?

Not at all. You just need to have the same cuda as your pytorch was built with, so just install a pytorch build that matches your system-wide cuda and make sure you got PATH and LD_LIBRARY_PATH set correctly. It's all documented here:

https://huggingface.co/transformers/main_classes/trainer.html#installation-notes

[stas00]

You can probably close this issue now.

[saichandrapandraju]

Hi @stas00
But I'm still facing the same issue after making following changes suggested in HF's installation-notes -

previously my paths were -

!which nvcc
/usr/local/cuda/bin/nvcc

!echo $LD_LIBRARY_PATH
!echo $PATH
/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
/home/jovyan/.local/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

and now they are modified as -

!echo $LD_LIBRARY_PATH
!echo $PATH
/usr/local/cuda-10.1/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
/usr/local/cuda-10.1/bin:/home/jovyan/.local/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

!which nvcc
/usr/local/cuda-10.1/bin/nvcc

I have my cuda-10.1 in /usr/local/, so above paths are correct.
Here is the ds_report -

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires one of the following commands '['llvm-config', 'llvm-config-9']', but it does not exist!
 [WARNING]  sparse_attn requires the 'cmake' command, but it does not exist!
sparse_attn ............ [NO] ....... [NO]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/jovyan/.local/lib/python3.6/site-packages/torch']
torch version .................... 1.8.0+cu101
torch cuda version ............... 10.1
nvcc version ..................... 10.1
deepspeed install path ........... ['/home/jovyan/.local/lib/python3.6/site-packages/deepspeed']
deepspeed info ................... 0.3.13, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.8, cuda 10.1

Also tried this from similar issue but didn't work.

[stas00]

Oh, so it's not colab that you're trying to get it to work on. OK!

But I was able to run DeepSpeed-0.3.10 with HuggingFace-4.3.2 and Torch-1.7.1+cu101 without any issue.

Can you build it by downgrading to torch-1.7.1+cu101? Just to validate that deepspeed master is not at fault since you had it working with 0.3.10 - but as you see you changed the pytorch version as well.

Where did you find torch-1.8.0-cu101?

I can see here only 10.2 or 11.1 at https://pytorch.org/get-started/locally/

Alternatively if your nvidia driver supports it move to 11.1, you also get a better cudnn along with newer drivers/cuda, which you want to upgrade too then. I use cuda-11.1 at the moment and it works well.

If you want to save the hassle of upgrading to 11.1, and keep 10.1, I'd do pre-building from the source, since it'd help you identify any problems easier. See the details here:
https://www.deepspeed.ai/tutorials/advanced-install/#pre-install-deepspeed-ops
This is the approach that I use most of the time on machines where building the extension at run-time proves to be problematic.

My build script is:

#!/bin/bash

rm -rf build
time TORCH_CUDA_ARCH_LIST="6.1;8.6" DS_BUILD_OPS=1 pip install -e . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check 2>&1 | tee build.log

you just need to adjust the arch list to match your hardware. and may be -j to match how many parallel makes you'd like to run. And it does the develop install. You can remove -e if you want to.

[saichandrapandraju]

Hi @stas00 ,

Can you build it by downgrading to torch-1.7.1+cu101? Just to validate that deepspeed master is not at fault since you had it working with 0.3.10 - but as you see you changed the pytorch version as well.

Tried this and still facing same issue.

Where did you find torch-1.8.0-cu101?

I downloaded cu101/torch-1.8.0%2Bcu101-cp36-cp36m-linux_x86_64.whl from https://download.pytorch.org/whl/torch_stable.html

If you want to save the hassle of upgrading to 11.1, and keep 10.1, I'd do pre-building from the source, since it'd help you identify any problems easier.

If this downloads and installs modules from external sources, my VM won't have open internet access and it has to go through my company's firewall. If this (downloading from external sources) is the case, I may not pre-build from source.
Plz confirm whether it does collect necessary things from external sources.

Alternatively if your nvidia driver supports it move to 11.1, you also get a better cudnn along with newer drivers/cuda, which you want to upgrade too then. I use cuda-11.1 at the moment and it works well.

I want this to be last option as it's not in my control and have to contact other team to upgrade.

Meanwhile, I tried prebuilding in colab with diff. combinations and all those worked fine and you can find detailed o/p's here

[stas00]

If you want to save the hassle of upgrading to 11.1, and keep 10.1, I'd do pre-building from the source, since it'd help you identify any problems easier.

If this downloads and installs modules from external sources, my VM won't have open internet access and it has to go through my company's firewall. If this (downloading from external sources) is the case, I may not pre-build from source.
Plz confirm whether it does collect necessary things from external sources.

Meanwhile, I tried prebuilding in colab with diff. combinations and all those worked fine and you can find detailed o/p's here

OK, so since you prebuilt from source on colab (thank you for sharing the outcomes), you now know what's involved. It'll install dependencies just like when you don't pre-build from source. So if you are able to do pip install deepspeed on your setup you can also do the same here. i.e. preinstall all the dependencies when you have the network just like you'd do normally.

Here is yet another approach to consider. Build a binary wheel on whatever normal machine where you have a similar cuda setup:

git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
TORCH_CUDA_ARCH_LIST="6.1;8.6" DS_BUILD_OPS=1 python setup.py build_ext -j8 bdist_wheel

adjust TORCH_CUDA_ARCH_LIST for the required archs on the target machine.

Now you have dist/deepspeed-0.3.13+8cd046f-cp38-cp38-linux_x86_64.whl (will be a different name depending on the build).

Now you can install it on your VM and you don't need to build anything at run time, you just do:

pip install deepspeed-0.3.13+8cd046f-cp38-cp38-linux_x86_64.whl

I presume you will already have the other dependencies installed since you already did that for pip install deepspeed.

I wonder if DeepSpeed should document this approach on their advanced install page.

[saichandrapandraju]

Thanks a lot @stas00

Finally it worked. As colab is having python-3.7, I replicated what you've said in my AWS EC2 instance where I had many CUDAs including 10.1 and 11.1.

Reiterating the steps I followed so that it can help someone with similar issues -

1. Created a conda environment with python-3.6.9 (because my target machine where I want to run DeepSpeed is having 3.6.9).

2. Changed PATH and LD_LIBRARY_PATH to point to CUDA-10.1 ( again because of my target machine) as suggested in HF's installation notes here. Below are the commands -

export PATH=/usr/local/cuda-10.1/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.1/lib64:$LD_LIBRARY_PATH

3. Install PyTorch (should be same version as in target machine). I installed with below command -

pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

verify torch versions with python -m torch.utils.collect_env

4. Execute below commands to pre-build DeepSpeed -

git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
time DS_BUILD_OPS=1 pip install . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check 2>&1 | tee build.log

Check whether compatible op's were installed or not with ds_report

5. Extract whl file for this DeepSpeed using below command-

rm -rf build
DS_BUILD_OPS=1 python setup.py build_ext -j8 bdist_wheel

6. Take whl from dist/ and install in target machine using pip install deepspeed-0.3.13+7fcc891-cp36-cp36m-linux_x86_64.whl