{ai}[foss/2023a] DeepSpeed v0.14.5, CUTLASS v3.5.1, DLPACK v0.8 w/ CUDA 12.1.1 #21438

VRehnberg · 2024-09-18T14:49:45Z

(created using eb --new-pr)
Requires:

Add DeepSpeed easyblock easybuild-easyblocks#3450

…tches: DeepSpeed-0.14.5_pic-compile.patch, DeepSpeed-0.14.2_no-ninja-dep.patch

VRehnberg · 2024-09-18T14:52:48Z

Will probably want to change triton used to #21318

VRehnberg · 2024-09-24T14:35:51Z

DLPack
Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis1-16 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 560.35.03, Python 3.6.8
See https://gist.github.com/VRehnberg/9e271e35ebb3964a7749770e14eb3c42 for a full test report.

VRehnberg · 2024-09-24T15:50:01Z

CUTLASS
Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis1-15 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 560.35.03, Python 3.6.8
See https://gist.github.com/VRehnberg/01f85607114e140d887898842effea9f for a full test report.

VRehnberg · 2024-09-24T15:52:56Z

DeepSpeed
Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis1-06 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 560.35.03, Python 3.6.8
See https://gist.github.com/VRehnberg/5a708f22771e1d099be9f64a37d3eccb for a full test report.

Typo in test command after linebreaks.

VRehnberg · 2024-09-25T08:46:06Z

Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis1-12 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 560.35.03, Python 3.6.8
See https://gist.github.com/VRehnberg/d519a58b098e8dcc932f3eaba43627cd for a full test report.

VRehnberg · 2024-09-25T09:23:45Z

Latest failures have in common that they use the multi-node launcher. Unsure if it's only the test that's broken or something else.

As an example of a failing command:

pdsh -S -f 1024 -w localhost export NCCL_IB_HCA=^mlx5_1; export PYTHONNOUSERSITE=1; export UCX_MODULE_DIR=[...]; export PYTHONPATH=[...]; /apps/Test2/software/Python/3.11.3-GCCcore-12.3.0/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --node_rank=%n --master_addr=127.0.0.1 --master_port=29500 /cephyr/NOBACKUP/priv/c3-staff/eb-tmp/eb-5t779o2m/pytest-of-c3-builder/pytest-0/test_user_args_True_I_m_going_0/user_arg_test.py --prompt "I\'m going to tell them \\"DeepSpeed is the best\\""\n'.decode

so probably is just because LD_LIBRARY_PATH is not also exported. Looking for where this command is built...

It seems like an add_export for that is missing here https://github.com/microsoft/DeepSpeed/blob/v0.14.5/deepspeed/launcher/runner.py#L564-L578

So should just be to add it to https://github.com/microsoft/DeepSpeed/blob/v0.14.5/deepspeed/launcher/runner.py#L34

Will try that.

VRehnberg · 2024-09-25T10:22:49Z

Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis1-02 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 560.35.03, Python 3.6.8
See https://gist.github.com/VRehnberg/03bc8fc4dfc955cd4a20c9c4f68a7fc8 for a full test report.

VRehnberg · 2024-09-26T14:00:52Z

Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis1-02 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 560.35.03, Python 3.6.8
See https://gist.github.com/VRehnberg/7bcc5750617d9cc507edc761af91d199 for a full test report.

VRehnberg · 2024-09-30T07:50:48Z

Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis1-01 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz, 1 x NVIDIA Tesla V100-SXM2-32GB, 560.35.03, Python 3.6.8
See https://gist.github.com/VRehnberg/1afe632871f634e216b2ee080529681f for a full test report.

VRehnberg · 2024-10-01T08:00:05Z

Compared the environment variables before and after loading the DeepSpeed module. Will probably update the pdsh-env-vars patch with those.

So remaining

Update pdsh-env-vars patch
Check deepspeed.env_report and see if everything was built alright
Run on some more hardware (especially Ampere or newer GPUs)
Get this reviewed by a maintainer

VRehnberg · 2024-10-01T14:51:26Z

Env report seems to indicate that pre-built ops are not being picked up properly:

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
async_io ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
 [WARNING]  NVIDIA Inference is only supported on Ampere and newer architectures
 [WARNING]  FP Quantizer is using an untested triton version (2.1.0), only 2.3.0 and 2.3.1 are known to be compatible with these kernels
fp_quantizer ........... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1
 [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/apps/Test2/software/PyTorch/2.1.2-foss-2023a-CUDA-12.1.1/lib/python3.11/site-packages/torch']
torch version .................... 2.1.2
deepspeed install path ........... ['/apps/Test2/software/DeepSpeed/0.14.5-foss-2023a-CUDA-12.1.1/lib/python3.11/site-packages/deepspeed']
deepspeed info ................... 0.14.5+unknown, unknown, unknown
torch cuda version ............... 12.1
torch hip version ................ None
nvcc version ..................... 12.1
deepspeed wheel compiled w. ...... torch 2.1, cuda 12.1
shared memory (/dev/shm) size .... 377.03 GB

VRehnberg · 2024-10-07T14:06:07Z

Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
alvis6-11 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A40, 560.35.03, Python 3.6.8
See https://gist.github.com/VRehnberg/d35d54e5f5c7926d09cd494d1addf826 for a full test report.

VRehnberg · 2024-10-08T10:20:01Z

Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
FAILED
Build succeeded for 4 out of 5 (3 easyconfigs in total)
alvis3-36 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/f4f23e0b89d8a5b5304d352fe0f04964 for a full test report.

VRehnberg · 2024-10-08T12:51:39Z

Three failing files in latest 4xA100 run:

tests/unit/inference/quantization/test_intX_quantization.py
tests/unit/ops/aio/test_aio.py
tests/unit/runtime/zero/test_nvme_checkpointing.py

Can't reproduce them when skipping test step and running them manually (though first one is unclear as it fails for another reason).

VRehnberg · 2024-10-14T12:02:04Z

Seems mostly fine now, rerunning build on Ampere GPU.

VRehnberg · 2024-10-15T09:36:20Z

Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
SUCCESS
Build succeeded for 1 out of 1 (1 easyconfigs in total)
alvis3-26 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 1 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/ce9fbac4ad17af130fbde5c5069bdd41 for a full test report.

VRehnberg · 2024-10-18T14:53:33Z

easybuild/easyconfigs/d/DeepSpeed/DeepSpeed-0.14.5_pdsh-env-vars.patch

+EXPORT_ENVS += [ # Extra based on what's added by module load DeepSpeed
+    'LD_LIBRARY_PATH', 'PATH', 'EB', 'TRITON', 'CUDA',  # important
+    'ACLOCAL', 'CMAKE', 'CPATH', 'LIBRARY_PATH', 'MPL', 'NCCL',
+    'PKG_CONFIG_PATH', 'XDG_DATA_DIRS',
+]


I'm on the fence on if all of these should be included or not. The alternative is to reduce these and add them case by case with a .deepspeed_env file. https://www.deepspeed.ai/getting-started/#multi-node-environment-variables

VRehnberg · 2024-10-23T07:53:14Z

One issue with virtualenv --system-site-package. Not unique to this ec, but still something that would be good to fix if I can find a solution.

VRehnberg · 2024-10-29T13:54:02Z

One issue with virtualenv --system-site-package. Not unique to this ec, but still something that would be good to fix if I can find a solution.

Hopefully solved now in easybuilders/easybuild-easyblocks#3450

VRehnberg · 2024-10-29T15:34:56Z

Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis4-17 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/4ec5a1dc6936d73cd0d3d3fd3a20653b for a full test report.

VRehnberg · 2024-11-01T17:14:03Z

Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis4-23 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/baa11497e89a38a9865cdc5493516019 for a full test report.

VRehnberg · 2024-11-04T07:24:35Z

Last failure was from two quantized nvme offload tests. Either error is because nvme_path is not actually pointing to NVME on that machine or it's this microsoft/DeepSpeed#5124. Stack trace was unhelpful, will try to reproduce manually.

VRehnberg · 2024-11-04T16:44:12Z

Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis4-27 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 4 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/d16a7398d9415427fdb92fc835e6871f for a full test report.

VRehnberg · 2024-11-05T10:33:00Z

FAILED

Meh, thought I was done. Why have new errors popped up :(

Missing import mpi4py
- Sure reasonable dependency easily fixed.
ImportError: dynamic module does not define module export function (PyInit_transformer_op)
- I've seen this before, thought it was solved now when ops built correctly apparently not. No idea what's causing it.
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
- get_accelerator() must have incorrectly output "cpu" instead of "cuda".

github-actions · 2024-11-08T14:10:04Z

Updated software `CUTLASS-3.5.1-foss-2023a-CUDA-12.1.1.eb`

Diff against CUTLASS-3.4.0-foss-2023a-CUDA-12.1.1.eb

easybuild/easyconfigs/c/CUTLASS/CUTLASS-3.4.0-foss-2023a-CUDA-12.1.1.eb

diff --git a/easybuild/easyconfigs/c/CUTLASS/CUTLASS-3.4.0-foss-2023a-CUDA-12.1.1.eb b/easybuild/easyconfigs/c/CUTLASS/CUTLASS-3.5.1-foss-2023a-CUDA-12.1.1.eb
index 3322e6b4b5..0d0caa2527 100644
--- a/easybuild/easyconfigs/c/CUTLASS/CUTLASS-3.4.0-foss-2023a-CUDA-12.1.1.eb
+++ b/easybuild/easyconfigs/c/CUTLASS/CUTLASS-3.5.1-foss-2023a-CUDA-12.1.1.eb
@@ -1,7 +1,7 @@
 easyblock = 'CMakeMake'
 
 name = 'CUTLASS'
-version = '3.4.0'
+version = '3.5.1'
 versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://github.com/NVIDIA/cutlass'
@@ -21,8 +21,13 @@ toolchain = {'name': 'foss', 'version': '2023a'}
 
 github_account = 'NVIDIA'
 source_urls = [GITHUB_LOWER_SOURCE]
+patches = ['CUTLASS-3.5.1_install_tools.patch']
+
 sources = ['v%(version)s.tar.gz']
-checksums = ['49f4b854acc2a520126ceefe4f701cfe8c2b039045873e311b1f10a8ca5d5de1']
+checksums = [
+    {'v%(version)s.tar.gz': '20b7247cda2d257cbf8ba59ba3ca40a9211c4da61a9c9913e32b33a2c5883a36'},
+    {'CUTLASS-3.5.1_install_tools.patch': '18fa5361b15848d98435b8b08bd921130718b963ca4ad47fa0db96fbe815e509'},
+]
 
 builddependencies = [
     ('CMake', '3.26.3'),
@@ -38,12 +43,13 @@ _copts = [
     '-DCUTLASS_NVCC_ARCHS="%(cuda_cc_cmake)s"',
     '-DCUTLASS_ENABLE_CUBLAS=1',
     '-DCUTLASS_ENABLE_CUDNN=1',
+    '-DCUTLASS_ENABLE_TOOLS=1',
 ]
 configopts = ' '.join(_copts)
 
 sanity_check_paths = {
     'files': ['include/cutlass/cutlass.h', 'lib/libcutlass.%s' % SHLIB_EXT],
-    'dirs': ['lib/cmake'],
+    'dirs': ['lib/cmake', 'tools/util/include'],
 }
 
 moduleclass = 'lib'

Diff against CUTLASS-2.11.0-foss-2022a-CUDA-11.7.0.eb

easybuild/easyconfigs/c/CUTLASS/CUTLASS-2.11.0-foss-2022a-CUDA-11.7.0.eb

diff --git a/easybuild/easyconfigs/c/CUTLASS/CUTLASS-2.11.0-foss-2022a-CUDA-11.7.0.eb b/easybuild/easyconfigs/c/CUTLASS/CUTLASS-3.5.1-foss-2023a-CUDA-12.1.1.eb
index f534333fec..0d0caa2527 100644
--- a/easybuild/easyconfigs/c/CUTLASS/CUTLASS-2.11.0-foss-2022a-CUDA-11.7.0.eb
+++ b/easybuild/easyconfigs/c/CUTLASS/CUTLASS-3.5.1-foss-2023a-CUDA-12.1.1.eb
@@ -1,7 +1,7 @@
 easyblock = 'CMakeMake'
 
 name = 'CUTLASS'
-version = '2.11.0'
+version = '3.5.1'
 versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://github.com/NVIDIA/cutlass'
@@ -17,33 +17,39 @@ tuned via custom tiling sizes, data types, and other algorithmic policy.
 The resulting flexibility simplifies their use as building blocks within
 custom kernels and applications."""
 
-toolchain = {'name': 'foss', 'version': '2022a'}
+toolchain = {'name': 'foss', 'version': '2023a'}
 
 github_account = 'NVIDIA'
 source_urls = [GITHUB_LOWER_SOURCE]
+patches = ['CUTLASS-3.5.1_install_tools.patch']
+
 sources = ['v%(version)s.tar.gz']
-checksums = ['b4394f1e080b63cfc54163069334096324c1262dfc66e67099880005d51b8af9']
+checksums = [
+    {'v%(version)s.tar.gz': '20b7247cda2d257cbf8ba59ba3ca40a9211c4da61a9c9913e32b33a2c5883a36'},
+    {'CUTLASS-3.5.1_install_tools.patch': '18fa5361b15848d98435b8b08bd921130718b963ca4ad47fa0db96fbe815e509'},
+]
 
 builddependencies = [
-    ('CMake', '3.24.3'),
-    ('Python', '3.10.4'),
+    ('CMake', '3.26.3'),
+    ('Python', '3.11.3'),
 ]
 
 dependencies = [
-    ('CUDA', '11.7.0', '', SYSTEM),
-    ('cuDNN', '8.4.1.50', versionsuffix, SYSTEM),
+    ('CUDA', '12.1.1', '', SYSTEM),
+    ('cuDNN', '8.9.2.26', versionsuffix, SYSTEM),
 ]
 
 _copts = [
     '-DCUTLASS_NVCC_ARCHS="%(cuda_cc_cmake)s"',
     '-DCUTLASS_ENABLE_CUBLAS=1',
     '-DCUTLASS_ENABLE_CUDNN=1',
+    '-DCUTLASS_ENABLE_TOOLS=1',
 ]
 configopts = ' '.join(_copts)
 
 sanity_check_paths = {
     'files': ['include/cutlass/cutlass.h', 'lib/libcutlass.%s' % SHLIB_EXT],
-    'dirs': ['lib/cmake'],
+    'dirs': ['lib/cmake', 'tools/util/include'],
 }
 
 moduleclass = 'lib'

Updated software `DLPack-0.8-GCC-12.3.0.eb`

Diff against DLPack-0.8-GCC-11.3.0.eb

easybuild/easyconfigs/d/DLPack/DLPack-0.8-GCC-11.3.0.eb

diff --git a/easybuild/easyconfigs/d/DLPack/DLPack-0.8-GCC-11.3.0.eb b/easybuild/easyconfigs/d/DLPack/DLPack-0.8-GCC-12.3.0.eb
index 97be9746c5..30c8128f84 100644
--- a/easybuild/easyconfigs/d/DLPack/DLPack-0.8-GCC-11.3.0.eb
+++ b/easybuild/easyconfigs/d/DLPack/DLPack-0.8-GCC-12.3.0.eb
@@ -7,7 +7,7 @@ homepage = 'https://dmlc.github.io/dlpack/latest/'
 description = """DLPack is a stable in-memory data structure for an ndarray
 system to interact with a variety of frameworks."""
 
-toolchain = {'name': 'GCC', 'version': '11.3.0'}
+toolchain = {'name': 'GCC', 'version': '12.3.0'}
 
 github_account = 'dmlc'
 source_urls = [GITHUB_LOWER_SOURCE]
@@ -15,7 +15,7 @@ sources = ['v%(version)s.tar.gz']
 checksums = ['cf965c26a5430ba4cc53d61963f288edddcd77443aa4c85ce722aaf1e2f29513']
 
 builddependencies = [
-    ('CMake', '3.24.3'),
+    ('CMake', '3.26.3'),
 ]
 
 sanity_check_paths = {

Diff against DLPack-0.3-GCC-10.3.0.eb

easybuild/easyconfigs/d/DLPack/DLPack-0.3-GCC-10.3.0.eb

diff --git a/easybuild/easyconfigs/d/DLPack/DLPack-0.3-GCC-10.3.0.eb b/easybuild/easyconfigs/d/DLPack/DLPack-0.8-GCC-12.3.0.eb
index c487f04f84..30c8128f84 100644
--- a/easybuild/easyconfigs/d/DLPack/DLPack-0.3-GCC-10.3.0.eb
+++ b/easybuild/easyconfigs/d/DLPack/DLPack-0.8-GCC-12.3.0.eb
@@ -1,21 +1,21 @@
 easyblock = 'CMakeMake'
 
 name = 'DLPack'
-version = '0.3'
+version = '0.8'
 
 homepage = 'https://dmlc.github.io/dlpack/latest/'
 description = """DLPack is a stable in-memory data structure for an ndarray
 system to interact with a variety of frameworks."""
 
-toolchain = {'name': 'GCC', 'version': '10.3.0'}
+toolchain = {'name': 'GCC', 'version': '12.3.0'}
 
 github_account = 'dmlc'
 source_urls = [GITHUB_LOWER_SOURCE]
 sources = ['v%(version)s.tar.gz']
-checksums = ['703149f5b39ead42cc734c03c7c4bd581fcad1c5a3939e7a4b5bc82f54c3c32a']
+checksums = ['cf965c26a5430ba4cc53d61963f288edddcd77443aa4c85ce722aaf1e2f29513']
 
 builddependencies = [
-    ('CMake', '3.20.1'),
+    ('CMake', '3.26.3'),
 ]
 
 sanity_check_paths = {

VRehnberg · 2024-11-08T16:38:35Z

Test report by @VRehnberg
Using easyblocks from PR(s) easybuilders/easybuild-easyblocks#3450
FAILED
Build succeeded for 0 out of 1 (1 easyconfigs in total)
alvis4-15 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz, 2 x NVIDIA NVIDIA A100-SXM4-40GB, 545.23.08, Python 3.6.8
See https://gist.github.com/VRehnberg/6c0b6af62d404244d15ec8f6d3d1dc33 for a full test report.

adding easyconfigs: DeepSpeed-0.14.5-foss-2023a-CUDA-12.1.1.eb and pa…

1613d53

…tches: DeepSpeed-0.14.5_pic-compile.patch, DeepSpeed-0.14.2_no-ninja-dep.patch

VRehnberg marked this pull request as draft September 18, 2024 14:52

VRehnberg added 2 commits September 23, 2024 14:13

Add Triton dependency

e81c5d2

Add easyconfigs: CUTLASS, DLPack

4070998

VRehnberg marked this pull request as ready for review September 23, 2024 14:24

VRehnberg changed the title ~~{ai}[foss/2023a] DeepSpeed v0.14.5 w/ CUDA 12.1.1~~ {ai}[foss/2023a] DeepSpeed v0.14.5, CUTLASS v3.5.0, DLPACK v0.8 w/ CUDA 12.1.1 Sep 23, 2024

Split long line for testing

0822a41

Fix whitespace in cmd after linebreaks

bf324ac

Add LD_LIBRARY_PATH and PATH to exported env vars

155c09d

VRehnberg added 2 commits September 26, 2024 13:00

Include more relevant env vars for pdsh

4d37513

Add mup dependencies

ee1f77b

VRehnberg added 2 commits September 27, 2024 07:28

Further update deps

6963a60

Fix style

891d556

VRehnberg added 2 commits October 1, 2024 08:08

Add all extra envvars from module

5b5081a

Move functionality between block and config

aaf9fbe

Prebuild ops and bump CUTLASS version

79c97ca

VRehnberg changed the title ~~{ai}[foss/2023a] DeepSpeed v0.14.5, CUTLASS v3.5.0, DLPACK v0.8 w/ CUDA 12.1.1~~ {ai}[foss/2023a] DeepSpeed v0.14.5, CUTLASS v3.5.1, DLPACK v0.8 w/ CUDA 12.1.1 Oct 10, 2024

VRehnberg commented Oct 18, 2024

View reviewed changes

VRehnberg mentioned this pull request Oct 24, 2024

[BUG] Deepspeed launcher not picking up virtualenv --system-site-packages microsoft/DeepSpeed#6664

Open

VRehnberg added 2 commits November 1, 2024 13:40

Add -Xcompiler to nvcc -fPIC patch

57d8582

Fix python shebangs

9b3e07b

Patch nvme offload test

06250f7

Readd DistributionTest nvme offload patch

9aba3e3

github-actions bot added new update labels Nov 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{ai}[foss/2023a] DeepSpeed v0.14.5, CUTLASS v3.5.1, DLPACK v0.8 w/ CUDA 12.1.1 #21438

{ai}[foss/2023a] DeepSpeed v0.14.5, CUTLASS v3.5.1, DLPACK v0.8 w/ CUDA 12.1.1 #21438

VRehnberg commented Sep 18, 2024 •

edited

Loading

VRehnberg commented Sep 18, 2024 •

edited

Loading

VRehnberg commented Sep 24, 2024 •

edited

Loading

VRehnberg commented Sep 24, 2024 •

edited

Loading

VRehnberg commented Sep 24, 2024 •

edited

Loading

VRehnberg commented Sep 25, 2024

VRehnberg commented Sep 25, 2024

VRehnberg commented Sep 25, 2024

VRehnberg commented Sep 26, 2024

VRehnberg commented Sep 30, 2024

VRehnberg commented Oct 1, 2024 •

edited

Loading

VRehnberg commented Oct 1, 2024

VRehnberg commented Oct 7, 2024

VRehnberg commented Oct 8, 2024

VRehnberg commented Oct 8, 2024

VRehnberg commented Oct 14, 2024

VRehnberg commented Oct 15, 2024

VRehnberg Oct 18, 2024

VRehnberg commented Oct 23, 2024

VRehnberg commented Oct 29, 2024

VRehnberg commented Oct 29, 2024

VRehnberg commented Nov 1, 2024

VRehnberg commented Nov 4, 2024

VRehnberg commented Nov 4, 2024

VRehnberg commented Nov 5, 2024

github-actions bot commented Nov 8, 2024

VRehnberg commented Nov 8, 2024

{ai}[foss/2023a] DeepSpeed v0.14.5, CUTLASS v3.5.1, DLPACK v0.8 w/ CUDA 12.1.1 #21438

Are you sure you want to change the base?

{ai}[foss/2023a] DeepSpeed v0.14.5, CUTLASS v3.5.1, DLPACK v0.8 w/ CUDA 12.1.1 #21438

Conversation

VRehnberg commented Sep 18, 2024 • edited Loading

VRehnberg commented Sep 18, 2024 • edited Loading

VRehnberg commented Sep 24, 2024 • edited Loading

VRehnberg commented Sep 24, 2024 • edited Loading

VRehnberg commented Sep 24, 2024 • edited Loading

VRehnberg commented Sep 25, 2024

VRehnberg commented Sep 25, 2024

VRehnberg commented Sep 25, 2024

VRehnberg commented Sep 26, 2024

VRehnberg commented Sep 30, 2024

VRehnberg commented Oct 1, 2024 • edited Loading

VRehnberg commented Oct 1, 2024

VRehnberg commented Oct 7, 2024

VRehnberg commented Oct 8, 2024

VRehnberg commented Oct 8, 2024

VRehnberg commented Oct 14, 2024

VRehnberg commented Oct 15, 2024

VRehnberg Oct 18, 2024

Choose a reason for hiding this comment

VRehnberg commented Oct 23, 2024

VRehnberg commented Oct 29, 2024

VRehnberg commented Oct 29, 2024

VRehnberg commented Nov 1, 2024

VRehnberg commented Nov 4, 2024

VRehnberg commented Nov 4, 2024

VRehnberg commented Nov 5, 2024

github-actions bot commented Nov 8, 2024

Updated software CUTLASS-3.5.1-foss-2023a-CUDA-12.1.1.eb

Updated software DLPack-0.8-GCC-12.3.0.eb

VRehnberg commented Nov 8, 2024

VRehnberg commented Sep 18, 2024 •

edited

Loading

VRehnberg commented Sep 18, 2024 •

edited

Loading

VRehnberg commented Sep 24, 2024 •

edited

Loading

VRehnberg commented Sep 24, 2024 •

edited

Loading

VRehnberg commented Sep 24, 2024 •

edited

Loading

VRehnberg commented Oct 1, 2024 •

edited

Loading

Updated software `CUTLASS-3.5.1-foss-2023a-CUDA-12.1.1.eb`

Updated software `DLPack-0.8-GCC-12.3.0.eb`