enhance GCC easyblock to add support for AMD GPU offloading #2578

nordmoen · 2021-09-15T11:37:43Z

(created using eb --new-pr)

nordmoen · 2021-09-15T11:44:21Z

To test these changes one needs a slightly modified GCCcore-11.2.0.eb:

easyblock = 'EB_GCC'

name = 'GCCcore'
version = '11.2.0'
# To build AMD GCN backend we need LLVM 9, hopefully can be upgraded to LLVM 12 in future
# https://gcc.gnu.org/wiki/Offloading#For_AMD_GCN:
local_llvm_version = '9.0.1'

homepage = 'https://gcc.gnu.org/'
description = """The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Java, and Ada,
 as well as libraries for these languages (libstdc++, libgcj,...)."""

toolchain = SYSTEM

source_urls = [
    'https://ftpmirror.gnu.org/gnu/gcc/gcc-%(version)s',  # GCC auto-resolving HTTP mirror
    'https://ftpmirror.gnu.org/gnu/gmp',  # idem for GMP
    'https://ftpmirror.gnu.org/gnu/mpfr',  # idem for MPFR
    'https://ftpmirror.gnu.org/gnu/mpc',  # idem for MPC
    'ftp://gcc.gnu.org/pub/gcc/infrastructure/',  # GCC dependencies
    'http://gcc.cybermirror.org/infrastructure/',  # HTTP mirror for GCC dependencies
    'http://isl.gforge.inria.fr/',  # original HTTP source for ISL
    'https://sourceware.org/pub/newlib/',  # for newlib
    'https://github.com/MentorEmbedded/nvptx-tools/archive',  # for nvptx-tools
    'https://github.com/llvm/llvm-project/releases/download/llvmorg-%s' % local_llvm_version,  # for LLVM project
]
sources = [
    'gcc-%(version)s.tar.gz',
    'gmp-6.2.1.tar.bz2',
    'mpfr-4.1.0.tar.bz2',
    'mpc-1.2.1.tar.gz',
    'isl-0.24.tar.bz2',
    'newlib-4.1.0.tar.gz',
    {'download_filename': 'd0524fb.tar.gz', 'filename': 'nvptx-tools-20210115.tar.gz'},
    'llvm-%s.src.tar.xz' % local_llvm_version,
    'lld-%s.src.tar.xz' % local_llvm_version,
]
patches = [
    'GCCcore-6.2.0-fix-find-isl.patch',
    'GCCcore-9.3.0_gmp-c99.patch',
    'GCCcore-9.x-11.x_fix-unsigned-fpe-traps.patch',
    ('llvm-9_include_limits_for_gcc11.patch', '../llvm-9.0.1.src/'),
]
checksums = [
    'f0837f1bf8244a5cc23bd96ff6366712a791cfae01df8e25b137698aca26efc1',  # gcc-11.2.0.tar.gz
    'eae9326beb4158c386e39a356818031bd28f3124cf915f8c5b1dc4c7a36b4d7c',  # gmp-6.2.1.tar.bz2
    'feced2d430dd5a97805fa289fed3fc8ff2b094c02d05287fd6133e7f1f0ec926',  # mpfr-4.1.0.tar.bz2
    '17503d2c395dfcf106b622dc142683c1199431d095367c6aacba6eec30340459',  # mpc-1.2.1.tar.gz
    'fcf78dd9656c10eb8cf9fbd5f59a0b6b01386205fe1934b3b287a0a1898145c0',  # isl-0.24.tar.bz2
    'f296e372f51324224d387cc116dc37a6bd397198756746f93a2b02e9a5d40154',  # newlib-4.1.0.tar.gz
    '466abe1cef9cf294318ecb3c221593356f7a9e1674be987d576bc70d833d84a2',  # nvptx-tools-20210115.tar.gz
    '00a1ee1f389f81e9979f3a640a01c431b3021de0d42278f6508391a2f0b81c9a',  # llvm-9.0.1.src.tar.xz
    '86262bad3e2fd784ba8c5e2158d7aa36f12b85f2515e95bc81d65d75bb9b0c82',  # lld-9.0.1.src.tar.xz
    '5ad909606d17d851c6ad629b4fddb6c1621844218b8d139fed18c502a7696c68',  # GCCcore-6.2.0-fix-find-isl.patch
    '0e135e1cc7cec701beea9d7d17a61bab34cfd496b4b555930016b98db99f922e',  # GCCcore-9.3.0_gmp-c99.patch
    '03a2e4aeda78d398edd680d6a1ba842b8ceb29c126ebb7fe2e3541ddfe4fbed4',  # GCCcore-9.x-11.x_fix-unsigned-fpe-traps.patch
    '50518969adc9dccd65536a6b60c7c7158af1ac6ad8ac499bef7e3c1b413866b1',  # llvm-9_include_limits_for_gcc11.patch
]

builddependencies = [
    ('M4', '1.4.19'),
    ('binutils', '2.37'),
]

languages = ['c', 'c++', 'fortran']

withisl = True
withnvptx = True
withamdgcn = True

# Perl is only required when building with NVPTX support
if withnvptx or withamdgcn:
    osdependencies = ['perl']
# When building offloading for AMD GCN we need CMake to build LLVM source
if withamdgcn:
    builddependencies += [('CMake', '3.18.4')]

moduleclass = 'compiler'

And the patch llvm-9_include_limits_for_gcc11.patch:

# In GCC 11 some headers were removed from the standard library. Some code
# which implicitly assumed that an include worked before is therefore no longer
# compiling. This is specially for "std::numeric_limits" which is the case for
# LLVM. Source: https://www.gnu.org/software/gcc/gcc-11/porting_to.html
--- llvm-9.0.1.src/utils/benchmark/src/benchmark_register.h.orig
+++ llvm-9.0.1.src/utils/benchmark/src/benchmark_register.h
@@ -1,6 +1,7 @@
 #ifndef BENCHMARK_REGISTER_H
 #define BENCHMARK_REGISTER_H
 
+#include <limits>
 #include <vector>
 
 #include "check.h"

easybuild/easyblocks/g/gcc.py

nordmoen · 2021-09-15T13:12:36Z

To test one can use the following simple program, device.c:

/* Simple test to ensure that devices are used and available through OpenMP
 * and OpenACC
 *
 * Copyright 2021 Jørgen Nordmoen
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * 	http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

#include <stdio.h>
#include <stdlib.h>
#ifdef _OPENMP
#include <omp.h>
#endif
#ifdef _OPENACC
#include <openacc.h>
#endif

int main () {
#ifdef _OPENMP
  printf ("\033[0;34mChecking OpenMP\n\033[0m");
  printf ("\tFound \033[0;35m%d\033[0m devices\n", omp_get_num_devices());
  if (omp_get_num_devices () > 0) {
    for (int i = 0; i < omp_get_num_devices (); i++) {
      omp_set_default_device(i);
      printf ("\tChecking device (\033[0;35m%d\033[0m) support: ", i);
      int on_host = 1;
      #pragma omp target map(from: on_host)
      {
        on_host = omp_is_initial_device();
      }
      printf (on_host == 0? "\033[0;32mDevice works\033[0m\n"
                          : "\033[0;31mDevice did not function\033[0m\n");
      }
  }
#else
  printf("\033[0;33mNot compiled with OpenMP support!\033[0m\n");
#endif
#ifdef _OPENACC
  printf("\033[0;34mChecking OpenACC:\033[0m\n");
  const int num_devices = acc_get_num_devices(acc_device_not_host);
  printf ("\tFound \033[0;35m%d\033[0m devices\n", num_devices);
  if (num_devices > 0) {
    for (int i = 0; i < num_devices; i++) {
      acc_set_device_num (i, acc_device_not_host);
      const acc_device_t type = acc_get_device_type();
      printf ("\tChecking device (\033[0;35m%d, %s - %s\033[0m) support: ",
              i,
              acc_get_property_string(i, type, acc_property_name),
              acc_get_property_string(i, type, acc_property_vendor));
      int on_device = 0;
      #pragma acc parallel copyout(on_device)
      {
        on_device = acc_on_device(type);
      }
      printf (on_device == 1? "\033[0;32mDevice works\033[0m\n"
                            : "\033[0;31mDevice did not function\033[0m\n");
      }
  }
#else
  printf("\033[0;33mNot compiled with OpenACC support!\033[0m\n");
#endif
  return EXIT_SUCCESS;
}

Compile with gcc -fopenacc -foffload=nvptx-none -o device device.c or for AMD gcc -fopenacc -foffload=amdgcn-amdhsa="-march=gfx908" -o device device.c

…hat is raised when required additional sources are missing

…onfiguring AMD GPU offload support in GCC easyblock

…e when configuring nvptx/AMD GCN target compiler

boegel

@nordmoen I stepped through this in detail, changes look good, but I made some minor code cleanup/enhancements on top which I'd like to get included, see nordmoen#2

I'm still testing those changes, but I think they shouldn't cause any trouble (famous last words...)

code cleanup for enhanced GCC easyblock which supports building with AMD GPU offload support

…s for optional dependencies for NVPTX and AMD GCN offload support from prepare_step to configure_step

easybuild/easyblocks/g/gcc.py

avoid breaking --module-only for GCC(core) by moving check for sources for optional dependencies for NVPTX and AMD GCN offload support from prepare_step to configure_step

boegel · 2021-10-20T13:22:24Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-11.2.0-amd-gpu-offload.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
pg-lab02 - Linux centos linux 8.3.2011, x86_64, AMD EPYC 7542 32-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/9bea3b9343980bd6320a5ed3fdd563cc for a full test report.

edit: system with two AMD MI100 GPUs, using tweaked easyconfig from #2578 (comment)

boegel · 2021-10-20T13:25:32Z

I've tested this on system without a GPU, with an NVIDIA A100, and with an AMD MI100, with GCCcore-11.2.0.eb, all checks out.

Example output of the test case:

non-GPU system:

$ gcc -fopenacc -foffload=nvptx-none -o gcc_gpu_offload_test gcc_gpu_offload_test.c
$ ./gcc_gpu_offload_test
Not compiled with OpenMP support!
Checking OpenACC:
        Found 0 devices

$ gcc -fopenmp -foffload=nvptx-none -o gcc_gpu_offload_test gcc_gpu_offload_test.c
[13:02:47] {skitty} vsc40023@node3157:~/data $ ./gcc_gpu_offload_test
Checking OpenMP
        Found 0 devices
Not compiled with OpenACC support!

NVIDIA A100 GPU (only 1 GPU available):

$ gcc -fopenacc -foffload=nvptx-none -o gcc_gpu_offload_test gcc_gpu_offload_test.c
$ ./gcc_gpu_offload_test
Not compiled with OpenMP support!
Checking OpenACC:
        Found 1 devices
        Checking device (0, NVIDIA A100-SXM-80GB - Nvidia) support: Device works

$ gcc -fopenmp -foffload=nvptx-none -o gcc_gpu_offload_test gcc_gpu_offload_test.c
$ ./gcc_gpu_offload_test
Checking OpenMP
        Found 1 devices
        Checking device (0) support: Device works
Not compiled with OpenACC support!

AMD MI100 GPU (2 available)

$  gcc  -fopenacc -foffload=amdgcn-amdhsa="-march=gfx908" gcc_gpu_offload_test.c -o gcc_gpu_offload_test
$ ./gcc_gpu_offload_test
Not compiled with OpenMP support!
Checking OpenACC:
        Found 2 devices
        Checking device (0, gfx908 - AMD) support: Device works
        Checking device (1, gfx908 - AMD) support: Device works

$ gcc  -fopenmp -foffload=amdgcn-amdhsa="-march=gfx908" gcc_gpu_offload_test.c -o gcc_gpu_offload_test
$ ./gcc_gpu_offload_test
Checking OpenMP
        Found 2 devices
        Checking device (0) support: Device works
        Checking device (1) support: Device works
Not compiled with OpenACC support!

I'll get a couple of test reports uploaded in this PR to ensure there's no fallout elsewhere...

boegel · 2021-10-20T16:21:56Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-10.2.0.eb
SUCCESS GCCcore-10.3.0.eb
SUCCESS GCCcore-11.1.0.eb
SUCCESS GCCcore-11.2.0.eb
SUCCESS GCCcore-9.4.0.eb
SUCCESS binutils-2.34.eb
SUCCESS GCCcore-9.3.0.eb

Build succeeded for 7 out of 7 (6 easyconfigs in total)
node3905.accelgor.os - Linux RHEL 8.4, x86_64, AMD EPYC 7413 24-Core Processor (zen2), 1 x NVIDIA NVIDIA A100-SXM-80GB, 470.57.02, Python 3.6.8
See https://gist.github.com/219b8c957ea5ae23f13a676e15947317 for a full test report.

boegel · 2021-10-20T17:19:29Z

Test report by @boegel

Overview of tested easyconfigs (in order)

SUCCESS GCCcore-10.2.0.eb
SUCCESS GCCcore-10.3.0.eb
SUCCESS GCCcore-11.1.0.eb
SUCCESS GCCcore-11.2.0.eb
SUCCESS GCCcore-9.3.0.eb
SUCCESS GCCcore-9.4.0.eb
SUCCESS GCCcore-8.3.0.eb
SUCCESS GCCcore-8.2.0.eb
SUCCESS GCCcore-7.3.0.eb
SUCCESS GCCcore-6.4.0.eb

Build succeeded for 10 out of 10 (10 easyconfigs in total)
node3157.skitty.os - Linux centos linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/56aa398ca38895e1874c96a4ed7e214c for a full test report.

boegel · 2021-10-20T17:52:16Z

Thanks a lot for the effort @nordmoen!

GCCcore: Add support for AMD GPU offloading

f9d40a2

Fixed prepare_step signature

53edf0d

boegel added the enhancement label Sep 15, 2021

boegel added this to the next release (4.5.0?) milestone Sep 15, 2021

boegel requested changes Sep 15, 2021

View reviewed changes

boegel changed the title ~~GCCcore: Add support for AMD GPU offloading~~ enhance GCC easyblock to add support for AMD GPU offloading Sep 15, 2021

Jørgen Nordmoen added 2 commits September 16, 2021 08:38

Update based on review

b0c8388

Removed unused import

79b2dbf

nordmoen requested a review from boegel September 16, 2021 07:16

boegel added 4 commits October 20, 2021 10:36

avoid duplicate code in prepare_step of GCC easyblock + tweak error t…

86abecc

…hat is raised when required additional sources are missing

avoid crash when failing to determine LLVM target for host CPU when c…

25b152f

…onfiguring AMD GPU offload support in GCC easyblock

refactoring in configure_step of GCC easyblock to avoid duplicate cod…

963c467

…e when configuring nvptx/AMD GCN target compiler

trivial code cleanup in custom install_step in GCC easyblock

b6080b6

boegel mentioned this pull request Oct 20, 2021

code cleanup for enhanced GCC easyblock which supports building with AMD GPU offload support nordmoen/easybuild-easyblocks#2

Merged

boegel requested changes Oct 20, 2021

View reviewed changes

nordmoen and others added 4 commits October 20, 2021 11:33

Merge pull request #2 from boegel/20210915133741_new_pr_gcc

a034e56

code cleanup for enhanced GCC easyblock which supports building with AMD GPU offload support

fix symlinking of newlib in correct directory in GCC easyblock

b4ddb37

avoid breaking --module-only for GCC(core) by moving check for source…

0605e0d

…s for optional dependencies for NVPTX and AMD GCN offload support from prepare_step to configure_step

test --module-only for GCC easyblock with 'withnvptx' enabled

6da4d4c

boegel mentioned this pull request Oct 20, 2021

avoid breaking --module-only for GCC(core) by moving check for sources for optional dependencies for NVPTX and AMD GCN offload support from prepare_step to configure_step nordmoen/easybuild-easyblocks#3

Merged

boegel requested changes Oct 20, 2021

View reviewed changes

easybuild/easyblocks/g/gcc.py Outdated Show resolved Hide resolved

Merge pull request #3 from boegel/20210915133741_new_pr_gcc

dc0be1b

avoid breaking --module-only for GCC(core) by moving check for sources for optional dependencies for NVPTX and AMD GCN offload support from prepare_step to configure_step

boegel approved these changes Oct 20, 2021

View reviewed changes

boegel merged commit 1ccb0f4 into easybuilders:develop Oct 20, 2021

nordmoen mentioned this pull request Feb 14, 2022

GCCcore AMD offloading support requires ROCR-Runtime easybuilders/easybuild-easyconfigs#14992

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance GCC easyblock to add support for AMD GPU offloading #2578

enhance GCC easyblock to add support for AMD GPU offloading #2578

nordmoen commented Sep 15, 2021

nordmoen commented Sep 15, 2021

nordmoen commented Sep 15, 2021

boegel left a comment

boegel commented Oct 20, 2021 •

edited

Loading

boegel commented Oct 20, 2021

boegel commented Oct 20, 2021

boegel commented Oct 20, 2021

boegel commented Oct 20, 2021

enhance GCC easyblock to add support for AMD GPU offloading #2578

enhance GCC easyblock to add support for AMD GPU offloading #2578

Conversation

nordmoen commented Sep 15, 2021

nordmoen commented Sep 15, 2021

nordmoen commented Sep 15, 2021

boegel left a comment

Choose a reason for hiding this comment

boegel commented Oct 20, 2021 • edited Loading

Overview of tested easyconfigs (in order)

boegel commented Oct 20, 2021

boegel commented Oct 20, 2021

Overview of tested easyconfigs (in order)

boegel commented Oct 20, 2021

Overview of tested easyconfigs (in order)

boegel commented Oct 20, 2021

boegel commented Oct 20, 2021 •

edited

Loading