PENG-2382 Create example benchmarking job-scripts for amd using HPL a…

…nd HPCG

hpcg-benchmark-oacc/Dockerfile

@@ -0,0 +1,23 @@
+FROM spack/ubuntu-jammy:0.23
+
+RUN apt-get update
+
+#setup aocc
+RUN spack install aocc +license-agreed
+RUN spack cd -i aocc && spack compiler add $PWD
+RUN spack load aocc
+RUN spack compiler find
+
+#install hpl
+RUN spack install hpcg%aocc +openmp ^openmpi fabrics=auto
+
+RUN spack load hpcg %aocc
+RUN mkdir /hpcg
+WORKDIR /hpcg
+COPY HPCG.dat .
+
+COPY entrypoint.sh /entrypoint.sh
+RUN chmod +x /entrypoint.sh
+
+ENTRYPOINT ["/entrypoint.sh"]
+CMD ["bash"]

hpcg-benchmark-oacc/HPCG.dat

@@ -0,0 +1,4 @@
+HPCG benchmark input file
+Demofile created by Vantage
+32 32 32
+60

hpcg-benchmark-oacc/README.md

@@ -0,0 +1,53 @@
+# Benchmark HPCG - High Performance Conjugate Gradient (HPCG) with AMD AOCc Optimization
+
+This version of the HPCG (High Performance Conjugate Gradient) benchmark leverages AMD's AOCc (AMD Optimizing Compiler) to enhance performance on AMD architectures. The HPCG benchmark evaluates the performance of high-performance computing systems, specifically focusing on memory-bound workloads and the use of conjugate gradient methods for solving sparse systems of linear equations.
+
+This project provides a simple-to-use setup to run the HPCG benchmark on a SLURM-managed cluster with AMD's AOCc optimization. The setup uses containers based on Ubuntu 22.04 and leverages OpenMP and OpenMPI for parallel execution, along with AMD-specific optimizations for enhanced performance.
+
+## Requirements
+
+To successfully run this script, the following system components are required:
+
+- **SLURM**: For job scheduling and resource management.
+- **Singularity/Apptainer**: For container execution.
+- **MPI**: For parallel execution of the benchmark.
+- **Shared File System**: For storing and sharing data across compute nodes.
+
+## Simulation Steps
+
+The script follows these steps to execute the benchmark:
+
+1. **Create Working Directory**: A unique working directory is created under `/nfs/mnt/hpcg_oacc/` to store data for the specific job run.
+2. **Fetch Apptainer Image**: If the Apptainer image is not available locally, the script pulls it from a public Amazon ECR repository.
+3. **Run the Benchmark**: The script runs the HPCG benchmark using the pre-configured `HPCG.dat` file inside the Apptainer container.
+
+## HPC Application
+
+This benchmark utilizes the following HPC applications:
+
+- **HPCG**: A high-performance conjugate gradient solver for evaluating the computational performance of systems on memory-bound problems. [HPCG Documentation](https://www.hpcg-benchmark.org/)
+
+The Apptainer image for HPCG is pre-built with AMD AOCc optimizations and other necessary dependencies, ensuring consistent execution and performance across different environments.
+
+## Result Interpretation
+
+The benchmark generates results stored in the `HPCG.out` file. Key performance metrics include:
+
+- **HPCG performance**: Measured in GFLOP/s, this represents the floating-point operations per second achieved during the conjugate gradient computations.
+- **System Configuration**: Details on matrix dimensions and process grid used during the benchmark.
+
+## Customization
+
+Several options for customization are available to tailor the benchmark to different setups:
+
+- **Configuration Parameters**: You can modify the `HPCG.dat` file to adjust parameters such as grid dimensions.
+
+  1. Download and edit the [HPCG.dat](https://omnivector-public-assets.s3.us-west-2.amazonaws.com/catalog/HPCG.dat) file.
+  2. Customize the parameters as needed.
+  3. Upload the modified `HPCG.dat` file for use in your benchmark.
+
+- **Parallelism**: Update the `--ntasks` (_SBATCH_) and `-np` (_MPIRUN_) values in the JobScript to test different parallel setups.
+
+## Known Issues
+
+- **Performance Tuning**: Performance can vary depending on system architecture and configuration. Using AMD-specific optimizations, such as AOCc and BLIS, should result in improved performance on AMD processors.

hpcg-benchmark-oacc/entrypoint.sh

@@ -0,0 +1,13 @@
+export CORES_PER_CCX=8
+
+export OMP_PROC_BIND=true
+export OMP_PLACES=cores
+export OMP_NUM_THREADS=2
+
+export NUM_CORES=$(nproc)
+export NUM_MPI_RANKS=$(( $NUM_CORES / $OMP_NUM_THREADS ))
+export RANKS_PER_CCX=$(( $CORES_PER_CCX / $OMP_NUM_THREADS ))
+
+source /opt/spack/share/spack/setup-env.sh
+spack load hpcg %aocc
+exec "$@"

hpcg-benchmark-oacc/hpcg-oacc.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+#SBATCH -J HPCG_oacc
+#SBATCH --ntasks=2
+#SBATCH --output=/nfs/mnt/hpcg_oacc/R-%x.%j.out
+#SBATCH --error=/nfs/mnt/hpcg_oacc/R-%x.%j.err
+#SBATCH -t 00:05:00
+
+WORK_DIR=/nfs/mnt/hpcg_oacc/$SLURM_JOB_NAME-Job-$SLURM_JOB_ID
+mkdir -p $WORK_DIR
+cd $WORK_DIR
+
+export APPTAINER_IMAGE=/nfs/mnt/vantage-jobs-catalog-hpcg_oacc.sif
+
+# download the singularity image if it is not available yet
+if [[ ! -f $APPTAINER_IMAGE ]]
+then
+    echo "Pulling the singularity image"
+    apptainer pull $APPTAINER_IMAGE oras://public.ecr.aws/omnivector-solutions/benchmark:hpcg-oacc-ubuntu22
+else
+    echo "Skipping the image fetch process...we already have the singularity image"
+fi
+
+mpirun -np 2 apptainer exec $APPTAINER_IMAGE bash -c "source /entrypoint.sh && xhcg"

hpcg-benchmark-oacc/metadata.yaml

@@ -0,0 +1,10 @@
+summary: Benchmark using HPCG built with AMD compilers
+icon-url: https://logosmarcas.net/wp-content/uploads/2020/03/AMD-Logo.png
+entrypoint: hpcg-oacc.sh
+supporting-files:
+  - HPCG.dat
+image-source:
+  - Dockerfile
+image-tags:
+  - "1.0.0"
+  - "latest"

hpcg-benchmark/Dockerfile

@@ -0,0 +1,27 @@
+FROM ubuntu:22.04
+
+RUN apt-get update
+RUN apt-get install -y --no-install-recommends \
+    build-essential \
+    libopenmpi-dev \
+    openmpi-bin \
+    gfortran
+
+RUN apt-get install -y wget cmake ca-certificates
+
+RUN apt-get clean && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /
+RUN wget https://www.hpcg-benchmark.org/downloads/hpcg-3.1.tar.gz
+RUN tar -xvzf hpcg-3.1.tar.gz
+RUN mv hpcg-3.1 hpcg
+RUN rm hpcg-3.1.tar.gz
+WORKDIR /hpcg
+
+COPY Make.ubuntu setup/
+
+RUN make arch=ubuntu
+
+WORKDIR /hpcg/bin
+
+CMD ["bash"]

hpcg-benchmark/HPCG.dat

@@ -0,0 +1,4 @@
+HPCG benchmark input file
+Demofile created by Vantage
+32 32 32
+60

hpcg-benchmark/Make.ubuntu

@@ -0,0 +1,59 @@
+# Reference: https://github.com/IBM/HPCG/blob/master/setup/Make.Linux_MPI
+
+SHELL        = /bin/sh
+#
+CD           = cd
+CP           = cp
+LN_S         = ln -s -f
+MKDIR        = mkdir -p
+RM           = /bin/rm -f
+TOUCH        = touch
+#
+# - HPCG Directory Structure / HPCG library
+#
+TOPdir       = .
+SRCdir       = $(TOPdir)/src
+INCdir       = $(TOPdir)/src
+BINdir       = $(TOPdir)/bin
+#
+# - Message Passing library (MPI)
+# MPinc tells the  C  compiler where to find the Message Passing library
+# header files,  MPlib  is defined  to be the name of  the library to be
+# used. The variable MPdir is only used for defining MPinc and MPlib.
+#
+MPdir        =
+MPinc        =
+MPlib        =
+#
+#
+# - HPCG includes / libraries / specifics
+#
+HPCG_INCLUDES = -I$(INCdir) -I$(INCdir)/$(arch) $(MPinc)
+HPCG_LIBS     =
+#
+# - Compile time options
+#
+# -DHPCG_NO_MPI	        Define to disable MPI
+# -DHPCG_NO_OPENMP	Define to disable OPENMP
+# -DHPCG_CONTIGUOUS_ARRAYS Define to have sparse matrix arrays long and contiguous
+# -DHPCG_DEBUG       	Define to enable debugging output
+# -DHPCG_DETAILED_DEBUG Define to enable very detailed debugging output
+#
+HPCG_OPTS     = -DHPCG_NO_OPENMP
+#
+#
+HPCG_DEFS     = $(HPCG_OPTS) $(HPCG_INCLUDES)
+#
+# - Compilers / linkers - Optimization flags
+#
+CXX          = mpicxx
+#CXXFLAGS     = $(HPCG_DEFS) -fomit-frame-pointer -O3 -funroll-loops -W -Wall
+CXXFLAGS     = $(HPCG_DEFS) -O3 -ffast-math -ftree-vectorize -ftree-vectorizer-verbose=0
+#
+LINKER       = $(CXX)
+LINKFLAGS    = $(CXXFLAGS)
+#
+ARCHIVER     = ar
+ARFLAGS      = r
+RANLIB       = echo
+#

hpcg-benchmark/README.md

@@ -0,0 +1,49 @@
+# Benchmark HPCG - High-Performance Conjugate Gradient (HPCG)
+
+The HPCG (High-Performance Conjugate Gradient) benchmark is designed to measure the performance of parallel computing systems on workloads that reflect modern scientific computations. It provides insights into memory access patterns, communication, and compute performance.
+
+This project provides a ready-to-use setup to run HPCG on a SLURM-managed cluster using containers built on Ubuntu 22.04 with MPI and necessary libraries.
+
+## Requirements
+
+To properly run this script, the system must meet the following requirements:
+
+- **SLURM:** For job scheduling and resource management.
+- **Singularity/Apptainer:** To execute containers.
+- **MPI:** For parallel execution.
+- **Shared File System:** To store and share data between the compute nodes.
+
+## Simulation Steps
+
+The script orchestrates the following steps:
+
+1. **Create Working Directory:** A unique working directory is created within the shared `/nfs/mnt/hpcg` directory.
+2. **Fetch Apptainer Image:** The script pulls the pre-built HPCG Apptainer image from a public Amazon ECR repository if it's not already available locally.
+3. **Run the Benchmark:** The script executes the HPCG benchmark with a pre-configured HPCG.dat file inside the Apptainer image.
+
+## HPC Application
+
+This script utilizes the following HPC application:
+
+- **HPCG:** A benchmark designed to complement HPL by addressing a broader set of computational and communication patterns. [HPCG Documentation](https://www.hpcg-benchmark.org)
+
+The Apptainer image for HPCG is pre-built with all necessary dependencies and configurations, simplifying execution and ensuring reproducibility across different HPC environments.
+
+## Result Interpretation
+
+The main output is stored in the `R-HPCG.<job_id>.out` file, located in the working directory created for the job. Key metrics include:
+
+- **Performance:** Measured in GFLOP/s, indicating billions of floating-point operations per second.
+- **System Configuration:** Details on problem size, MPI task count, and runtime.
+
+## Customization
+
+The script offers several customization options:
+
+- **Configuration Parameters:** Edit the `HPCG.dat` file to modify the benchmark settings:
+
+  1. Download the [HPCG.dat](https://omnivector-public-assets.s3.us-west-2.amazonaws.com/catalog/HPCG.dat);
+  2. Modify the `HPCG.dat` as needed;
+  3. Place the customized file in the working directory or bind it during execution.
+
+- **Parallelism:** Update the `--ntasks` (_SBATCH_) and `--np` (_MPIRUN_) values in the JobScript to test with different levels of parallel execution.

hpcg-benchmark/hpcg.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+#SBATCH -J HPCG
+#SBATCH --ntasks=2
+#SBATCH --output=/nfs/mnt/hpcg/R-%x.%j.out
+#SBATCH --error=/nfs/mnt/hpcg/R-%x.%j.err
+#SBATCH -t 00:05:00
+
+WORK_DIR=/nfs/mnt/hpcg/$SLURM_JOB_NAME-Job-$SLURM_JOB_ID
+mkdir -p $WORK_DIR
+cd $WORK_DIR
+
+export APPTAINER_IMAGE=/nfs/mnt/vantage-jobs-catalog-hpcg.sif
+
+# download the singularity image if it is not available yet
+if [[ ! -f $APPTAINER_IMAGE ]]
+then
+    echo "Pulling the singularity image"
+    apptainer pull $APPTAINER_IMAGE oras://public.ecr.aws/omnivector-solutions/hpcg:latest
+else
+    echo "Skipping the image fetch process...we already have the singularity image"
+fi
+
+mpirun --np 2 apptainer exec --bind ./HPCG.dat:/hpcg/bin/hpcg.dat $APPTAINER_IMAGE /hpcg/bin/xhpcg

hpcg-benchmark-intel/metadata.yaml → hpcg-benchmark/metadata.yaml

@@ -1,7 +1,9 @@
 summary: Benchmark using HPCG
 icon-url: https://www.hpcg-benchmark.org/images/header.gif
-entrypoint: job-script.sh
+entrypoint: hpcg.sh
+supporting-files:
+  - HPCG.dat
 image-source: Dockerfile
 image-tags:
-- "1.0.0"
-- "latest"
+  - "1.0.0"
+  - "latest"