Skip to content

Commit 56f1ae0

Browse files
drroedrroe
authored
Add PME for GIST (#889)
* Start adding a new class, GIST_PME, that will inherit from Ewald_ParticleMesh and implement GIST-related PME stuff * Add recip and self6 calcs * Add long range correction for gist * Add direct space sum routine. * Bring HelPME up to PR #54 * Expose sumq and vdw_recip_term via functions * Add some per-atom vdw stuff needed for PME_GIST * Expose more of Ewald to inheriting classes * Expose more of Ewald_ParticleMesh for inherting classes. Add implementation of ErfcFxn * Add direct space calcs * Make switch function available to inheriting classes * Fix exclusion array type * GIST with LJPME not ready for primetime * Add PmeOptions class. * Add help keywords. Ensure all variables are initialized. * Add functions to return private vars * Add Init for Ewald_ParticleMesh with PmeOptions * Start adding PME to GIST. Initialize some uninitialized vars * Create non-inlined version of adjust for GIST_PME * Make LJ PME keywords a separate function for things that do not support it * Start adding PME GIST data sets * LJ PME not yet allowed for GIST * Ensure energy calc is done for any occupied voxel (occupancy > 0 instead of > 1). * Update tip4p and tip5p tests for occupancy threshold change * Finish PME init * Add debug level. Do PME init and setup. * Add solute/water id and solute index arrays * Start adding actual PME calc. Need to save whether atom is solvent or solute separate from atom_voxel since we now might care about whether solute is on or off the grid. * atom_voxel_ and atomIsSolute_ arrays will be accessed by atom # * Add PME solute grid assignment * Un-comment the pme calc function * Do the order calc for CUDA as well - not sure why that was behind the ifdef * Enable pme calc * Add avg voxel energy calc for pme data * Print out sums - they seem to be only for debug * Start adding separate avg routine for non pme energy * Use new averaging routine * Add nopme keywords * Add orthogonal pme test * Fix default lj pme assignment * Add non-orthogonal test * Add info comparisons * Move DEBYE_EA to Constants * Add separate PME printout for testing * Add headers for PME output * Dipole calc should be done whether or not we skip energy * Remove duplicated code * Add code docs * Print options when using pme * Add regular ewald options. * Rename ; will use for all Ewald * Change to EwaldOptions * Use EwaldOptions in PME * Use Ewald_Regular * Use EwaldOptions * ewcoefflj keyword can turn on LJ pme * Have GIST PME use EwaldOptions. Update depends * Fix up help for energy. Pass LJ switch width for regular ewald * Remove old code * Fix printout of LJ options; now all in EwaldOptions * Reenable some timers * Move var closer to where it is set * Start fixing openmp * Add more internal arrays * Internal arrays are per atom, not voxel... * Add doc * Ensure direct arrays are zeroed out. * Ensure contributions from other threads are summed into 0 arrays * atom_voxel was unused * Add access to internal arrays * Add function to return energy on a specified atom * Use reworked GIST_PME. Make numthreads a class variable * Comment out some unused stuff. * Remove old code. Add Ewald timing to output * The PME GIST grid arrays do not need to be threaded * Hide some debug info. Fix citation in output * Minor version bump for GIST nw_total > 0 fix and addition of PME * doeij does not work with PME, trap it. Also make doeij with cuda an error. * Fix spacing * Make function const * Was accidentally doing the order calculation twice. * Do not run the PME tests for cuda. Slightly increase the test tolerance for GPU. * Try to fix CUDA compile. Need a better way to determine arch flags... * Add list of cuda flags. * Try a better way to set up the shader model flags * Add shader model cuda version check * Remove old configure logic * Consolidate direct space energy calc into one function. Add adjust energy kernel. Use adjust energy kernel. Make PME GIST output file v3 because of additional PME columns * Compare PME GIST output if present. Not uploading it because its too big for GitHub * Make GIST PME off the default for now until the output stabilizes. * Fix help option * Fix GIST and energy command entries * Protect when no LIBPME * Add tolerance to the info comparisons. * Break up giant apt-get install command into separate ones to make it easier to debug * These are better described as to-do * Try to DL and build our own netcdf * Try to make sure netcdf binaries are in the PATH. Try to fix cmake variable names. * Make var point to actual library * Cmake build seems to have problems with the static netcdf compile. Try using .so instead.
1 parent 113407f commit 56f1ae0

35 files changed

+875496
-3600
lines changed

‎.github/workflows/merge-gate.yml‎

Lines changed: 19 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -41,19 +41,13 @@ jobs:
4141
steps:
4242
- name: Install prerequisite packages
4343
run: |
44-
sudo apt-get install gfortran \
45-
libbz2-dev \
46-
libblas-dev \
47-
liblapack-dev \
48-
libnetcdf-dev \
49-
libfftw3-dev \
50-
netcdf-bin \
51-
clang \
52-
openmpi-bin \
53-
openmpi-common \
54-
libopenmpi-dev \
55-
cmake-data \
56-
cmake
44+
sudo apt-get install gfortran
45+
sudo apt-get install libbz2-dev
46+
sudo apt-get install libblas-dev liblapack-dev
47+
sudo apt-get install libfftw3-dev
48+
sudo apt-get install clang
49+
sudo apt-get install openmpi-bin openmpi-common libopenmpi-dev
50+
sudo apt-get install cmake-data cmake
5751
5852
- name: Checkout source code
5953
uses: actions/checkout@v2
@@ -68,6 +62,15 @@ jobs:
6862
mkdir -p include && mv AmberTools/src/sander/sander.h include
6963
mv lib include $HOME
7064
65+
curl -OL ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.6.1.tar.gz
66+
tar -zxf netcdf-4.6.1.tar.gz
67+
cd netcdf-4.6.1
68+
./configure --disable-netcdf-4 --disable-dap --disable-doxygen --prefix=$HOME
69+
make -j2
70+
make install
71+
cd ..
72+
export PATH=$HOME/bin:$PATH
73+
7174
if [ $USE_OPENMP = "yes" ]; then
7275
export OPT="openmp"
7376
export OMP_NUM_THREADS=4
@@ -92,13 +95,14 @@ jobs:
9295
cd build
9396
cmake .. $BUILD_FLAGS -DCOMPILER=${COMPILER^^} -DINSTALL_HEADERS=FALSE \
9497
-DCMAKE_INSTALL_PREFIX=$installdir -DCMAKE_LIBRARY_PATH=$HOME/lib \
95-
-DPRINT_PACKAGING_REPORT=TRUE
98+
-DPRINT_PACKAGING_REPORT=TRUE -DNetCDF_LIBRARIES_C=$HOME/lib/libnetcdf.so \
99+
-DNetCDF_INCLUDES=$HOME/include
96100
make -j2 install
97101
cd ..
98102
export PATH=$installdir/bin:$PATH
99103
else
100104
export LD_LIBRARY_PATH=$HOME/lib:${LD_LIBRARY_PATH}
101-
./configure ${BUILD_FLAGS} ${COMPILER}
105+
./configure --with-netcdf=$HOME ${BUILD_FLAGS} ${COMPILER}
102106
make -j2 install
103107
fi
104108
cd test && make $TEST_TYPE

‎configure‎

Lines changed: 122 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -75,16 +75,21 @@ UsageFull() {
7575
echo " NVCCFLAGS : Flags to pass to the nvcc compiler."
7676
echo " DBGFLAGS : Any additional flags to pass to all compilers."
7777
echo " SHADER_MODEL : (-cuda) Should be set to 'sm_XX', where XX is CUDA compute architecture."
78-
echo " SM6.2 = GP10B"
79-
echo " SM6.1 = GP106 = GTX-1070, GP104 = GTX-1080, GP102 = Titan-X[P]"
80-
echo " SM6.0 = GP100 / P100 = DGX-1"
81-
echo " SM5.3 = GM200 [Grid] = M60, M40?"
82-
echo " SM5.2 = GM200 = GTX-Titan-X, M6000 etc."
83-
echo " SM5.0 = GM204 = GTX980, 970 etc"
84-
echo " SM3.7 = GK210 = K80"
85-
echo " SM3.5 = GK110 = K20[x], K40, GTX780, GTX-Titan, GTX-Titan-Black, GTX-Titan-Z"
86-
echo " SM3.0 = GK104 = K10, GTX680, 690 etc."
87-
echo " SM2.0 = All GF variants = C2050, 2075, M2090, GTX480, GTX580 etc."
78+
echo " sm_86 = GA102, 104, 106, 107"
79+
echo " sm_80 = GA100"
80+
echo " sm_75 = Turing"
81+
echo " sm_72 = GV10B"
82+
echo " sm_70 = GV100"
83+
echo " sm_62 = GP10B"
84+
echo " sm_61 = GP106 = GTX-1070, GP104 = GTX-1080, GP102 = Titan-X[P]"
85+
echo " sm_60 = GP100 / P100 = DGX-1"
86+
echo " sm_53 = GM200 [Grid] = M60, M40?"
87+
echo " sm_52 = GM200 = GTX-Titan-X, M6000 etc."
88+
echo " sm_50 = GM204 = GTX980, 970 etc"
89+
echo " sm_37 = GK210 = K80"
90+
echo " sm_35 = GK110 = K20[x], K40, GTX780, GTX-Titan, GTX-Titan-Black, GTX-Titan-Z"
91+
echo " sm_30 = GK104 = K10, GTX680, 690 etc."
92+
echo " sm_20 = All GF variants = C2050, 2075, M2090, GTX480, GTX580 etc."
8893
echo " EXPERIMENTAL OPTIONS:"
8994
echo " --compile-verbose : Turn on compile details."
9095
echo " -profile : Use Gnu compiler profiling (>= V4.5)*"
@@ -1204,8 +1209,18 @@ SetupLibraries() {
12041209
fi
12051210
lflag=${LIB_FLAG[$i]}
12061211
else
1207-
# Lib home specified
1208-
linc="-I$lhome/include"
1212+
# Lib home specified.
1213+
# Determine include directory.
1214+
incdir="$lhome/include"
1215+
linc="-I$incdir"
1216+
if [ ! -d "$incdir" ] ; then
1217+
# include dir is not in the usual place, happens with e.g. some CUDA installs.
1218+
if [ -d "$lhome/targets/x86_64-linux/include" ] ; then
1219+
linc="-I$lhome/targets/x86_64-linux/include"
1220+
else
1221+
WrnMsg "Include dir $incdir not found. Linking ${LIB_CKEY[$i]} may fail."
1222+
fi
1223+
fi
12091224
# Check if architecture-specific lib dir exists. Use that if so.
12101225
lhdir="$lhome/lib"
12111226
ladir="$lhome/lib$NBITS"
@@ -1688,6 +1703,65 @@ SetupMKL() {
16881703
fi
16891704
fi
16901705
}
1706+
# ------------------------------------------------------------------------------
1707+
# Define different shader models/compute architectures and their CUDA limits
1708+
# CUDA version
1709+
# 3.0-3.1 ...
1710+
# 3.2 .........
1711+
# 4.0-4.2 .........
1712+
# 5.X ...........................
1713+
# 6.0 ...........................
1714+
# 6.5 ...................................................
1715+
# 7.X ...................................................
1716+
# 8.X .....................................................................
1717+
# 9.X .....................................................................
1718+
# 10.X ...........................................................................
1719+
# 11.X ...........................................................................
1720+
CUDA_SM_LIST='sm_20 sm_21 sm_30 sm_32 sm_35 sm_37 sm_50 sm_52 sm_53 sm_60 sm_61 sm_62 sm_70 sm_72 sm_75 sm_80 sm_86'
1721+
1722+
# SetSupportedSM <major v> <minor v>
1723+
# Set Shader models supported by current cuda version
1724+
SetSupportedSM() {
1725+
if [ $1 -lt 3 ] ; then
1726+
Err "CUDA < 3 not supported."
1727+
fi
1728+
if [ $1 -eq 3 ] ; then
1729+
if [ $2 -ge 2 ] ; then
1730+
CUDA_SM_LIST='sm_20 sm_21'
1731+
else
1732+
CUDA_SM_LIST='sm_20'
1733+
fi
1734+
elif [ $1 -eq 4 ] ; then
1735+
CUDA_SM_LIST='sm_20 sm_21'
1736+
elif [ $1 -eq 5 ] ; then
1737+
CUDA_SM_LIST='sm_20 sm_21 sm_30 sm_32 sm_35'
1738+
elif [ $1 -eq 6 ] ; then
1739+
if [ $2 -ge 5 ] ; then
1740+
CUDA_SM_LIST='sm_20 sm_21 sm_30 sm_32 sm_35 sm_37 sm_50 sm_52 sm_53'
1741+
else
1742+
CUDA_SM_LIST='sm_20 sm_21 sm_30 sm_32 sm_35'
1743+
fi
1744+
elif [ $1 -eq 7 ] ; then
1745+
CUDA_SM_LIST='sm_20 sm_21 sm_30 sm_32 sm_35 sm_37 sm_50 sm_52 sm_53'
1746+
elif [ $1 -eq 8 ] ; then
1747+
CUDA_SM_LIST='sm_20 sm_21 sm_30 sm_32 sm_35 sm_37 sm_50 sm_52 sm_53 sm_60 sm_61 sm_62'
1748+
elif [ $1 -eq 9 ] ; then
1749+
CUDA_SM_LIST='sm_30 sm_32 sm_35 sm_37 sm_50 sm_52 sm_53 sm_60 sm_61 sm_62 sm_70 sm_72'
1750+
elif [ $1 -eq 10 ] ; then
1751+
CUDA_SM_LIST='sm_30 sm_32 sm_35 sm_37 sm_50 sm_52 sm_53 sm_60 sm_61 sm_62 sm_70 sm_72 sm_75'
1752+
else # >= 11
1753+
CUDA_SM_LIST='sm_35 sm_37 sm_50 sm_52 sm_53 sm_60 sm_61 sm_62 sm_70 sm_72 sm_75 sm_80 sm_86'
1754+
fi
1755+
echo " Supported shader models: $CUDA_SM_LIST"
1756+
}
1757+
1758+
# SetCudaArch <sm>
1759+
# Set CUDA_ARCH variable with compute_XX value for given SM
1760+
SetCudaArch() {
1761+
smversion=${1#sm_}
1762+
CUDA_ARCH="compute_$smversion"
1763+
#echo "$1 $CUDA_ARCH"
1764+
}
16911765

16921766
# ------------------------------------------------------------------------------
16931767
# Check that CUDA_HOME is defined and set up flags for nvcc
@@ -1696,44 +1770,52 @@ SetupCUDA() {
16961770
Err "CUDA_HOME not set. Set CUDA_HOME to point to your NVIDIA tools installation."
16971771
fi
16981772
if [ ! -x "$CUDA_HOME/bin/nvcc" ]; then
1699-
Err "Error: nvcc cuda compiler not found in $CUDA_HOME/bin"
1773+
Err "nvcc cuda compiler not found in $CUDA_HOME/bin"
17001774
fi
17011775
if [ -z "$NVCC" ]; then NVCC="$CUDA_HOME/bin/nvcc"; fi
17021776
cuda_version=`$NVCC --version | grep 'release' | cut -d' ' -f5 | cut -d',' -f1`
1777+
cuda_major_version=`echo "$cuda_version" | awk 'BEGIN{FS=".";}{printf("%i", $1);}'`
1778+
cuda_minor_version=`echo "$cuda_version" | awk 'BEGIN{FS=".";}{printf("%i", $2);}'`
17031779
echo " CUDA version $cuda_version detected."
1704-
SM_CONFIG="Configuring for $SHADER_MODEL"
1780+
# A zero version indicates version detection failed.
1781+
if [ $cuda_major_version -lt 1 ] ; then
1782+
Err "CUDA version detection failed."
1783+
fi
1784+
SetSupportedSM $cuda_major_version $cuda_minor_version
1785+
17051786
if [ -z "$NVCCFLAGS" -a -z "$SHADER_MODEL" ] ; then
1787+
# Compile for multiple shader models
17061788
WrnMsg "SHADER_MODEL not set. Compiling for multiple architectures."
17071789
WrnMsg "To compile for a specific architecture set SHADER_MODEL"
17081790
WrnMsg "to 'sm_XX', where XX is the shader model version."
1709-
# NOTE: From AmberTools configure2
1710-
#Note at present we do not include SM3.5 or SM3.7 since they sometimes show performance
1711-
#regressions over just using SM3.0.
1712-
# TODO fix for volta?
1713-
sm70flags='-gencode arch=compute_60,code=sm_70'
1714-
sm62flags='-gencode arch=compute_62,code=sm_62'
1715-
sm61flags='-gencode arch=compute_61,code=sm_61'
1716-
sm60flags='-gencode arch=compute_60,code=sm_60'
1717-
sm53flags='-gencode arch=compute_53,code=sm_53'
1718-
sm52flags='-gencode arch=compute_52,code=sm_52'
1719-
sm50flags='-gencode arch=compute_50,code=sm_50'
1720-
sm37flags='-gencode arch=compute_37,code=sm_37'
1721-
sm35flags='-gencode arch=compute_35,code=sm_35'
1722-
sm30flags='-gencode arch=compute_30,code=sm_30'
1723-
sm20flags='-gencode arch=compute_20,code=sm_20'
1724-
if [ "$cuda_version" = '9.0' -o "$cuda_version" = '9.1' -o "$cuda_version" = '9.2' -o "$cuda_version" = "10.0" -o "$cuda_version" = "10.1" ] ; then
1725-
SM_CONFIG="Configuring for SM3.0, SM5.0, SM5.2, SM5.3, SM6.0, SM6.1, and SM7.0"
1726-
NVCCFLAGS="$sm30flags $sm50flags $sm52flags $sm53flags $sm60flags $sm61flags $sm70flags"
1727-
elif [ "$cuda_version" = '8.0' ] ; then
1728-
SM_CONFIG="Configuring for SM2.0, SM3.0, SM5.0, SM5.2, SM5.3, SM6.0 and SM6.1"
1729-
NVCCFLAGS="$sm20flags $sm30flags $sm50flags $sm52flags $sm53flags $sm60flags $sm61flags"
1730-
else
1731-
SM_CONFIG="Configuring for SM2.0, SM3.0, SM5.0, SM5.2 and SM5.3"
1732-
echo "BE AWARE: CUDA < 8.0 does not support GTX-1080, Titan-XP, DGX-1 or other Pascal based GPUs."
1733-
NVCCFLAGS="$sm20flags $sm30flags $sm50flags $sm52flags $sm53flags"
1791+
# TODO determine why Amber has arch=compute_60 for 70 and 75
1792+
SM_CONFIG="Configuring for"
1793+
NVCCFLAGS="$DBFLAG"
1794+
# Loop over supported shader models for this CUDA
1795+
for sm in $CUDA_SM_LIST ; do
1796+
SetCudaArch $sm
1797+
SM_CONFIG="$SM_CONFIG $sm"
1798+
NVCCFLAGS="$NVCCFLAGS -gencode arch=$CUDA_ARCH,code=$sm"
1799+
done
1800+
elif [ -z "$NVCCFLAGS" -a ! -z "$SHADER_MODEL" ] ; then
1801+
# Compile for single shader model
1802+
SM_CONFIG="Configuring for $SHADER_MODEL"
1803+
# See if it is supported.
1804+
sm_is_supported=0
1805+
for sm in $CUDA_SM_LIST ; do
1806+
if [ "$sm" = "$SHADER_MODEL" ] ; then
1807+
sm_is_supported=1
1808+
break
1809+
fi
1810+
done
1811+
if [ $sm_is_supported -eq 0 ] ; then
1812+
Err "Shader model $SHADER_MODEL is not supported by CUDA $cuda_version"
17341813
fi
1814+
NVCCFLAGS="$DBFLAG -arch=$SHADER_MODEL"
1815+
else
1816+
# Use specified NVCC flags
1817+
SM_CONFIG="Using NVCCFLAGS: $NVCCFLAGS"
17351818
fi
1736-
if [ -z "$NVCCFLAGS" ]; then NVCCFLAGS="$DBFLAG -arch=$SHADER_MODEL"; fi
17371819
if [ ! -z "$picflag" ] ; then
17381820
NVCCFLAGS="--compiler-options $picflag $NVCCFLAGS"
17391821
fi

0 commit comments

Comments
 (0)