Installing HPC Version in a Singularity Container

[marinsiebert]

Hello everyone,

i want to use the MOOSEFRAMEWORK on a HPC-Cluster which doesn't have the python development libraries installed and is not planning on doing so. I have currently a functioning version running with the conda environment, however in the installation guide it says that the HPC installation will perform better.
The HPC support refers me to using a singularity container, install the required packages there and use that to run the desired software.
However, I am not able to run the tests in the container after sucessfully compiling, as following traceback is produced (seems to have to do with python multiprocessing):

 ./run_tests -j 6
Traceback (most recent call last):
  File "./run_tests", line 8, in <module>
    TestHarness.buildAndRun(sys.argv, None, MOOSE_DIR)
  File "/root/projects/moose/python/TestHarness/TestHarness.py", line 180, in buildAndRun
    harness = TestHarness(argv, moose_dir, app_name=app_name, moose_python=moose_python)
  File "/root/projects/moose/python/TestHarness/TestHarness.py", line 342, in __init__
    self.initialize(argv, app_name)
  File "/root/projects/moose/python/TestHarness/TestHarness.py", line 924, in initialize
    self.scheduler = self.factory.create(scheduler_plugin, self, plugin_params)
  File "/root/projects/moose/python/FactorySystem/Factory.py", line 28, in create
    return self.objects[type](*args, **kwargs)
  File "/root/projects/moose/python/TestHarness/schedulers/RunParallel.py", line 27, in __init__
    Scheduler.__init__(self, harness, params)
  File "/root/projects/moose/python/TestHarness/schedulers/Scheduler.py", line 74, in __init__
    self.run_pool = ThreadPool(processes=self.available_slots)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
    Pool.__init__(self, processes, initializer, initargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
  File "/usr/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/usr/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
FileNotFoundError: [Errno 2] No such file or directory

Is there a known quick and easy way around this or is this to singularity-specific to be solved here?

[lindsayad]

@milljm

[milljm]

I have never encountered this error before. Seems to originate from within Python's multiprocessing package. But I don't know why that would be the case. What flavor of Linux did you base this container off of?

Also, I think we can get a real clean picture of what went into this container if you can supply the results of the following comand:

apptainer inspect --deffile yourcontainer.sif

Please look through the results before posting, and comment out anything you think may be sensitive.

[marinsiebert]

edit: I am right now in the process of creating a new container and complete the whole process from scratch, will upload the results later on.

this just produces:
bootstrap: localimage
from: ubuntu-test/

I created the container using
$singularity build --sandbox ubuntu docker://ubuntu
$sudo singularity shell --writable ubuntu

I should mention i did this on two different machines using the ubuntu image above or the ubuntu20.04 image, respectively.

I followed the installation for both HPC Clusters and for Offline (air-gapped), the traceback is the same.
Below is the command history for the installation of the required packages along the way. I now remembered, that there where some issues with the installation of gcc and mpich, which is why I used the standard packages from build-essential and apt - could this maybe a configuration problem?

1  ls
2  apt install wget
3  apt-get install wget
4  ls
5  cd ..
6  ls
7  cd home
8  ls
9  pip install python3

10 apt-get install python3
11 apt update
12 apt -upgrade
13 apt upgrade
14 apt-get install python3
15 apt-get install curl
16 curl -L -O http://mirrors.concertpass.com/gcc/releases/gcc-10.4.0/gcc-10.4.0.tar.gz
17 tar -xf gcc-10.4.0.tar.gz -C .
18 cd gcc-10.4.0
19 ./contrib/download_prerequisites
20 ls
21 cd ..
22 ls
23 rm * gcc-10.4.0
24 rm -rf gcc-10.4.0/
25 git clone git://gcc.gnu.org/git/gcc.git gcc
26 apt-get install git
27 git clone git://gcc.gnu.org/git/gcc.git gcc
28 cd gcc/
29 ls
30 ./contrib/download_prerequisites
31 apt-get install bzip2
32 ./contrib/download_prerequisites
33 mkdir gcc-build
34 cd gcc-build
35 ../configure --prefix=/target/installation/path/gcc --disable-multilib --enable-languages=c,c++,fortran,jit --enable-checking=release --enable-host-shared --with-pic
36 apt-get install build-essential
37 ../configure --prefix=/target/installation/path/gcc --disable-multilib --enable-languages=c,c++,fortran,jit --enable-checking=release --enable-host-shared --with-pic
38 make -j 6
39 make install
40 rm config.cache gcc/config.cache
41 apt-get install g++
42 apt get install gcc-c++
43 apt-get install gcc-c++
44 g++ --version
45 gcc --vesrion
46 gcc --version
47 dpkg -l | grep gcc | awk '{print $2}'
48 apt remove gccv
49 apt remove gcc
50 dpkg -l | grep gcc | awk '{print $2}'
51 apt remove gcc-11
52 dpkg -l | grep gcc | awk '{print $2}'
53 apt autoremove
54 dpkg -l | grep gcc | awk '{print $2}'
55 apt autoremove
56 apt autoremove gcc-12-base
57 apt autoremove
58 dpkg -l | grep gcc | awk '{print $2}'
59 ls
60 cd ..
61 ls
62 cd ..
63 ls
64 ls
65 cd ..
66 ls
67 ls
68 cd home
69 cd gcc
70 ls
71 gcc --version
72 cd gcc-build/
73 ls
74 rm -rf
75 ls
76 ls
77 rf -rf *
78 rm -rf *
79 ls
80 ../configure --prefix=/target/installation/path/gcc --disable-multilib --enable-languages=c,c++,fortran,jit --enable-checking=release --enable-host-shared --with-pic
81 apt install build-essential
82 gcc --version
83 dpkg -l | grep gcc | awk '{print $2}'
84 ../configure --prefix=/target/installation/path/gcc --disable-multilib --enable-languages=c,c++,fortran,jit --enable-checking=release --enable-host-shared --with-pic
85 make -j 6
86 make install
87 ls
88 apt remove gcc
89 apt remove gcc-11
90 apt-remove gcc-12
91 apt remove gcc-12
92 dpkg -l | grep gcc | awk '{print $2}'
93 gcc --version
94 sudo apt install build-essential
95 apt install build-essential
96 gcc --version
97 which mpicc mpicxx mpif90
98 ls
99 cd ..
100 ls
101 cd ..
102 ls
103 cd ..
104 ls
105 cd home
106 rm -rf gcc
107 gcc -version
108 gcc --version
109 curl -L -O http://www.mpich.org/static/downloads/4.0.2/mpich-4.0.2.tar.gz
110 tar -xf mpich-4.0.2.tar.gz -C .
111 mkdir mpich-4.0.2/gcc-build
112 cd mpich-4.0.2/gcc-build
113 ../configure --prefix=/target/installation/path/mpich-4.0.2 --enable-shared --enable-sharedlibs=gcc --enable-fast=O2 --enable-debuginfo --enable-totalview --enable-two-level-namespace CC=gcc CXX=g++ FC=gfortran F77=gfortran F90='' CFLAGS='' CXXFLAGS='' FFLAGS='-fallow-argument-mismatch' FCFLAGS='-fallow-argument-mismatch' F90FLAGS='' F77FLAGS=''
114 ls
115 cd ..
116 ls
117 cd ..
118 ls
119 cd ..
120 ls
121 cd home/
122 ls
123 rm mpich-4.0.2*
124 rm -rf mpich-4.0.2/
125 wget https://www.mpich.org/static/downloads/4.1.2/mpich-4.1.2.tar.gz
126 apt-get install wget
127 wget https://www.mpich.org/static/downloads/4.1.2/mpich-4.1.2.tar.gz
128 tar xfz mpich.tar.gz
129 tar xfz mpich-4.1.2.tar.gz
130 ls
131 cd mpich-4.1.2
132 fortran --version
133 which gfortran
134 ls
135 cd ..
136 ls
137 rm -rf *
138 ls
139 cd ..
140 ls
141 which gcc
142 cd /usr/bin/gcc
143 ls
144 cd home
145 wget https://www.mpich.org/static/downloads/4.1.2/mpich-4.1.2.tar.gz
146 tar xfz mpich-4.1.2.tar.gz
147 cd mpich-4.1.2
148 l
149 ls
150 ls
151 mkdir gcc-build
152 cd gcc-build/
153 ../configure --prefix=/target/installation/path/mpich-4.1.2 --enable-shared --enable-sharedlibs=gcc --enable-fast=O2 --enable-debuginfo --enable-totalview --enable-two-level-namespace CC=gcc CXX=g++ FC=gfortran F77=gfortran F90='' CFLAGS='' CXXFLAGS='' FFLAGS='-fallow-argument-mismatch' FCFLAGS='-fallow-argument-mismatch' F90FLAGS='' F77FLAGS=''
154 which gfortran
155 gfortran --version
156 fortran --verseion
157 fortran --version
158 apt-get install gfortran
159 ../configure --prefix=/target/installation/path/mpich-4.1.2 --enable-shared --enable-sharedlibs=gcc --enable-fast=O2 --enable-debuginfo --enable-totalview --enable-two-level-namespace CC=gcc CXX=g++ FC=gfortran F77=gfortran F90='' CFLAGS='' CXXFLAGS='' FFLAGS='-fallow-argument-mismatch' FCFLAGS='-fallow-argument-mismatch' F90FLAGS='' F77FLAGS=''
160 apt-get install python3
161 python-dev
162 apt-get install python-dev
163 apt-get install python-dev-is-python3
164 apt -get install cmake
165 apt-get install cmake
166 ../configure --prefix=/target/installation/path/mpich-4.1.2 --enable-shared --enable-sharedlibs=gcc --enable-fast=O2 --enable-debuginfo --enable-totalview --enable-two-level-namespace CC=gcc CXX=g++ FC=gfortran F77=gfortran F90='' CFLAGS='' CXXFLAGS='' FFLAGS='-fallow-argument-mismatch' FCFLAGS='-fallow-argument-mismatch' F90FLAGS='' F77FLAGS=''
167 make -j 6
168 make install
169 echo $CC $CXX $FC $F90 $F77
170 export CC=mpicc CXX=mpicxx FC=mpif90 F90=mpif90 F77=mpif77
171 vi ~/.bash_profile
172 apt-get install vim
173 vi ~/.bash_profile
174 cd ..
175 ls
176 cd ..
177 cd ..
178 cd ..
179 ls
180 cd etc/
181 ls
182 cd profile
183 vi profile
184 echo $CC $CXX $FC $F90 $F77
185 cd ~
186 ls
187 ls
188 ls
189 cd home
190 ls
191 cd ..
192 ls
193 cd home
194 ls
195 rm mpich-4.1.2.tar.gz
196 mkdir -p ~/projects
197 cd ~/projects
198 git clone https://github.com/idaholab/moose.git
199 cd moose
200 git checkout master
201 cd ~/projects/moose/scripts
202 export MOOSE_JOBS=6 METHODS=opt
203 ./update_and_rebuild_petsc.sh
204 ./update_and_rebuild_libmesh.sh
205 ./update_and_rebuild_wasp.sh
206 apt-get install dpkg
207 apt-get install reinstall build-essential
208 apt-get install --reinstall build-essential
209 apt-get install -reinstall gcc
210 apt-get install --reinstall gcc
211 dpkg-reconfigure build-essential
212 dpkg-reconfigure gcc
213 apt install gobjc++
214 ./update_and_rebuild_petsc.sh --with-mpi-dir=/home/mpich-4.1.2/
215 which mpich
216 which mpi
217 mpi
218 mpich
219 mpicc
220 which mpicc
221 mpicc --version
222 ls
223 ls
224 cd ..
225 ls
226 cd ..
227 cd ..
228 ls
229 cd ..
230 ls
231 cd ~
232 ls
233 cd home
234 ls
235 cd ~
236 cd ..
237 cd home
238 ls
239 pwd
240 ls
241 cd ..
242 ls
243 cd ~
244 ls
245 pwd
246 ls
247 cd ~/home
248 cd ..
249 ls
250 cd ..
251 ls
252 cd home
253 ls
254 rm -rf *
255 cd ~
256 which mpi
257 wget https://www.mpich.org/static/downloads/4.1.2/mpich-4.1.2.tar.gz
258 ls
259 ls
260 tar mpich-4.1.2.tar.gz
261 tar -zxvf mpich-4.1.2.tar.gz
262 ls
263 rm mpich-4.1.2.tar.gz
264 cd mpich-4.1.2/
265 mkdir gcc-build
266 ls
267 cd gcc
268 cd gcc-build/
269 ../configure --prefix=/target/installation/path/mpich-4.1.2 --enable-shared --enable-sharedlibs=gcc --enable-fast=O2 --enable-debuginfo --enable-totalview --enable-two-level-namespace CC=gcc CXX=g++ FC=gfortran F77=gfortran F90='' CFLAGS='' CXXFLAGS='' FFLAGS='-fallow-argument-mismatch' FCFLAGS='-fallow-argument-mismatch' F90FLAGS='' F77FLAGS=''
270 make -j 6
271 make install
272 ls
273 cd ..
274 ls
275 cd ..
276 ls
277 echo $CC $CXX $FC $F90 $F77
278 ls
279 cd projects/moose/
280 ls
281 cd /projects/moose/scripts
282 export MOOSE_JOBS=6 METHODS=opt
283 ./update_and_rebuild_petsc.sh
284 ./update_and_rebuild_libmesh.sh
285 ./update_and_rebuild_wasp.sh
286 ~
287 ls
288 ./update_and_rebuild_petsc.sh --with-mpi-dir=/mpich-4.1.2/
289 cd ~/mpich-4.1.2/
290 ls
291 which mpi
292 find /usr -name "mpich"
293 rm -rf mpich-doxygen.in
294 ls
295 cd ..
296 rm -rf mpich-4.1.2/
297 apt install openmpi
298 apt-get install openmpi
299 apt-get install mpich
300 ./update_and_rebuild_petsc.sh
301 cd projects/moose/
302 ./update_and_rebuild_petsc.sh
303 ls
304 pwd
305 cd scripts/
306 ./update_and_rebuild_petsc.sh
307 apt-get install ptscotch
308 apt-get install flex
309 ./update_and_rebuild_petsc.sh
310 ./update_and_rebuild_petsc.sh --download-bison
311 ./update_and_rebuild_libmesh.sh
312 ./update_and_rebuild_wasp.sh
313 cd ~/projects/moose/test
314 make -j 6
315 apt-get install yaml
316 pip install yaml
317 apt-get install python3-pip
318 pip install yaml
319 pip install PyYAML
320 apt-get install packaging
321 pip install packaging
322 make -j 6
323 cd ~/projects/moose/test
324 ./run_tests -j 6
325 ls
326 cd tests
327 ls
328 cd ..
329 ls
330 pwd
331 ./run_tests -j 6
332 ./run_tests -j46
333 ./run_tests -j 4
334 ./run_tests -j
335 ./run_tests -j 1
336 add-apt-repository ppa:deadsnakes/ppa
337 pip install moose-tools
338 apt-get install moose-tools
339 ./run_tests -j 1
340 lös
341 ls
342 logout
343 exit
344 ls
345 exit
346 ./run_tests -j 1
347 ls
348 cd projects/moose/test/
349 ./run_tests -j 1
350 apptainer inspect --deffile yourcontainer.sif
351 exit
352 apptainer inspect --deffile yourcontainer.sif
353 exit
354 exit
355 cat ~./bash_history
356 history
Singularity>

[marinsiebert]

terminal_output.txt

This is the terminal output during the installation process until the first error regarding some issue with fortran-compiler. This was the point where I switched to using the build-essential gcc version as I was unable to fix the error.