-
Notifications
You must be signed in to change notification settings - Fork 28
Flatiron Quickstart
Talk slides (PDF) ChaNGAGasolineFlatiron2022.pdf
This is a Quick Start guide to support the Gasoline/ChaNGa tutoral on June 28, 2022 at the NBSE workshop at the Flatiron Institute. It assumes that you are running on the VM supplied and documented by Phil Chang.
ChaNGa is built on the Charm++ parallel library and runtime system. Version 7.0.0 is assumed for this workshop. It can be downloaded directly from the Charm group's source archive using the Git version control system.
Check out the latest development version of Charm++ from the repository:
$ git clone https://github.com/UIUC-PPL/charm
This will create a directory named charm. Move to this directory:
$ cd charm
To obtain the current stable release, 7.0.0, checkout the v7.0.0 tag:
$ git checkout v7.0.0
And now build Charm with the netlrts-linux machine layer:
$ ./build ChaNGa netlrts-linux-x86_64 --with-production
Note: the ''ChaNGa'' argument in the above build script is an instruction to build all the charm++ libraries on which ChaNGa depends. ChaNGa itself is built in the next step. C++ is compute intensive to compile; appending a ''-jN'' where N is the number of cores on your system can significantly speed up the build process by compiling in parallel.
ChaNGa itself can also be downloaded from the N-Body Shop source archive using ''git''. ChaNGa depends on a utility module which also needs to be downloaded.
git clone https://github.com/N-BodyShop/changa.git git clone https://github.com/N-BodyShop/utility.git
This will create a directories named changa and utility. Move to the changa directory:
$ cd changa
Now configure ChaNGa for the science you are simulating. ChaNGa is always able to do simpler cases (e.g. dark matter only cosmology) based on the .param (parameter file) used to start the job. We will additionally enable the galaxy formation code that would be suitable for cosmological galaxies and/or the AGORA isolated galaxy disk and use single executable for all tests. We use GASOLINE2 SPH (hydrodynamics): Geometric Density Force, Diffusion, dt Adjusting and the Wendland kernel, the metal cooling module and superbubble feedback (note that Geometric Densisty Forces, Diffusion, and dt Adjusting are enabled by default, you do not need to use --enable-diffusion --enable-dtadjust --enable-gdforce )
./configure --enable-cooling=metal --enable-superbubble --enable-wendland
A summary of the compile flags will be displayed when configure is finished. Look out for ''WARNING''s which usually indicate a mis-typed option. Now compile:
make
Again, a ''-j'' option will rung the compile in parallel. The result will be two executables: ''charmrun'' and ''ChaNGa''. ''charmrun'' is a helper program to start all parallel instances of ''ChaNGa'' and start them communicating with each other.
The changa source directory contains a number of basic tests. These include a Sod shock tube, ''testshock'', a King model, ''teststep'', an Evrard collapse, ''testcollapse'' and a low resolution cosmological volume, ''testcosmo''.
As an example, run the ''teststep'' King model by changing directories.
cd teststep
and running changa in parallel, in this example on 4 processors.
../charmrun +p 4 ++local ../ChaNGa test_pg.param
The ''++local'' flag tells charmrun to run all 4 processes on the local machine. Otherwise it will attempt to ssh to machines, including the localhost, to start the parallel processes.
cd ../testcosmo ../charmrun +p 4 ++local ../ChaNGa cube300.param
Tipsy is a program that can be used to quickly visualize ChaNGa/Gasoline output. To build it, additional development libraries are needed. Install them with the following:
sudo apt-get install libx11-dev sudo apt-get install libxt-dev sudo apt-get install libxaw7-dev sudo apt-get install libncurses5-dev
Obtain Tipsy using ''git'':
git clone https://github.com/N-BodyShop/tipsy.git
Move to the tipsy code directory:
cd tipsy/code
Configure and make:
./configure; make
This will build an executable ''tipsy''. Move this executable to where you can easily find it. To quickly visualize the ''teststep'' I.C.s:
$ tipsy openb king_soft.bin loads 0 zall
Use the left mouse button to zoom in and the center mouse button to zoom out.
git clone https://github.com/N-BodyShop/flatiron-tests.git
Suggested first test is evrardcollapse. cd into the evrardcollapse directory and run using the ChaNGA executable you built previously:
cd evrardcollapse path/charmrun +p 4 ++local path/ChaNGa evrardcollapse >& screen &
Note that path is the path to wherever you put ChaNGa.
Using the screen output option saves the output for later viewing. This is typically done automatically on supercomputer systems.
You can see a radial profile of the collapse final output (step 30) using
python plot_collapse.py
If you have pynbody successfully installed. Typically this can be added to your python3 install with
pip install pynbody
You can also look at this output with tipsy and spin it around.
A quick tool to examine outputs at the command line is printtipsy which is provided in the flatiron-tests repository. It can be compiled with
cd printtipsy gcc printtipsy.c -lm -o printtipsy
printtipsy autodetects the format and provides a brief summary of the file contents. It is useful for checking ICs (e.g. ones you make yourself).
pytipsy is a module that reads and writes tipsy format files (and autodetects the format). It is useful for making ICs yourself. If you have pytipsy.py (also provided in the above reposity) in your PYTHONPATH (e.g. the current directory or a place you keep modules) it can be used automatically. You can try it out with the cosmo16 example. You will need to have python working to do this.
cd cosmo16 python make16cubed.py
This creates a 16x16x16 particle dark matter plus gas IC in tipsy format that obeys the units rules for Gasoline/ChaNGA. It is just a simple grid created in python and saved using pytipsy.
Have a look at it with printtipsy
printtipsy -prop cosmo16cubed.std printtipsy -g -prop cosmo16cubed.std printtipsy -d -prop cosmo16cubed.std
The total mass in the box is Omega matter following the units rules. The -g gives the gas mass (Omega_gas) and -d the dark mass (Omega_DM). Since we did not perturb the positions no structure develops so this box evolves like a piece of the IGM when simulated.
path/charmrun +p 4 ++local path/ChaNGa cosmo16.param >& screen &
This parameter file is set-up to run cosmo16cubed.std for the age of the universe with 1024 steps. The gas heats and ionizes in response to the cosmic UV background (Haardt + Madau 2012). You can view the behaviour with a python script:
python plot_cosmo16.py
Note that the param file specifies outputs every 16 steps (~ 200 Myr) which is a long time at high redshift. Specifying a .red file with the same name as the output files (AchOutName = cosmo16) like cosmo16.red allows you to indicate redshifts you also want outputs for. ChaNGa/Gasoline will make a best effort to use outputs close to these redshifts. They are also listed in the .log file at runtime.
sod_shocktube ---- relaxed glass sod shock tube (cf. Gasoline2 paper and others)
glass16 ---- example script to make glass ICs
blob ---- test3 of Wengen (Agertz et al. 2007) cold blob in wind tunnel (low res)
AGORA ---- Low resolution AGORA galaxy IC with step 100 (100 Myr) output (Kim et al. 2016)
The blob and the AGORA test are more expensive to run (on a laptop at least, not really on any decent multi-core system). Outputs are provided that you can look at with tipsy or otherwise. The AGORA output can be compared to the results from Kim et al. 2016 which use an ineffective feedback (Thermal energy dumps or no feedback).
You can look at step 100 from AGORA with
python plot_galaxy.py
Note that for superbubble, we plot 'Tinc' not 'temp' to see the average temperature of the hot and cold phases. 'temp' is just the cold phase temperature for two phase particles (or temperature for single phase). See the colormap in the first figure. The galaxy is only partially relaxed at this time after an initial burst of star formation (see second figure). Note that the last point in the histogram is low due to binning.