developer installation for the new version ?

[amoradinejad]

Hi Joe,

I have extended the old version of cosmosis to compute galaxy power spectrum and bispectrum. I wanted to upgrade to the new version, but I noticed that in the new version, the main cosmosis repository is not downloaded. For my extensions of cosmosis, I need to add C functions to read and write to data block specific formats. In the new version, how can I have access to the c_datablock.cc file? Should I just download the main cosmosis repository from github in addition to the cosmosis-standard library? Or is there a developer installation version?

Thanks in advance,
Azadeh

[joezuntz]

Yep, if you'd like to modify cosmosis itself instead of the cosmology library then you can clone it with:

git clone https://github.com/joezuntz/cosmosis

Then you can edit it as you wish and then install it with pip install . run from the top-level directory. You probably want to remove the existing installation, if you have one through conda, with conda remove cosmosis. Do let me know if you have any problems with this.

If you are just adding new functions and not modifying existing ones then it might be easier to just make a new library for them, and link to that as well as to the main cosmosis library. Although I see you're using C++ so that might not be straightforward.

[amoradinejad]

Many thanks for the info. I followed your suggestions to get the main cosmosis repository, and successfully pip installed. The changes I need to make to main cosmosis repository are rather minimal, including introducing new sections in datablock to hold linear and nonlinear galaxy statistics and new functions in c_datablock.cc and c_datablock.h to "put" and "get" new types of grids to the datablock. Successfully done.

I still have a problem, though, with how to install the standard library. In addition to the changes to datablock, I have written several modules for cosmosis-standard-library, which do the actual calculations of the galaxy stats. These modules are in C and are correctly interfaced with cosmosis. I thought that after installing the main cosmosis repo, I should clone the standard library git clone https://github.com/joezuntz/cosmosis-standard-library, add my modules to it and make it. But this didnt work. How should I do the configure? Previous edition of cosmosis included a setup-cosmosis files in cosmosis/config folder which I had to first source, which is not included in the new version.

Again, thanks in advance.

[joezuntz]

That's great on the first issue. On the second, these are the commands to build:

source cosomosis-configure
make

The first step is the equivalent of the setup script, and should be available if everything is built and installed correctly.

But you will also need to add the subdirectories containing your new modules to the SUBDIRS variable in the Makefile in the directory above them. That makes the build system descend into their directories.

For example, if your new module has a Makefile in structure/my_new_module/Makefile then you would change the last line in structure/Makefile to read SUBDIRS = crl_eisenstein_hu growth_factor FrankenEmu cosmic_emu my_new_module

[amoradinejad]

I tried to do source cosomosis-configure, but I get the error of cosomosis-configure: No such file or directory. So perhaps I am doing sth wrong still. Let me just put here the steps I am taking:

git clone https://github.com/joezuntz/cosmosis
cd cosmosis
pip install . 
git clone https://github.com/joezuntz/cosmosis-standard-library
source cosmosis-configure
make

perhaps I am in a wrong directory when doing source cosomosis-configure ?

[joezuntz]

Sorry, I had a typo above, it should say source cosmosis-configure (there was an extra letter in what I typed).

[amoradinejad]

I actually typed it correctly in my terminal, while above just repeated your type. So the error of source cosmosis-configure isnt due to the typo.

One question, on my machine I needed to do pip3 install, since I have both python 2.7 and 3.9, and the default pip is for python 2.7. This shouldnt cause any issue for cosmosis installation, right? Or shall I change something in the setup file to indicate that it should use my python3.9 and not the default python which is python 2.7?

[joezuntz]

Ah, okay, in that case can you show me the output of pip install .?

As long as you used pip3 then it should be okay.

[amoradinejad]

Here it is:

Azadeh-macbookpro:cosmosis azadeh$ pip3 install .
Defaulting to user installation because normal site-packages is not writeable
Processing /Volumes/Data/Documents/Git/cosmosis
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting future
  Using cached future-0.18.2.tar.gz (829 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: emcee in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from cosmosis==2.0.3) (3.1.1)
Requirement already satisfied: pybind11 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from cosmosis==2.0.3) (2.9.0)
Requirement already satisfied: pyyaml in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from cosmosis==2.0.3) (6.0)
Requirement already satisfied: dynesty in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from cosmosis==2.0.3) (1.2.2)
Requirement already satisfied: scipy in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from cosmosis==2.0.3) (1.7.3)
Requirement already satisfied: zeus-mcmc in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from cosmosis==2.0.3) (2.4.1)
Requirement already satisfied: matplotlib in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from cosmosis==2.0.3) (3.5.1)
Requirement already satisfied: threadpoolctl in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from cosmosis==2.0.3) (3.1.0)
Requirement already satisfied: numpy in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from cosmosis==2.0.3) (1.22.1)
Requirement already satisfied: cycler>=0.10 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->cosmosis==2.0.3) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->cosmosis==2.0.3) (4.28.5)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->cosmosis==2.0.3) (1.3.2)
Requirement already satisfied: packaging>=20.0 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->cosmosis==2.0.3) (21.3)
Requirement already satisfied: pillow>=6.2.0 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->cosmosis==2.0.3) (8.4.0)
Requirement already satisfied: pyparsing>=2.2.1 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->cosmosis==2.0.3) (3.0.4)
Requirement already satisfied: python-dateutil>=2.7 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from matplotlib->cosmosis==2.0.3) (2.8.2)
Requirement already satisfied: tqdm in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from zeus-mcmc->cosmosis==2.0.3) (4.64.0)
Requirement already satisfied: setuptools in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from zeus-mcmc->cosmosis==2.0.3) (60.5.0)
Requirement already satisfied: seaborn in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from zeus-mcmc->cosmosis==2.0.3) (0.11.2)
Requirement already satisfied: pytest in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from zeus-mcmc->cosmosis==2.0.3) (7.1.2)
Collecting sklearn
  Using cached sklearn-0.0.tar.gz (1.1 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: six>=1.5 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from python-dateutil>=2.7->matplotlib->cosmosis==2.0.3) (1.16.0)
Requirement already satisfied: attrs>=19.2.0 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from pytest->zeus-mcmc->cosmosis==2.0.3) (21.2.0)
Requirement already satisfied: py>=1.8.2 in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from pytest->zeus-mcmc->cosmosis==2.0.3) (1.11.0)
Requirement already satisfied: pluggy<2.0,>=0.12 in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from pytest->zeus-mcmc->cosmosis==2.0.3) (1.0.0)
Requirement already satisfied: tomli>=1.0.0 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from pytest->zeus-mcmc->cosmosis==2.0.3) (2.0.0)
Requirement already satisfied: iniconfig in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from pytest->zeus-mcmc->cosmosis==2.0.3) (1.1.1)
Requirement already satisfied: pandas>=0.23 in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from seaborn->zeus-mcmc->cosmosis==2.0.3) (1.4.2)
Requirement already satisfied: scikit-learn in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from sklearn->zeus-mcmc->cosmosis==2.0.3) (1.0.2)
Requirement already satisfied: pytz>=2020.1 in /opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from pandas>=0.23->seaborn->zeus-mcmc->cosmosis==2.0.3) (2021.3)
Requirement already satisfied: joblib>=0.11 in /Users/azadeh/Library/Python/3.9/lib/python/site-packages (from scikit-learn->sklearn->zeus-mcmc->cosmosis==2.0.3) (1.1.0)
Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
Using legacy 'setup.py install' for sklearn, since package 'wheel' is not installed.
Building wheels for collected packages: cosmosis
  Building wheel for cosmosis (pyproject.toml) ... done
  Created wheel for cosmosis: filename=cosmosis-2.0.3-py3-none-any.whl size=757420 sha256=cc1d3c75b1e4eaaa09e10891bd9c9ceb06da2a8dfe0ab0c9cd820dd833de7ea2
  Stored in directory: /private/var/folders/h5/srv8bmg173jbvynhq5yyjcy00000gn/T/pip-ephem-wheel-cache-_jyi8qe3/wheels/82/0e/17/7be6bf5b912596216cbe3509be530702eac59d0e4f971c6200
Successfully built cosmosis
Installing collected packages: sklearn, future, cosmosis
    Running setup.py install for sklearn ... done
    Running setup.py install for future ... done
Successfully installed cosmosis-2.0.3 future sklearn

[joezuntz]

Does the cosmosis command get installed? You might need to open a new terminal to get things working.

Also I notice that you're using the global installation, rather than a virtual environment. I would usually recommend the latter.

Finally, can I ask how you're installing dependencies like GSL?

[amoradinejad]

Thanks for the info. I managed to make some progress. I re-started from scratch, as you suggested installing within a virtual environment. To be more specific, I did the following:

git clone https://github.com/joezuntz/cosmosis
cd cosmosis
git clone https://github.com/joezuntz/cosmosis-standard-library
python3 -m venv ./env
source env/bin/activate
pip3 install camb astropy fitsio cython scikit-learn fast-pt
pip3 install .
source cosmosis-configure
cd cosmosis-standard-library
make

So now with source cosmosis-configure I get the message of Your shell is now configured to build the CosmoSIS standard library and I have the cosmosis command installed.

As of dependencies, I have everything installed with macports. Without exporting the paths as you described in https://cosmosis.readthedocs.io/en/latest/intro/installation.html, I get all kinds of compilation errors, ex. f77: No such file or directory implying that FC variable not pointing to gfortran. Once I import the paths, I mange to compile the standard library. Since it is common for many users to have macport installations, would it be possible to add in cosmosis/configure.py, an option for paths for macports installated packages as now there are homebrew and conda options?

[joezuntz]

That's great, glad it seems to be working for you! Do the demos work?

If you could send me the variables you needed to make MacPorts work I could include them - that would be very handy, thanks!

[amoradinejad]

I only checked demo1 and demo15 as test cases, which I could successfully run.

On a different note, I noticed in demo1.ini that you still have the halofit module listed, but in the new version of the standard-library, the boltzmann module does not include the halofit (and halofit_takahashi). Are they now located in another directory?

Here are the variables I had to set (the default installation path of macports is /opt/local)

export GSL_INC=/opt/local/include
export GSL_LIB=/opt/local/lib
export CFITSIO_LIB=/opt/local/lib
export CFITSIO_INC=/opt/local/include
export FFTW_LIBRARY=/opt/local/lib
export FFTW_INCLUDE_DIR=/opt/local/include
export LAPACK_LINK="-L/opt/local/lib -llapack -lblas"
export CXX=/opt/local/bin/g++
export CC=/opt/local/bin/gcc
export FC=/opt/local/bin/gfortran
export MPICC=mpicc
export MPICXX=mpicxx
export MPIFC=mpifort

One last question, if installing on a Linux cluster, where the dependencies are loaded as modules, do I need to do all the exports? In the previous version of the code, you had a setup-cosmosis configure file was taking care of setting the paths correctly, but in the new version I dont see where this is done.

[joezuntz]

That's great, glad it's working. In the newest version we just use the halofit versions that are built into camb, instead of a separate module. I should remove the old demo text.

The equivalent of the setup-cosmosis script is the source cosmosis-configure command (after setting up your virtual environment also) - if you put that in your cluster it should work.

Many thanks for the macports options - I will add a flag to use them in the configure script.

[joezuntz]

I think that closes this issue, but if you have any more questions please do re-open it.

[amoradinejad]

Many thanks, yes the issue is totally sorted out. Thanks for the prompt and step-by-step guidance.