Docker and WMCore

Eric has started building some docker images that are stored on Docker Hub. The Dockerfiles (instructions on how to build the images) are stored in a DMWM repository: https://github.com/dmwm/Docker. Notice how wmcore_test depends on wmcore_base which in turn depends on cms_grid. To build your own copy of these images, just check out the repository, cd into one of the directories and:

docker build .; docker tag `docker images -q | head -1` [your label]

which just gives you an easy way to refer to the image later. You can then start a container with

docker run -it --entrypoint /bin/bash

If you are looking for the wmcore_test container or derive your own container from it, it comes with a file system setup, but no services running. After you start the container you do

source env_unittest.sh 
$manage start-services

inside the container to have something with CouchDB and MariaDB running. From there you can run tests, etc. With some changes, you may be able to run the entire agent stack.

The dmwm/Docker GitHub repository is mirrored to CERN gitlab at https://gitlab.cern.ch/CMSDOCKS/dmwm and changes to GitHub will, after an hour or so, trigger a build of the image(s) at CERN. You can get these copies of the images with commands like

docker pull gitlab-registry.cern.ch/cmsdocks/dmwm:wmcore_test
docker run gitlab-registry.cern.ch/cmsdocks/dmwm:wmcore_test

Install python2 packages via pip

In case we want to test new python packages from the WMAgent docker container under:

gitlab-registry.cern.ch/cmsdocks/dmwm:wmcore_tests

we need to first have access to the root user account, which can be done with an extra -u option, such as:

docker run -u 0 ...

This executes the container with the root account (ID=0). Next we source the agent environment:

[root@59a797f0f172 ~]# source env_unittest.sh 

to load our WMCore python shipped within the container.

Now it's time to install pip on this container, from the instructions found here: https://www.liquidweb.com/kb/how-to-install-pip-on-centos-7/

we can download this script:

[root@59a797f0f172 Utils_t]# curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"

and execute it as:

[root@59a797f0f172 Utils_t]# python get-pip.py

This installs pip in the same python path as all the other python libraries we have in the container:

...
Installing collected packages: pip, wheel
Successfully installed pip-20.1.1 wheel-0.34.2
[root@59a797f0f172 Utils_t]# which pip
/home/dmwm/unittestdeploy/wmagent/1.3.6/sw/slc7_amd64_gcc630/external/python/2.7.13-comp/bin/pip

And we should be all set by now. So proceed and install whatever python packages you want, e.g.:

[root@59a797f0f172 Utils_t]# pip install coverage==4.4.1
...
[root@59a797f0f172 Utils_t]# pip install funcsigs==1.0.2
...

Install python3 packages via pip

In order to install pip3 in our python3 docker container, you can run the container as usual (no need for root) and source the usual environment script:

source env_unittest_py3.sh 

then you can install pip with the following command (which will be using the python3 binary shipped with WMCore, currently 3.8.2):

python3 -m ensurepip

If you get a cryptography/openssl error, you need to first set this environment variable:

export CRYPTOGRAPHY_ALLOW_OPENSSL_102=1

and run again the python3 -m ensurepip command. It should succeed this time.

Now you can go ahead and install whatever python3 packages you would like to, e.g.:

pip3 install rucio-clients==1.25.5

In case you want to check at which location it was installed, you can check it with:

pip3 show rucio-clients

NOTE: you might have to update LD_LIBRARY_PATH, PATH and PYTHONPATH environment variables.