Skip to content

Commit 1f93fac

Browse files
fmoessbauerfuchsto
fmoessbauer
authored and
fuchsto
committed
Sup 76 cidoc (#84)
* Added basic description of DASH's CI scripts * added documentation of docker containers * added documentation of circeci and travis
1 parent 978ff2f commit 1f93fac

File tree

2 files changed

+157
-0
lines changed

2 files changed

+157
-0
lines changed

‎doc/CI/index.md

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# Continuous Integration
2+
Taking care of the various differences in HPC software and architectures is not an easy task. For a seamless integration of both new features and bugfixes DASH provides support for continuous integration (CI).
3+
If CI passes, there is a high change that your changes do not hurt other parts of the code. However always keep in mind that a CI is only able to falsify but not to verify code.
4+
5+
## Unit Tests
6+
For unit testing DASH uses the widely used GoogleTest framework.
7+
For each feature of DASH there should be a corresponding unit test `dash/test`. It is recommended that the developer of the feature does not write the tests himself.
8+
Whether the tests should be build is set in the build script using `-DBUILD_TESTS=ON`. The testsuite can then be easily executed using `mpirun -n <procs> ./build/bin/dash-test-mpi`. The output should be self explaining.
9+
10+
### Code Coverage
11+
Code coverage tests are useful to verify if the whole public API is covered by the unit tests.
12+
13+
#### Dependencies
14+
15+
- gcc / g++
16+
- gcov
17+
- lcov
18+
19+
#### How to get the results?
20+
21+
``` {.bash}
22+
./build.cov.sh
23+
24+
cd build.cov
25+
# build tests
26+
make
27+
# execute coverage measurement and generate reports
28+
make coverage
29+
30+
# open results in browser
31+
firefox coverage/index.html
32+
```
33+
34+
If some tests do not work with just one unit, they can be excluded using
35+
the filter string. Multiple tests can be separated by “:”. The pattern
36+
is as follows:
37+
38+
``` {.bash}
39+
# export GTEST_FILTER="POSTIVE_PATTERNS[-NEGATIVE_PATTERNS]"
40+
41+
# Run all tests which start with S or L,
42+
# but not LocalRangeTest.LargeArray and not SUMMA tests
43+
export GTEST_FILTER="S*:L*:-LocalRangeTest.LargeArray:SUMMA*"
44+
45+
# more common example
46+
export GTEST_FILTER="-LocalRangeTest.LargeArray"
47+
```
48+
49+
#### CMake Bug
50+
51+
The best solution would be to use the coverage integration in cmake by
52+
setting `CMAKE\_BUILD\_TYPE=Coverage`. This is not possible as the
53+
coverage flags are inserted before the `-fopenmp` flag. In modern
54+
versions of gcc this is not allowed and leads to an linker error.
55+
56+
Currently `CMAKE\_BUILD\_TYPE=Profile` is used.
57+
58+
## CI Scripts
59+
60+
The DASH CI scripts located in `dash/scripts` automate the process of building DASH in various configuration and executing the tests.
61+
There are three main scripts:
62+
63+
- `dash-ci-deploy.sh`: deploys various DASH configurations.
64+
- `dash-test.sh`: runs a given target with a varying number of nodes and parses the output.
65+
- `dash-ci.sh`: calls `dash-ci-deploy.sh` and executes the tests using `dash-test.sh`.
66+
67+
### Deployment
68+
69+
Each CI configuration is identified by `$BUILD_TYPE`. If you intend to add another config, just add another case in the if/else statement.
70+
The `$BUILD_SETTINGS` holds the settings which are passed to cmake. When building, the build of each target is written to a folder named like the target.
71+
For example, the `Minimal` target creates a folder `dash-ci/Minimal` where dash is build and installed.
72+
73+
There are some environment variables that are used to modify the build settings. This is especially useful to customize or speedup the deployment if using the online CI providers.
74+
If you intend to add another environment variable always add a default value if it is not set.
75+
76+
- `$DASH_BUILDEX`: ("ON"|"OFF") specifies if the examples should be build in this deployment run.
77+
- `$DASH_MAKE_PROCS`: max number of parallelism in make. Limit this value to reduce memory consumption during compilation. If not set the number of available processors is used.
78+
79+
### Execute Tests
80+
In `dash-test.sh` the environment is checked (TODO DOC) and the tests are executed using a specified set of mpi processes per run. This is specified using `run_suite <nprocs>`.
81+
There are never more processes spin up than the host provides CPU cores. Furthermore some characteristics can be specified using environment variables:
82+
83+
- `$DASH_MAX_UNITS`: use at most this number of processes.
84+
- `$MPI_EXEC_FLAGS`: pass these flags to the mpirun command.
85+
86+
### Execute CI
87+
88+
Call `dash-ci.sh <target> ..` to run the CI for a list of targets. The target configuration has to be set in the deployment script. If no targets are given, a list of default targets is build and executed.
89+
If the logs of each target contain no errors, the CI return exit code 0, otherwise 1.
90+
91+
## CI Environments
92+
93+
To simulate different common environments DASH uses Docker containers. For further information on Docker see the [vendor documentation](https://docs.docker.com/). The containers are build using the dockerfiles located in `dash/scripts/docker-testing/<env>`. This can be done either locally or by using the pre build images located on [Dockerhub](https://hub.docker.com/u/dashproject/). The containers hosted on Dockerhub are automatically build from the official development branch of DASH.
94+
95+
### Build from Dockerfile
96+
97+
A docker container image named `dashproject/ci-testing:<env>` can be build using the following command. `env` has to be substituted with the name of the environment.
98+
99+
```
100+
# build container
101+
docker build --tag dashproject/ci-testing:<env> dash/scripts/docker-testing/<env>
102+
# alternatively pull official image
103+
docker pull dashproject/ci-testing:<env>
104+
```
105+
106+
Note
107+
108+
These containers can also be used as a good starting point for developing DASH applications.
109+
110+
As the containers only provide an environment but no DASH installation, a DASH repository should be mounted as shared folder.
111+
The following command starts an interactive container with DASH located in /opt/dash, assumed that the command is at the top level of the DASH repository.
112+
113+
```bash
114+
# mount current folder to /opt/dash
115+
docker run -it -v $(pwd):/opt/dash dashproject/ci-testing:<env>
116+
```
117+
118+
## Online CI providers
119+
120+
Currently we use the two online CI providers [Travis](https://travis-ci.org/dash-project/) and [CircleCI](https://circleci.com/gh/dash-project). The main testing logic is equal for both providers and follows this scheme:
121+
122+
1. pull docker image for specified env
123+
2. run tests inside docker
124+
3. analyze logfile for errors
125+
4. copy artefacts / print output messages
126+
127+
Note
128+
129+
As the exit code of a Docker container is not reliable, the output is parsed for errors outside docker. This is done by the `run-docker.sh` script.
130+
131+
To limit the complexity in the yml files, each ci folder (`dash/scripts/<ci>`) contains shell scripts for building / pulling and starting the docker containers. The file names should be self-explaining.
132+
In the `run-docker.sh` file are also the environment variables for the Docker container set. These are then accessed by the `dash-ci.sh` script which is executed inside the container.
133+
134+
The following command is taken from `travisci/run-docker.sh` and annotated to explain the basic logic:
135+
```bash
136+
docker run -v $PWD:/opt/dash dashproject/ci:$MPIENV /bin/sh -c "export DASH_MAKE_PROCS='4'; <...> sh dash/scripts/dash-ci.sh $BUILD_CONFIG | grep -v 'LOG =' | tee dash-ci.log 2> dash-ci.err;"
137+
# | | | | | |
138+
# mount DASH use container named ... execute sh as shell | execute CI script Do not ouput Log lines
139+
# set env variables
140+
```
141+
142+
### Travis
143+
144+
This provider is used for compilation tests (including examples) and supports direct checking of pull requests. Also the openmpi environment is tested. The commands are specified in `.travis.yml`.
145+
146+
### CircleCI
147+
148+
The main advantage of CircleCI is that multiple (up to 4) environments can be tested in parallel which reduces the test time significantly.
149+
Another advantage is that the xml output of googletest is parsed and the failed tests are presented nicely.
150+
The basic logic is specified in `circle.yml`.
151+
152+
The `run-docker.sh` file is worth taking a closer look at. In `$MPIENVS` a list of environments is specified. These are then evenly split over the available CircleCI instances. If more environments are specified than instances available, the containers are executed sequentally. This affects only the time for the CI to complete, but has no effect on the test results.
153+
154+
After the completion of each Docker container, the test results are gathered and copied to the artifacts directory. The directory/file structure is as follows: `<env>/<build_type>/dash-test-<nprocs>.xml`. For further information on CircleCI artifacts see the [official documentation](https://circleci.com/docs/build-artifacts/).

‎doc/index.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,6 @@
3333
- [Utilities](/Utilities)
3434

3535
- [Locality Hierarchies](/Utilities/LocalityHierarchies)
36+
37+
##Developer Notes
38+
- [Continuous Integration](/CI)

0 commit comments

Comments
 (0)