WIP: Sanitizer build with Ubuntu 20.04 #3636

RudolfWeeber · 2020-04-06T07:29:32Z

No description provided.

codecov · 2020-04-06T07:58:23Z

Codecov Report

Merging #3636 into python will increase coverage by 0%.
The diff coverage is 84%.

@@           Coverage Diff            @@
##           python   #3636     +/-   ##
========================================
  Coverage      88%     88%             
========================================
  Files         524     523      -1     
  Lines       23471   22446   -1025     
========================================
- Hits        20658   19845    -813     
+ Misses       2813    2601    -212

Impacted Files	Coverage Δ
src/core/TabulatedPotential.hpp	`100% <ø> (ø)`
src/core/bond_error.hpp	`0% <0%> (ø)`
src/core/bonded_interactions/angle_cosine.hpp	`100% <ø> (ø)`
src/core/bonded_interactions/angle_cossquare.hpp	`100% <ø> (ø)`
src/core/bonded_interactions/angle_harmonic.hpp	`100% <ø> (ø)`
...re/bonded_interactions/bonded_interaction_data.cpp	`91% <ø> (+<1%)`	⬆️
src/core/electrostatics_magnetostatics/mmm1d.cpp	`94% <ø> (-1%)`	⬇️
src/core/grid_based_algorithms/halo.cpp	`63% <ø> (+<1%)`	⬆️
src/core/grid_based_algorithms/lb.cpp	`96% <ø> (-1%)`	⬇️
src/core/immersed_boundary/ImmersedBoundaries.hpp	`100% <ø> (ø)`
... and 198 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8f105e3...0ac6628. Read the comment docs.

KaiSzuttor · 2020-04-06T11:14:54Z

you can try locally by:
docker run -it docker.pkg.github.com/espressomd/docker/ubuntu-20.04:a1192b35590a1a474c55fe1e9a1e6c25758454ea bash

jngrad · 2020-04-06T11:53:11Z

2382 /builds/espressomd/espresso/build/src/python/espressomd/collision_detection.cpp:10435:5: error: 'tp_print' is deprecated [-Werror,-Wdeprecated-declarations]
2383     0,
2384     ^
2385 /usr/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
2386     Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
2387     ^
2388 /usr/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
2389 #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))

To resolve that issue, it seems we need to either downgrade Cython 0.29.14 to 0.29.13, or add the compiler flag -Wno-deprecated-declaration (pandas-dev/pandas#30609 (comment))

RudolfWeeber · 2020-04-06T13:03:12Z

docker run -it docker.pkg.github.com/espressomd/docker/ubuntu-20.04:a1192b35590a1a474c55fe1e9a1e6c25758454ea bash

I’m aware of that. The deprecation stuff didn’t happen, locally, for whatever reason.

RudolfWeeber · 2020-04-06T18:02:21Z

Boost MPI again, it seems:
https://gitlab.icp.uni-stuttgart.de/espressomd/espresso/-/jobs/221196

fweik · 2020-04-06T19:42:47Z

Yup, the offending line is /usr/include/boost/mpi/detail/binary_buffer_iprimitive.hpp:44, which calls front() on std::vector, which is undefined if the vector is empty.

fweik · 2020-04-06T19:43:18Z

It's also trivial to fix, which doesn't help because we don't control the code.

RudolfWeeber · 2020-04-06T20:03:23Z

It's also trivial to fix, which doesn't help because we don't control the code.

Suppression file it is, then.

fweik · 2020-04-06T20:05:43Z

I don't know. The sanitizer says this is a nullpointer dereference, so this might also segfault

fweik · 2020-04-06T20:07:50Z

Ah, wait a second. The buffer is passed into the class, maybe it can be fixed from the outside...

fweik · 2020-04-06T20:13:08Z

No, it can't. This was also introduced quite recently. We should really ship our own version of boost mpi.

.gitlab-ci.yml

Co-Authored-By: Jean-Noël Grad <jgrad@icp.uni-stuttgart.de>

KaiSzuttor · 2020-04-07T14:02:52Z

.gitlab-ci.yml

+  script:
+    - export myconfig=maxset with_cuda=false with_coverage=false with_static_analysis=true with_asan=true with_ubsan=true test_timeout=900 ASAN_OPTIONS="allocator_may_return_null=1" OMPI_CC=clang-9 OMPI_CXX=clang++-9 CC=clang-9 CXX=clang++-9 UBSAN_OPTIONS=suppressions=${CI_PROJECT_DIR}/maintainer/CI/ubsan.supp


Suggested change

script:

- export myconfig=maxset with_cuda=false with_coverage=false with_static_analysis=true with_asan=true with_ubsan=true test_timeout=900 ASAN_OPTIONS="allocator_may_return_null=1" OMPI_CC=clang-9 OMPI_CXX=clang++-9 CC=clang-9 CXX=clang++-9 UBSAN_OPTIONS=suppressions=${CI_PROJECT_DIR}/maintainer/CI/ubsan.supp

variables:

myconfig: 'maxset'

with_cuda: 'false'

with_coverage: 'false'

with_static_analysis: 'true'

with_asan: 'true'

with_ubsan: 'true'

test_timeout: '900'

ASAN_OPTIONS: 'allocator_may_return_null=1'

OMPI_CC: 'clang-9'

OMPI_CXX: 'clang++-9'

CC: 'clang-9'

CXX: 'clang++-9'

UBSAN_OPTIONS: 'suppressions=${CI_PROJECT_DIR}/maintainer/CI/ubsan.supp'

script:

KaiSzuttor

use env vars

jngrad · 2020-04-09T14:33:21Z

@RudolfWeeber This PR looks ready to me. What is still missing?
@KaiSzuttor We should delay your proposed env var changes to avoid conflicts with #3582, or only apply it to the Ubuntu 20.04 job.

…thon

NVIDIA/thrust#1032

…ython

jngrad · 2020-04-09T16:13:29Z

src/core/cuda_wrapper.hpp

@@ -22,6 +22,7 @@
 #if defined(__CUDACC__)

 #include <cuda.h>
+#include "clang_thrust_workaround.cuh"


The file clang_thrust_workaround.cuh seems to be missing. Also, I just found out the issue was fixed in thrust 1.9.5 (I'm writing a report for them)

Suggested change

#include "clang_thrust_workaround.cuh"

#if THRUST_VERSION < 100905

#include "clang_thrust_workaround.cuh"

#endif

Here is my investigation: NVIDIA/thrust#1032 (comment); if CUDA 10.1 is shipped with thrust 1.9.5, we should install it on ICP machines and drop support for the combination Clang+CUDA10.0 is ESPResSo.

@RudolfWeeber sorry, I forgot that editing a message doesn't send an email. In my previous message, I meant to say that you probably forgot to commit file clang_thrust_workaround.cuh.

fweik · 2020-04-09T17:04:37Z

For the record: I think the workaround is a bad idea.

jngrad · 2020-04-09T17:08:13Z

I'm also not sure if it can be done like this from the ESPResSo side. When I tinkered with that solution, I found the only way to get it to work was to apply it directly to the thrust code, but maybe I'm wrong.

RudolfWeeber · 2020-04-09T19:24:11Z

It looks like the workaround is already present in the OSX container...

Let's make a decision about how to proceed in the ES meeting.

jngrad · 2020-04-15T09:24:09Z

offline core team discussion: if Clang 8.0 can build with thrust 1.9.3 keep it, otherwise add thrust 1.9.5 as a submodule.

RudolfWeeber · 2020-04-15T12:26:33Z

Core team discussion yielded that @jngrad will take over this PR.

Sanitizers work with Clang9.
At the ES meeting, it was decided to use Clang 8 and the latest GCC versions in the image and at the ICP rather than patching system headers to allow compilation with Cuda and more recent compilers
The thrust workaround mentioned above might have to be removed again when building with Clang-8. If it is still needed, we decided to ship a new enough thrust with Esprseso as git submodule for the time being.

I'll post Docker changes to the Ubuntu 20 container separately.

Fixes #3627, fixes #3650, closes #3636 Description of changes: - run Clang-Tidy, ASAN, UBSAN in the Ubuntu 20.04 image - re-enable Clang-Tidy on CUDA code - fix Clang-Tidy 9 warnings - replace `boost::optional` by custom code in CUDA sources - remove unused code

RudolfWeeber added 4 commits April 6, 2020 09:24

CMake: Exclude some boost MPI from clang-tidy

bc403cb

CI: Add Ubuntu 20.04 build

eb2164e

Formatting

79638aa

CI: Disable Cuda on Ubuntu 20.04 job

c979dff

RudolfWeeber force-pushed the u2004 branch from 46f0d56 to 462556e Compare April 6, 2020 09:05

CI: Ubuntu 20.04 select clang

fc393d0

RudolfWeeber force-pushed the u2004 branch from 462556e to fc393d0 Compare April 6, 2020 10:42

RudolfWeeber force-pushed the u2004 branch from 4b7f21c to 34ed5ea Compare April 6, 2020 13:52

Cython: build with -Wno-deprecated-declarations

c1beb3d

RudolfWeeber force-pushed the u2004 branch from 34ed5ea to c1beb3d Compare April 6, 2020 16:42

RudolfWeeber added 2 commits April 7, 2020 13:04

UBSAN: Suppressions file

c8b988c

CI: Use UB Sanitizer suppressions in Ubuntu 20.04 job

3b4eb1e

jngrad requested changes Apr 7, 2020

View reviewed changes

.gitlab-ci.yml Outdated Show resolved Hide resolved

RudolfWeeber and others added 2 commits April 7, 2020 15:38

CI: Path to UBSAN supressions

2657aff

Co-Authored-By: Jean-Noël Grad <jgrad@icp.uni-stuttgart.de>

Merge branch 'python' into u2004

9eab049

KaiSzuttor reviewed Apr 7, 2020

View reviewed changes

fweik assigned jngrad Apr 9, 2020

RudolfWeeber added 4 commits April 9, 2020 17:01

Merge branch 'python' of git://github.com/espressomd/espresso into py…

a88e7bd

…thon

Workaround Thrust compilation with Clang bug

a6f5040

NVIDIA/thrust#1032

CI: Enable CUDA on Ubuntu 20.04 sanitizer build

1954401

Merge branch 'u2004' of ssh://github.com/RudolfWeeber/espresso into p…

8af479d

…ython

jngrad reviewed Apr 9, 2020

View reviewed changes

RudolfWeeber added 2 commits April 9, 2020 18:31

Missing file

12dcf03

Include file sorting

246afd4

RudolfWeeber force-pushed the u2004 branch from 66cfe44 to 246afd4 Compare April 9, 2020 16:47

RudolfWeeber added 2 commits April 9, 2020 21:16

CI: 2h time out for Ubuntu 20 sanitizer job

1c8c37d

Clang+Thrust: Guard against double presence of workaround

0ac6628

jngrad mentioned this pull request Apr 24, 2020

Move Clang-Tidy and sanitizers to Clang 9 in Ubuntu 20.04 #3684

Merged

kodiakhq bot closed this in #3684 Apr 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Sanitizer build with Ubuntu 20.04 #3636

WIP: Sanitizer build with Ubuntu 20.04 #3636

RudolfWeeber commented Apr 6, 2020

codecov bot commented Apr 6, 2020 •

edited

Loading

KaiSzuttor commented Apr 6, 2020

jngrad commented Apr 6, 2020

RudolfWeeber commented Apr 6, 2020 via email

RudolfWeeber commented Apr 6, 2020

fweik commented Apr 6, 2020

fweik commented Apr 6, 2020

RudolfWeeber commented Apr 6, 2020 via email

fweik commented Apr 6, 2020

fweik commented Apr 6, 2020

fweik commented Apr 6, 2020 •

edited

Loading

KaiSzuttor Apr 7, 2020 •

edited

Loading

KaiSzuttor left a comment

jngrad commented Apr 9, 2020

jngrad Apr 9, 2020 •

edited

Loading

jngrad Apr 9, 2020

jngrad Apr 9, 2020

fweik commented Apr 9, 2020

jngrad commented Apr 9, 2020

RudolfWeeber commented Apr 9, 2020

jngrad commented Apr 15, 2020

RudolfWeeber commented Apr 15, 2020

		script:
		- export myconfig=maxset with_cuda=false with_coverage=false with_static_analysis=true with_asan=true with_ubsan=true test_timeout=900 ASAN_OPTIONS="allocator_may_return_null=1" OMPI_CC=clang-9 OMPI_CXX=clang++-9 CC=clang-9 CXX=clang++-9 UBSAN_OPTIONS=suppressions=${CI_PROJECT_DIR}/maintainer/CI/ubsan.supp

-  script:
-    - export myconfig=maxset with_cuda=false with_coverage=false with_static_analysis=true with_asan=true with_ubsan=true test_timeout=900 ASAN_OPTIONS="allocator_may_return_null=1" OMPI_CC=clang-9 OMPI_CXX=clang++-9 CC=clang-9 CXX=clang++-9 UBSAN_OPTIONS=suppressions=${CI_PROJECT_DIR}/maintainer/CI/ubsan.supp
+   variables:
+     myconfig: 'maxset'
+     with_cuda: 'false'
+     with_coverage: 'false'
+     with_static_analysis: 'true'
+     with_asan: 'true'
+     with_ubsan: 'true'
+     test_timeout: '900'
+     ASAN_OPTIONS: 'allocator_may_return_null=1'
+     OMPI_CC: 'clang-9'
+     OMPI_CXX: 'clang++-9'
+     CC: 'clang-9'
+     CXX: 'clang++-9'
+     UBSAN_OPTIONS: 'suppressions=${CI_PROJECT_DIR}/maintainer/CI/ubsan.supp'
+  script:

WIP: Sanitizer build with Ubuntu 20.04 #3636

WIP: Sanitizer build with Ubuntu 20.04 #3636

Conversation

RudolfWeeber commented Apr 6, 2020

codecov bot commented Apr 6, 2020 • edited Loading

Codecov Report

KaiSzuttor commented Apr 6, 2020

jngrad commented Apr 6, 2020

RudolfWeeber commented Apr 6, 2020 via email

RudolfWeeber commented Apr 6, 2020

fweik commented Apr 6, 2020

fweik commented Apr 6, 2020

RudolfWeeber commented Apr 6, 2020 via email

fweik commented Apr 6, 2020

fweik commented Apr 6, 2020

fweik commented Apr 6, 2020 • edited Loading

KaiSzuttor Apr 7, 2020 • edited Loading

Choose a reason for hiding this comment

KaiSzuttor left a comment

Choose a reason for hiding this comment

jngrad commented Apr 9, 2020

jngrad Apr 9, 2020 • edited Loading

Choose a reason for hiding this comment

jngrad Apr 9, 2020

Choose a reason for hiding this comment

jngrad Apr 9, 2020

Choose a reason for hiding this comment

fweik commented Apr 9, 2020

jngrad commented Apr 9, 2020

RudolfWeeber commented Apr 9, 2020

jngrad commented Apr 15, 2020

RudolfWeeber commented Apr 15, 2020

codecov bot commented Apr 6, 2020 •

edited

Loading

fweik commented Apr 6, 2020 •

edited

Loading

KaiSzuttor Apr 7, 2020 •

edited

Loading

jngrad Apr 9, 2020 •

edited

Loading