Use matrix in CI for Linux builds

[kunaltyagi]

Your Environment

Your Environment

• Operating System and version: All CI

Context

Each Linux build has potential to be very similar to each other, except those based on CUDA availability (aka ubuntu:latest).

Expected Behavior

Using matrix for linux builds allows:

• DRY: no one likes repeated content
• Easy scaling: add multiple compiler versions and required library versions

[shrijitsingh99]

Could you expand more on what all specifically jobs we can use matrix for?
Some I think which would be useful would be:

1. Multiple compilers i.e. GCC and Clang
2. Various build types i.e. debug, release
3. We could target multiple compiler versions too, not sure on the specifics
4. Multiple library versions like boost, eigen etc., need to specify the versions though
   Any other ideas?

[kunaltyagi]

Multiple OS. Currently, we support only Ubuntu 16.04 and 19.10 (that too, independently). I'd like to have a single compile step for them. I believe only some flags should differ (-Werror for one).

Multiple compiler versions isn't a big priority as long as we cover the default one for an OS and the latest one a similar OS.

So in order of priority:

• multiple OS: at least all Ubuntu LTS and Ubuntu latest. Maybe Arch or some bleeding edge OS
• multiple compilers: GCC, clang (default only if a bleeding edge OS is present, else latest release too)
• build type: Debug, Release, Package (this I want to add somehow to allow for simple releases)
• build options: SSE, OpenMP, etc. ON and OFF. How to handle these (independently or together needs more discussion)
• libraries: Boost, VTK, Eigen, Flann, etc.: same approach as compilers: a default and a bleeding edge.

Of course, this is not an exhaustive list. For starting on this, I'd consider multiple compilers (default available) and build type (only Debug and Release) only.

PS: This isn't a good-first-issue.

[SergioRAgostinho]

Before you dive into this: are all these options even realistic given our limited number of jobs on Azure pipelines?

[kunaltyagi]

even realistic given our limited number of jobs

Sort of. Some of the jobs would likely be daily/on-push(merge) master only. Others are viable per PR. Currently we spend 1.75 hour per Ubuntu/Windows build which would mean a simple combo of GCC and clang default + 2 Ubuntu OS in one matrix build would need 2 slots. We have 7 slots available. The very basic would require all of them slots:

• 1 for Mac (1) X (1) (It's mac)
• 2 for Windows (x32, x64) X (MSVC)
• 4 for Ubuntu (LTS, default) X (GCC, clang)

This implies one push per 2 hours will be max (12 successful pushes a day, theoretically). This is currently more than what we have right now. Even with the help of students, we're working on 6 PR on average per day (10 in 24 hours has been the max in last week). A compile error results in shorter compiles so we can fit more (practically).

If we add more stuff, we'd have to pipeline it. Eg: format before everything else. I'm working on some things in my free time to help (git mtimes, optimized compilers) but they are a long-way in the making.

[shrijitsingh99]

I was able to implement some of the mentioned features here - (CI):

1. Multiple OS
2. Multiple Compilers (Haven't added latest release one, currently it just take default one)

Some issues:

1. Build type, I don't think we will be able to support anything currently besides Release due do to lack of disk space on Azure.
2. Build options are also possible but they will be taking to many jobs, so we will have to consider that aspect.
3. Currently Azure doesn't support cross-product matrix strategy, we could do something like this.

[kunaltyagi]

This looks nicer, but adds 2 more jobs on the CI. Let's do the following:

• Comment out the clang jobs (even though they are faster)
• Replace the 2 ubuntu yaml with one integrated yaml

The parametrized yaml file looks lit, but this should suffice for now.

[shrijitsingh99]

This looks nicer, but adds 2 more jobs on the CI. Let's do the following:

• Comment out the clang jobs (even though they are faster)
• Replace the 2 ubuntu yaml with one integrated yaml

The parametrized yaml file looks lit, but this should suffice for now.

Great, I will make these changes. Will create a draft PR after #3745 is merged.

@taketwo You had mentioned we had gotten 2 extra parallel jobs here. Is that still valid?

[taketwo]

Yes, we still have 12 jobs:

[kunaltyagi]

That's awesome. Let's still keep clang disabled. Let me think of some strategy to save time and keep different PR CI going in parallel.

@shrijitsingh99 Looking forwards to the PR

GitLab CI keeps looking better and better in comparison (except the lack of support for Mac)

[shrijitsingh99]

That's awesome. Let's still keep clang disabled. Let me think of some strategy to save time and keep different PR CI going in parallel.

@shrijitsingh99 Looking forwards to the PR

Great! I Will put it up, though some issues are there currently. Sudo access is disabled on the containers so I had to create and use my own docker image for this as clang was not installed on the one we used, so we will have to push the new images to Docker hub.

I really liked this idea and will automate the manual labor involved currently with docker images. If you guys give the go-ahead, I think I can get something like that to work. It would allow a lot of flexibility in the future and reduce work.

GitLab CI keeps looking better and better in comparison (except the lack of support for Mac)

I have used it, it pretty good, it has lots of features. They even provide the Gold tier to open-source projects.

• build options: SSE, OpenMP, etc. ON and OFF. How to handle these (independently or together needs more discussion)

This idea sounds great. What did you have in mind for this?

#3594 will help save some time. I will try and incorporate that too.

[kunaltyagi]

Sudo access is disabled on the containers

I'm sorry I didn't quite understand this. You have full access inside a container (without using sudo)

we will have to push the new images to Docker hub

taketwo and I need to sit together and figure out why I'm unable to create the get access and then I can start playing with the pipelines. Then we can create a pipeline to auto-update the docker images.

What did you have in mind for this?

Nothing much, since it's not a big issue right now (and personally a lower priority for me).

I know it'd be quite important once we start adding OpenMP and SSE more uniformly across PCL. For example, if we implement my unified API using executor proposal, we'd need compile time tests to verify the flow under different conditions. I haven't thought yet how to manage that. The combinations are already at 8 (AVX, SSE, OpenMP: On and Off) and growing exponentially. I feel this would require a combination of refactoring, CMake and CI to achieve any sane compilation times. On top of my head:

• extract the AVX, SSE, OpenMP and related functionality in different files. That'd allow ccache to help speed up between different builds
• Use CMake to allow SSE flags to be modified independently (they are disabled in Release, IIRC) etc...

[shrijitsingh99]

Sudo access is disabled on the containers

I'm sorry I didn't quite understand this. You have full access inside a container (without using sudo)

Relevant discussion.

We will have to install sudo to the docker images in order to get root access.

[shrijitsingh99]

Implemented the automatic docker pushing over here. (CI Results)
Maybe we can create a separate issue for this, don't want to clutter this discussion.

Currently looking into a method to only running the docker stage when there is an update to the Dockerfile. Shouldn't be that complicated using git diff.

[kunaltyagi]

Maybe we can create a separate issue for this

Yes please. Let's carry on the discussion there.