Convert CI to multi-stage pipelines

[kunaltyagi]

Your Environment

• Operating System and version: All CI

Context

Currently, each CI is single stage, completely independent of each other.

Expected Behavior

Splitting CI into stages will allow:

• Better diagnostics: pipeline A failed at stage: build or test
• More free time:
  • sometimes jobs fail formatting, but continue consuming the CI time for build+test. This will be done again after formatting is fixed.
  • tutorials are built in parallel, leading to double compilation of base library
• More flexibility: addition of API/ABI breakage will be easier

[taketwo]

I was interested in making "Tutorials" a follow-up stage for the library build, so I've looked into this some time ago. I do not recall what was the culprit, but I could not achieve this either at all, or in reasonable time. So not sure if this qualifies as "good first issue". Things might have changed since then though.

[shrijitsingh99]

We can have a pipeline with stages like this?
Formatting -> Build -> Tutorials -> Documentation
We can make Tutorials & Documentation reuse already built modules using caching.
Why do we have separate jobs for tutorial and documentation, shouldn't they be combined?

[kunaltyagi]

Slight difference in build flags (PCL_NO_PRECOMPILE ON/OFF). But formatting should be a precursor to all other jobs.

But yes, Format -> Build matrix -> Test -> Tutorial is a nice sequence. I don't know where documentation would fit best

[taketwo]

Why do we have separate jobs for tutorial and documentation, shouldn't they be combined?

'Tutorials' job is building the tutorials code, not the tutorials HTML pages.

[shrijitsingh99]

Slight difference in build flags (PCL_NO_PRECOMPILE ON/OFF). But formatting should be a precursor to all other jobs.

But yes, Format -> Build matrix -> Test -> Tutorial is a nice sequence. I don't know where documentation would fit best

We can have a parallel job for Documentation.

[kunaltyagi]

Dividing this issue request in 2 parts:

• Break OS CI into build + test ([CI] Separate Build and Test as different stages for CI #3947)
• Break Tutorial CI into (build install) + tutorials

[stale]

Marking this as stale due to 30 days of inactivity. It will be closed in 7 days if no further activity occurs.