Speedup your CI

There is an article in Russian about speeding up building and testing in CI. This page contains a list of tools that helps with speedup.

CI/CD Pipelines are the fundamental building blocks for CI/CD. Making pipelines more efficient helps you save developer time, which:

• Speeds up your DevOps processes
• Reduces costs
• Shortens the development feedback loop

It’s common that new teams or projects start with slow and inefficient pipelines, and improve their configuration over time through trial and error. A better process is to use pipeline features that improve efficiency right away, and get a faster software development lifecycle earlier.

Source code

• Code layout Good structure of files in a project helps to make proper dependencies. The rule of thumb is putting tests on the same level with source code that these tests covers. See recommendations for CMake in How to structure your project.
• Replace asynchronous synchronization (sleep()) with synchronous one in tests and production source code.

Fast source code checkout

• Use --depth N in Git.
• Enable parallel mode in Git:
  • git config fetch.parallel 0
  • git config submodule.fetchJobs 0

Hints with CI setup

• Profile your CI to eliminate bottlenecks:
  • github-actions-profile - is a profiler for GitHub Actions workflow.
• Analyze the performance of your pipeline to find ways to improve efficiency.
  • Spendesk/github-actions-exporter - github-actions-exporter for Prometheus.
  • mvisonneau/gitlab-ci-pipelines-exporter - Prometheus / OpenMetrics exporter for GitLab CI pipelines insights.
  • cpanato/github_actions_exporter - Prometheus exporter exposing GitHub Actions metrics.
  • Gitlab CI helps with pipeline efficiency.
• Ignoring paths with non-relevant files
  • Github CI: paths-ignore (Documentation)
  • Gitlab CI: rules:changes (Documentation)
• Use caching dependencies
  • Github Actions (Documentation)

Hints with OS setup

• Disable unnecessary triggers in packages (Processing triggers for man-db (2.9.4-2) ...).
  • Debian - https://wiki.debian.org/DpkgTriggers

Compilation and linking

• Profile build system to eliminate bottlenecks:
  • ninjatracing - converts .ninja_log files to chrome's about:tracing format.
  • buildbloat converts ninja build logs to webtreemap JSON files.
  • Debugging Build Performance
• Use parallel mode.
  • Make - option -j
  • Ninja - option -j
  • Meson - option -j
  • CMake - https://www.kitware.com//cmake-building-with-all-your-cores/
• Use fast build system. In some projects, Ninja could be faster than Make. CMake supports both build systems, so you could compare build time for both and choose the fastest. See build time comparison.
• Replace default linker with Mold. Faster in 17 times in comparison with GNU gold and 3-5 times in comparison with LLVM lld.
• Include what you use - is a tool for use with clang to analyze #includes in C and C++ source files.
• Use cache: -ccache - is a compiler cache. ccache could speed up your build in 30 (!) times. See performance results.
  • sccache - is ccache with cloud storage. Supported C, C++ and Rust.
  • firebuild - is an automatic build accelerator. It works by caching the outputs of executed commands and replaying the results when the same commands are executed with the same parameters within the same environment.
• Use distributed compilation (boost 2-4 times)
  • Goma (C/C++) - is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache. Used by Google.
  • nocc (C/C++) - is distributed C++ compiler. Used by VK.
  • distcc is a fast, free distributed C/C++ compiler. See performance results.
  • icecream - is a distributed compiler with a central scheduler to share build load. Created by SUSE.

Running unit tests

• Use distributed running
  • gg - the Stanford Builder. Uses lambdas for running self-contained binaries, see a paper From Laptop to Lambda: Outsourcing Everyday Jobs to Thousands of Transient Functional Containers.
  • llama - is a tool for running UNIX commands inside of AWS Lambda. Its goal is to make it easy to outsource compute-heavy tasks to Lambda, with its enormous available parallelism, from your shell.

Running e2e tests

• Test frameworks:
  • Speeding up pytest: https://github.com/zupo/awesome-pytest-speedup
  • Javascript: Speeding up Javascript Test Time 1000x
• Profile your tests to eliminate bottlenecks:
  • Python pytest-profiling, read Profiling and improving the runtime of a large pytest test suite.
  • Ruby test-prof
• Fail fast
• Parallel execution
• Test prioritization (TODO)
• Test minimization (TODO)
• Use test selection
  • Ruby: Crystalball
  • Python: https://python-tia.readthedocs.io/en/latest/alternatives.html

Code coverage

• fastcov - a massively parallelized gcov wrapper. Time to process all gcda and parse all gcov: fastcov: ~700ms, lcov: ~90s, gcovr: ~30s.

Setup test environment

• DjangoCon 2021 | Speed up your tests with setUpTestData
• Improving C++ Compilation Times: Tools & Techniques

References

• Get up to speed with partial clone and shallow clone
• https://www.stickyminds.com/article/test-faster-how-we-cut-our-test-cycle-time-half
• 2020-08-25 GitLab CI Monitoring
• Faster Python Tests — Ramp Engineering
• Predicting Test Failures
• Stuck with slow tests? Speed up your feedback loop