Speed up CI cycle

[kirillmakhonin]

The time of CI cycle has to be decreased. Currently it takes about 2 hours to fully check PR readiness for merge.

Ideas:

• Decouple infrastructure automation code from Legion core services code - (will be done as separate task.)
• Deploy Legion and run tests on already created clusters for Legion services code changes
• Create/terminate cluster and run infrastructure tests for PRs with infrastructure automation code changes
• Parallel tests execution wherever this is possible
• Parallel execution of CI pipeline stages/steps wherever this is possible
• Full integration test that includes cluster creation and execution of all test suites execute only for nightly build

Next Actions:

1. Keep current CI approach only for nightly build with one modification - turn off docker layer caching for all containers that are built during CI pipeline - DONE.
2. Create Jenkins job that will create/terminate CI k8s cluster by schedule to keep CI cluster running only during working hours. - DONE.
3. Create new CI pipeline that will - DONE.
   • build PR with code changes only in Legion services.
   • deploy/redeploy Legion services on preconfigured CI k8s cluster.
   • execute all test suites
4. Create new CI pipeline that will - DONE.
   • build PR with code changes only in Legion infrastructure scripts
   • create Legion k8s cluster
   • deploy last stable version of Legion services
   • execute limited test suite that is related to Legion infrastructure components
   • terminate cluster

[alexander-semenets]

more ideas to the list:

• parallel deploy tests during build stage
• install dependencies for legion at edge before the legion package
• parallel robot tests