chore: Parallelize CircleCI lint build

[vitsalis]

The sequential execution of all of the CircleCI steps takes too long. In this PR we parallelize them to improve CI times.

[vitsalis]

Didn't see much of an improvement on CI times when I tried parallelizing the test and build jobs individually. Probably due to the resources being stretched across the jobs. Further, for some reason the integration tests failed (might be worth checking out the log @KonradStaniec). For that, I made the PR to only parallelize the lint job.

[SebastianElvis]

Didn't see much of an improvement on CI times when I tried parallelizing the test and build jobs individually. Probably due to the resources being stretched across the jobs.

Another possible reason might be that the build-test step (which builds Babylon and executes all tests sequentially) dominates the execution time. From my experience lint takes negligible time (<1min) within a CI execution.

[SebastianElvis]

Lgtm!

[SebastianElvis]

Given that lint takes little time and its result does not depend on build-test, maybe putting lint before build-test is better?

[vitsalis]

Those are running in parallel so it doesn't matter which one we put first.

[SebastianElvis]

Ah ok you are right. All good then

[KonradStaniec]

Further, for some reason the integration tests failed (might be worth checking out the log @KonradStaniec). For that, I made the PR to only parallelize the lint job.

Yea some of integration tests needs improvements in terms of being deterministc. As we are after whole update cycle in terms of cosmos libraries this probably good moment to do that.

In general it maybe worth doing e2e tests and integration tests in parallel after building docker image ?

One thing that I am also considering is to to remove integration tests and leaving only e2e tests, as this e2e tests are much more powerfull in terms of configuration. After adding few more scenarios, they could test the same things as integration tests. This would definitly speed up CI.

[KonradStaniec]

maybe it makes sense to use smaller image just for linting ? (like medium)

[vitsalis]

In general it maybe worth doing e2e tests and integration tests in parallel after building docker image ?

Not entirely sure how we can accomplish that as iirc parallel jobs work on their own environment. Eitherway, the e2e tests do not need a Docker image right? I tried parallelizing the jobs before but I suffer the determinism issues outlined previously

One thing that I am also considering is to to remove integration tests and leaving only e2e tests, as this e2e tests are much more powerfull in terms of configuration. After adding few more scenarios, they could test the same things as integration tests. This would definitly speed up CI.

This sounds like a good idea. Currently, the integration tests are very slow given that the localnet needs to come up before they are executed.

[KonradStaniec]

Eitherway, the e2e tests do not need a Docker image right? I tried parallelizing the jobs before but I suffer the determinism issues outlined previously

They actually do, but they are running after integration tests and integration tests build a babylon docker image, and as both integration tests and e2e tests run in the same enviroments the image is available for integration tests.

[vitsalis]

Eitherway, the e2e tests do not need a Docker image right? I tried parallelizing the jobs before but I suffer the determinism issues outlined previously

They actually do, but they are running after integration tests and integration tests build a babylon docker image, and as both integration tests and e2e tests run in the same enviroments the image is available for integration tests.

I see. Let's find a way to speed integration + e2e as it is very difficult to test right now using them, especially if a build fails. Otherwise, let's try to make the integration tests more deterministic as most of the failures happen due to non-determinism.