split CI tests

[miraclx]

Current CI runs use upwards of an hour to complete. — https://github.com/near/cargo-near/actions/runs/2872416612

This is an attempt at cutting that time by using multiple nodes to run the tests in parallel.

[austinabell]

neat trick to split tests. Unfortunate how many distinct tests there are with the 4 combinations of 5 tests, though

[austinabell]

note to self or for someone else to remember, if we remove the SDK dependency in #23, we can remove this ignore

[miraclx]

Turns out, GitHub has a hard limit on how many jobs run in parallel at any point in time.

It's 20 in total and a maximum of 5 of them for macOS.

https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits

And from the tests, it seems to prioritize running Ubuntu over macOS. Perhaps I should switch the matrix order.

[austinabell]

Turns out, GitHub has a hard limit on how many jobs run in parallel at any point in time.

It's 20 in total and a maximum of 5 of them for macOS.

docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits

And from the tests, it seems to prioritize running Ubuntu over macOS. Perhaps I should switch the matrix order.

Or maybe just split into fewer groups? There are 6 non-batch options, so if it was split into 4 groups, it would be one over that 20 total? no opinions here

[miraclx]

It's 20 in total and a maximum of 5 of them for macOS.

Looking at https://github.com/orgs/community/discussions/26343#discussioncomment-3251562, turns out that limit is organization-wide.

So all repositories under the near org have only 5 macOS runners to share.

This affects us quite a bit, as the current PR state requires 6 of those runners.

And looking at the last CI run – https://github.com/near/cargo-near/actions/runs/2877970694/usage

The macOS jobs seem to have waited on each other. Increasing our CI runtime even more than originally.

For this reason, I'm just going to close this PR.

I wonder if it's worth it to explore using the same node to test with the stable and 1.56.0 toolchains.

Quick question though? why are we testing 1.56.0? is that our MSRV?

[itegulov]

Quick question though? why are we testing 1.56.0? is that our MSRV?

That is indeed MSRV, but I wonder if MSRV even makes sense for a binary...

Another thing to consider: maybe we should only run integration tests as a cron job (e.g. once per day) and ping us all if it fails?

[miraclx]

That is indeed MSRV, but I wonder if MSRV even makes sense for a binary...

Not really. But even if it did, we wouldn't need to run tests for MSRV. A simple cargo check should suffice.

maybe we should only run integration tests as a cron job (e.g. once per day) and ping us all if it fails?

Yeah, we can do that. Although, would there ever be a case where the tests would yield different results between runs? Since all the variables that go into the tests are constant. I guess the only case would be dependency version changes.

[itegulov]

Not really. But even if it did, we wouldn't need to run tests for MSRV. A simple cargo check should suffice.

Ok, let's just get rid of 1.56.0 then. Probably not worth even doing cargo check.

Yeah, we can do that. Although, would there ever be a case where the tests would yield different results between runs? Since all the variables that go into the tests are constant. I guess the only case would be dependency version changes.

I guess this would only be useful if this repo had >1 PRs merged per day, so yeah probably does not make sense for now. Only running integration tests on master (i.e. after you merge a PR) would be a good middle ground though I think.

[austinabell]

That is indeed MSRV, but I wonder if MSRV even makes sense for a binary...

It only matters such that if you run cargo install, you need to have that toolchain installed.

[miraclx]

It only matters such that if you run cargo install, you need to have that toolchain installed.

For that, we can use the rust-version field in Cargo.toml.

[austinabell]

For that, we can use the rust-version field in Cargo.toml.

Well, that indicates which version is required. The reason I'm saying MSRV is useful in CI is that you don't introduce changes that unnecessarily increase MSRV, requiring devs to upgrade/install newer toolchain to install. Nbd though

[miraclx]

You're right. We can reintroduce it, but leave it at cargo check instead of running the full test suite. - #30