CI: Reduce our monthly minutes usage #1366
Comments
|
For reference, per minute rates for runners are as follows (https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#per-minute-rates):
I believe that if you don't specify a runner, you get the 2 core linux one (I thought until now that the 4 core one was the default). So everything else is a multiple of this cost essentially based on the core count. Addendum: When we use the basic runner, it is included in the free monthly mionutes that we get as an org (which we may exceed anyway many months, but would need to check again). |
|
We could further reduce the CI time by sharing a substrate-node across tests. Although we should consider this at a later time since the other proposed ideas are easier to implement. Currently, each test will spawn a substrate node; and submit a few transactions and rpc calls (some with side-effects others not). Further, some tests will wait for the local substrate node to produce a few blocks. By sharing the substrate node, we eliminate the overhead of spawning the node and waiting for blocks. We could still have valid tests that produce side effects:
To mitigate this we could initialize the substrate node with multiple test accounts; and each test would use a different account provided by our testing backend. We may need to add a custom testing RPC call to populate accounts, since last time I checked the initial accounts are seeded from polkadot-sdk code. Would love to get some feedback on this 🙏 What do you guys think? That said, I believe most of the CI time is related to my PRs investigating the timeout issues (lightclient and unstable backend). And we should indeed expect a normal return of CI consumption once debugging stops. |
|
Just for the complexity and fear of weird interactions between tests, I'd share a substrate-ndoe binary as a bit of a last resort really! I am hopeful that we can get our CI down to a good level with smaller timeouts, fixing the issue where we actually hit them, and running clippy+fmt+whatever first to fail early in the common case :) |
|
I think we've addressed these for now, so I'll close this and we can re-visit when we get feedback on our usage from now. |
We need to reduce our monthly CI usage so that we don't exceed an org imposed (I think) cap on monthly CI minutes. Currently, Subxt is the worst offender.
The 45 minute timeout added to our tests should already help with this a lot (and perhaps we can further reduce this to say 30mint; we'd have to see! We don't want potentially successful jobs timing out).
Further, once we fix the unstable backend test timeouts (@lexnv is working on this; my hope is we can merge a fix in the next couple of days) we should reduce our usage some more.
Next, let's see whether using the faster
ubuntu-8-coreetc runners (which we can assume use a multiple of minutes depending on how many multiples of 4 cores they are, eg 8 core would be 2x minutes) actually are worth it where used, and revert to slower machines if the total time is not decreased significantly. We need to find a good balance here between test speed and minute use.Perhaps we could also optimise our CI pipeline so that we run fast checks first (fmt and clippy for instance, but check the times these take to decide) in one block, and then the rest in another block (thanks for the idea @tadeohepperle). This will hopefully catch the quite frequent/basic fmt/clippy fails and prevent excess minutes being used until they are all fixed up, only running the long tests when the basics are all good.
Related issues:
The text was updated successfully, but these errors were encountered: