Investigate how to speedup SDK chain build times

[ValarDragon]

Summary

Currently, doing a clean build of an SDK application with all of its applications takes over 2 minutes in CI. (And over a minute locally) Due to issues with docker building, we can't cache the golang building steps on repeat builds. (Its a pretty annoying issue - I have a set of links here w/ more explanation: osmosis-labs/osmosis#2264 (comment) )

But for now, to my understanding best we can do to speedup docker builds in CI is:

• Cache downloading dependencies (example done in that PR I linked)
• Precompile in docker some dependencies (TBD how to make this work cleanly with go mod updates)
• Speedup compilation time itself

(Would love for someone more knowledgeable on go build process or docker, to help find any other caching / common precomputation tricks we can do to lower CI docker time!)

I consider docker build times an extremely important metric to speedup, as its part of the development hot loop of testing.

Steps to improving

We can see at a high level where time goes in building, by adding a -v flag. Its a bit unclear still with go downloading what may already get built, but heres an example of time when downloads are not pre-done:

• https://pipelines.actions.githubusercontent.com/serviceHosts/b037856a-653a-4571-93fd-062f342bb9d1/_apis/pipelines/1/runs/31351/signedlogcontent/2?urlExpires=2022-08-01T03%3A24%3A26.7722866Z&urlSigningMethod=HMACV1&urlSignature=%2B0O80ULzqU1sY%2BLpzRh22gVbjPKxWj7%2BMbFPDmhHRyA%3D
• CTRL+F to go build -v to see the start of build time

I don't know if thats enough detail yet to start finding overheads / looking for dependencies we can try to take out / build flag out. We I think we need to understand how parallelism in build is done and how to time track w/ that in mind, and how dependencies are selected to get compiled.

Another approach is that we can look into: golang/go#45474 , it may be the case that we can determine a large amount of the main dependencies as things we can precompute go build on, to improve CI time?

Proposal

• Start thinking about how to look into our compilation times
• See and understand if theres things we can do to lower compile time, e.g. build flagging things out, or being smarter with dependency graph / imports / compile directives
• Other ways we can eliminate docker build times

[ValarDragon]

I came across this recently, which could help us understand where build time comes from: https://github.com/unravelin/actiongraph

(I am still running into build times as a testing efficiency blocker)

[tac0turtle]

amazing, rosetta will be removed soon, then after that would be nice to understand the next blockers

[robert-zaremba]

Hub doesn't require rosetta any more?

[tac0turtle]

Removed was the wrong wording. It will be removed from being part of the node and instead be a standalone binary

[ValarDragon]

Would be helpful if we can get more investment in reduced build times!

[tac0turtle]

I dont think this is a high priority right now. With serverv2 it should go down for us, but for users with custom modules its unclear what is affecting their build times