builds: Enable lld, optimize linking more, faster cargo-test build #25116
Conversation
def-
force-pushed
the
pr-ci-cargo-test
branch
15 times, most recently
from
82134dd
to
e3556e5
Compare
def-
changed the title
def-
force-pushed
the
pr-ci-cargo-test
branch
from
e3556e5
to
d344f7c
Compare
| "-C", | ||
| "link-arg=-Wl,--compress-debug-sections=zlib-gabi", | ||
| "-Clink-arg=-Wl,--compress-debug-sections=zlib", | ||
| "-Clink-arg=-Wl,-O2", |
There was a problem hiding this comment.
This is going to apply to local developer builds too, no? I don’t think everyone has easy access to lld.
There was a problem hiding this comment.
Yes, but on Linux only. I think we should set lld everywhere on Linux actually, and devs who are on Linux should be able to easily install it or override RUSTFLAGS. lld is definitely faster than ld, so I don't see a downside.
There was a problem hiding this comment.
I'm somewhat reticent to set any compile flags that will make it so that cargo build doesn't work with a standard Rust installation. Historically the main motivation behind this philosophy has been to encourage external contributions from folks who might not want to customize their toolchains to match Materialize's specifications. That's less of an issue now that we've gone cloud native. But recently @ParkMyCar ran into trouble while trying to get the Fivetran destination deployed because Fivetran's hardware is too old for the target CPU/features we specify in .cargo/config. So I'm increasingly pondering whether we should declare that .cargo/config should be the lowest common denominator, and we'd provide a custom profile like dev-fast for folks who want to opt into faster compiles at the cost of installing extra tools. But that's not viable until custom profiles learn to support configuring rustflags.
Anyway, no qualms with trying this out. We can always find a different course if it proves to be a big problem for external contributors or partners who need to build the repository. Just one ask: can you check with folks in #epd-announce and make sure everyone's up for installing lld? We should also update the developer guide with a note that you need to install lld on Linux (or override rustflags).
There was a problem hiding this comment.
The experience with getting the Fivetran Destination built on their hardware definitely surprised me. Also not opposed to adding these for now, but I agree the best approach would be having separate profiles like dev-fast and prod-like, with the default being the lowest common denominator, possibly even no special flags.
There was a problem hiding this comment.
I asked in #epd-announce but didn't get much feedback either way: https://materializeinc.slack.com/archives/CUXUBJH7B/p1707756913865829
|
Deferring to @benesch here. |
def-
force-pushed
the
pr-ci-cargo-test
branch
from
d344f7c
to
c37a18c
Compare
Seems to be ~10% faster for cargo-test.
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.