Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross-compile ARMv6 & ARMv7 release binaries (?) #1238

Closed
rvagg opened this issue Apr 16, 2018 · 7 comments
Closed

Cross-compile ARMv6 & ARMv7 release binaries (?) #1238

rvagg opened this issue Apr 16, 2018 · 7 comments

Comments

@rvagg
Copy link
Member

rvagg commented Apr 16, 2018

Sadly, I think it might be time to shift to cross-compiling ARMv6 & ARMv7 release binaries. We currently build ARMv6 on Raspberry Pi 1 B+'s and ARMv7 builds on the Marvell Armada 370/XP devices that we get from Scaleway.

The ARMv7 builds haven't been speedy but are usually there on time and with little trouble. The ARMv6 builds on the other hand are a totally different matter. Depending on freshness of cache and whether the release Pi's are alive at the time, we can get delays in the order of days at the worst of times. Usually it's not this bad and we normally get nightlies out just fine every day. It's not been easy to keep the Pi's happy for this with them regularly freezing and needing a hard reboot—although this seems to have been mostly solved with a firmware upgrade and running Debian 9 (Stretch) and compiling inside Docker. Michael's backup Pi also has had mixed success, often being unresponsive to builds even though it reports to being online. I also recently had to turn down xz compression to -6 from -9e to save on memory and time (so much time) on the Raspberry Pis. It doesn't cost very much in .tar.xz size but it's a compromise nonetheless. So ARMv6 binaries are kind of complicated and can get a little crazy sometimes.

Now we have a new dilemma. As per #1229 we are bumping binutils minimum version to >=2.26. Unfortunately, unless we jump straight to Debian 9 (Stretch) as our minimum version we have small problem since gcc 4.9.3 is bundled with 2.25. We may be able to get around this, like in #1235, but it's already meant we have to do some gymnastics with our cross compiler and I now have a custom-built one in play for the test infra (for Raspberry Pi 1,2,3 testing at least, not the plain ARMv7 tests we also run in Scaleway). See #1237.

There's also the fact that we don't even test ARMv6 binaries produced the way that we do them for release! We cross-compile for our tests, but native-compile for releases.

So, I'd like us to discuss the pros and cons of cross-compiling our release binaries. I currently have a new cross compiler set up in ci-release to test nightlies and have taken the debian8 and pi1-docker configurations out of action for Node 10+ for now. This is easily reversable of course but we'd need to solve the binutils problem on both of these builders if we go that route. I'll test over the next couple of days with nightlies and report back anything that might be cause for concern.

@nodejs/build

@rvagg
Copy link
Member Author

rvagg commented Apr 17, 2018

so I've tested the binaries on some different ARM devices and they work as expected:

  • The ARMv6 binaries work on ARMv6 & ARMv7 machines for Jessie+
  • The ARMv7 binaries work on ARMv7 machines for Jessie+

Will test more when the next v10 test build comes out. I'll also confirm that they work on ARM64 and might fish around for more ARM devices to try on. I don't see any reason they'd be too different from the native-compiled binaries but I want to have higher confidence in this process.

@gibfahn
Copy link
Member

gibfahn commented Apr 17, 2018

Cross-compiling sounds good to me, especially if that's how we build in CI.

@refack
Copy link
Contributor

refack commented Apr 17, 2018

+1
P.S. didn't someone mention merging ARM efforts with @chrislea?

@refack
Copy link
Contributor

refack commented Apr 17, 2018

P.P.S. regarding xz is it possible to just tar "natively" then xz on a stronger machine?
P.P.P.S. there's buzz about Jenkins X and containerized CI1,2 (plus Rodd's latest venture). Is it worth considering running the release process in sanitized containers, and eliminating the release cluster?

@gibfahn
Copy link
Member

gibfahn commented Apr 18, 2018

Is it worth considering running the release process in sanitized containers, and eliminating the release cluster?

For the machines we can containerize that sounds great, but I'd assume that some of them would be harder to get set up (I think Jenkins X requires Kubernetes). Also Jenkins X sounds cool, but it seems entirely focused on building docker apps that get deployed onto a Kubernetes cluster, which doesn't really help us.

@joaocgreis
Copy link
Member

I'm +1, I though about cross compiling releases a number of times but was never able to move forward. This should take care of the issues we have with the Pi 1s on the release Jenkins, and actually release a binary that is more like the one we test in CI.

On the down side, we lose the last barrier we have ensuring that node can be compiled natively in a Raspberry Pi. I don't think this is a big issue if we point people towards the new cross compilers (part of #1237) and tell them how to use it. If compiling natively is actually important for anyone, we could add a new subjob to the daily jobs in CI to compile and test (not to node-test-commit because it takes hours).

On a related thought, this is not likely to happen quickly but I think our CI should move away form the current logic of build and test on each platform. Instead, build like a release and test the resulting binaries on the platforms where they apply (similar to using the current cross compiling logic for everything). We could still have the current jobs for platforms that we don't publish binaries for, and for platforms that are too different (like newer versions) than the ones where we compile the releases. The main advantage of this would be, if we found a good way to manage temporaries, to run CI on a release before promoting.

@refack
Copy link
Contributor

refack commented Apr 16, 2019

IIUC this has been done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants