x/build/builders: ARM builders

[bradfitz]

This bug is about creating ARM qemu full-system-emulation environments for running Go
builds. This is part of http://golang.org/s/builderplan

For ARM5,
The versatilepb machine (ARM5?) is quite slow can can run "make.bash" is about
25 minutes:
https://people.debian.org/~aurel32/qemu/armel/

We could cross-compile make.bash from x86 and just shard (a subset of?) run.bash on
qemu-system-arm -M versatilepb.

For ARM7+, it seems qemu in recent versions supports newer CPUs and supports multiple
cores as well? That might help.

In any case, this will be slow, and will depend on being able to shard tests out over
perhaps hundreds of CPUs (each running a full Linux kernel + tiny userspace which we
control, doing nothing but running our builders).

It'd be nice if the tiny userspace running inside the qemu-system-arm ran Docker, so
then we'd have a snapshotting filesystem which we could send around the network, which
we'll want for sharding.

We can use this bug to discuss and collect notes.

[gopherbot]

Comment 1 by AndrewChamberss:

Saving for reference - https://wiki.ubuntu.com/ARM/RootfsFromScratch/QemuDebootstrap
I'll test this out for building an ARM debian rootfs entirely from within docker.

[gopherbot]

Comment 2 by AndrewChamberss:

https://wiki.ubuntu.com/ARM/RootfsFromScratch
The Debootstrap page actually suggested other methods from this page, rootstock looks
promising.

[bradfitz]

Comment 3:

I'm glad you're working on this. :)

[gopherbot]

Comment 4 by AndrewChamberss:

https://wiki.ubuntu.com/Core
more reference. 
No problem :) I wish gc had mips support too since I actually have the hardware.

[gopherbot]

Comment 5 by AndrewChamberss:

One final reference for buildroot:
http://buildroot.uclibc.org/
http://git.buildroot.net/buildroot/tree/board/qemu/arm-versatile/readme.txt
Buildroot isn't a full distro (it uses busybox etc) and everything is built from source.
If I mirror the tarballs that our config needs, then it is really versatile as you can
easily swap kernel versions and the compiled images are under 20 megs.
What building a filesystem looks like in buildroot:
make goarmbuilder_defconfig
make
I'll hopefully get a prototype Dockerfile running tonight (New Zealand time).

[minux]

Comment 6:

i'm not against this idea, but previous experiences
showed that qemu-system doesn't do well at exciting
concurrent code paths, so we will still need real
hardware to catch, e.g., sync/atomic and runtime
problems.

[gopherbot]

Comment 7 by AndrewChamberss:

Yeah, this is really trying to be a sanity test and a way for people to get a working
arm build to debug things.
more notes:
debootstrap and cdebootstrap are failing on the most recent ubuntu version and docker.
I'll need to investigate those further.
Buildroot worked well until I realized they removed support for installing a gcc
toolchain in the built filesystem. I need to configure my own way to cross compile gcc
to run on arm before I can use it.

[bradfitz]

Comment 8:

Labels changed: added builder.

[gopherbot]

Comment 9 by AndrewChamberss:

My plan is to generate an arm ubuntu or debian base image, install ssh + ssh keys into
the image, run the image with qemu-system-arm in docker, then forward commands and scp
files from docker to it using qemu userspace tcp redirection. I'm still trying to find a
reliable image building system we can use.
My next bet to get a vexpress arm ubuntu image is following this guide:
https://wiki.ubuntu.com/Kernel/Dev/QemuARMVexpress.
The problem with this is that docker build can't mount loop devices (only run
--privileged) and I haven't yet tested qemu user mode emulation inside docker which does
something to install itself into the system elf execution.
Maybe building the arm image base to the point it can run ssh may need to be a different
script (possibly with vagrant vm if docker can't do it well enough.).
The output of this script is a kernel + initrd + ext filesystem, which can be saved and
the docker file consumes it.

[gopherbot]

Comment 10 by AndrewChamberss:

Can someone try this docker file out:
https://github.com/andrewchambers/golang-armbuilder-tests
It runs a single core arm7 processor with debian using the aurel32 armhf images. I don't
really know how to setup and test the builder, but this docker image at least lets you
ssh in so it can be used as a starting point. 
I worry that the debian apt-get install gcc might stop working if it gets out of date
and the aurel32 person hasn't updated the images in a while, it might be worth hosting
our own image with everything needed preinstalled.
I also created a vagrant file and some scripts following
https://wiki.ubuntu.com/Kernel/Dev/QemuARMVexpress which can generate a dual core ubuntu
arm image:
https://github.com/andrewchambers/arm-vexpress-dualcore-image
Unfortunately it seems really unstable, and randomly fails to boot/hangs with kernel
errors.

[bradfitz]

I'm making this bug a bit more general about updating all our ARM builders in the new http://golang.org/s/builderplan world, be it emulated, real hardware, or an ARM hosting service like http://labs.online.net/ ... I imagine we'll do all three.

For real hardware I want to use Docker, which means we'll need an ARM distro that does Docker. I hear Arch does it.

/cc @minux