x/build: linux-arm trybot/slowbot failure

[cherrymui]

What version of Go are you using (go version)?

tip (d856e05)

What did you do?

Run linux-arm trybot on tip (with a dummy change, CL https://go-review.googlesource.com/c/go/+/207440)
It failed with https://storage.googleapis.com/go-build-log/09ec0769/linux-arm_4add410f.log

I also got a few failures on a different CL (https://go-review.googlesource.com/c/go/+/207350)
https://storage.googleapis.com/go-build-log/3c197532/linux-arm_4a2eca13.log
https://storage.googleapis.com/go-build-log/3c197532/linux-arm_e2868c28.log
https://storage.googleapis.com/go-build-log/3472bd16/linux-arm_0d7459e0.log

I cannot reproduce it with gomote. Also the linux-arm builder is happy.

I think it may be related to that the trybot does a cross-compilation and then ships the binaries to the ARM machine.

cc @bradfitz

[bradfitz]

I suspect this was "broken" by https://go-review.googlesource.com/c/build/+/205603 which upgraded the ARM environment (where tests run) to Buster, but the compilation half on x86 was unchanged.

I say "broken" in quotes because this was never a problem until slowbots which let you do this again.

But it used to work with trybots on by default a few years ago, so it does work if we configure both halves the same.

Will fix.

[cherrymui]

Thanks @bradfitz

[bradfitz]

Hmm, this isn't as obvious as I'd hoped.

Both environments are Debian buster:

Cross-compilation:
https://github.com/golang/build/blob/master/env/crosscompile/linux-armhf-cross/Dockerfile

Linux-arm on Scaleway:
https://github.com/golang/build/blob/master/env/linux-arm/scaleway/Dockerfile

@rsc, you probably remember me debugging similar issues in the past. I'd love some better tooling to help debug these sorts of issues. It doesn't affect many people, though. But if we could do this sort of cross-compilation (make.bash in one place, run tests elsewhere) for more builders (riscv, mips, arm, etc) we could get much better throughput from our slower builders. But I'm always scared of doing it due to issues like this.

[mundaym]

Ping. Still seeing this and it means we can't run the trybots on arm targets:

https://storage.googleapis.com/go-build-log/986a3d00/linux-arm_a2dcef07.log

[mundaym]

Unassigning @bradfitz since I don't think he's working on x/build at all anymore.

[cherrymui]

In the mean time, I have been using gomote for ARM. Using android-arm is another option, which is not the same as linux-arm but also has many similarities.

[cagedmantis]

/cc @toothrot @dmitshur

[josharian]

This cost me a chunk of time this morning. Can we disable linux/arm slowbots until this is fixed? Or have gopherbot emit a warning and reference to this issue when it recognizes linux/arm?

[cherrymui]

If this cannot be fixed soon, can we just disable/reject linux-arm for trybot? And maybe use android-arm for "TRY=arm"?

The failure has confused people several times (e.g. https://go-review.googlesource.com/c/go/+/248684).