env/windows-arm64: add Ubuntu/Windows ARM64 VM configuration

This change introduces a Windows ARM64 builder image for Go, running on
Ubuntu on an a1.metal instance on AWS.

The AMI image successfully boots, and automatically starts a Windows
ARM64 VM, exposing port 443. The buildlet still needs to be updated for
the appropriate mingw path. Additionally, the qemu script does not
yet automatically halt the instance on termination, which makes
debugging easier.

- Ubuntu was chosen over Debian, as the official Debian AMI would not
boot on a1.metal instances.
- a1.metal is the most affordable AWS instance to support KVM on ARM64.
- A prepared qemu image exists in the Go AWS account, under
s3://go-builder-data.
- Only 4 processors are currently supported in order to avoid a
CLOCK_WATCHDOG_TIMEOUT. We're working on investigating how to increase
this.
- The EC2 metadata service is proxied to the guest OS via qemu.

For golang/go#42604

Change-Id: I2a505a8218c2a818f0e52d179c02e6d264d2e422
Reviewed-on: https://go-review.googlesource.com/c/build/+/321959
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>

env/windows-arm64/README.md

@@ -0,0 +1,64 @@
+# Windows buildlet images
+
+Windows images are built by creating and configuring VMs hosted on AWS
+a1.metal instances then saving the image manually.
+
+## Build and test the Windows builder image
+
+- Prepare the linux QEMU host image by following the instructions in
+  `env/windows-arm64/aws`.
+- Create an a1.metal instance (or other instance that supports KVM)
+  in AWS.
+- Download a Windows 10 ARM64 image.
+    - Convert vhdx images to qcow2 via the following command:
+
+      ```shell
+      qemu-image convert -O qcow2 win.vhdx win.qcow2
+      ```
+
+- SSH to your instance tunneling port 5901, and run `win10-arm64.sh`
+  script to boot the Windows VM.
+    - You may need to stop the current VM: `sudo systemctl stop qemu`
+- VNC to the tunneled port 5901.
+- Open the device control panel, and use the "Search for Drivers"
+  button to search the virtio drive `D:` for drivers.
+    - Matching drivers will be automatically installed.
+    - This is necessary for networking to work on Windows in qemu.
+- Download the `startup.ps1` script to the Windows instance, and run
+  in PowerShell. Check thoroughly for errors.
+    - Alternatively, you can modify `win10-arm64.sh` to forward ssh
+      access to the VM, and run PowerShell in the CLI, which is a bit
+      easier than through VNC.
+- Once the image is complete, download the image to your workstation
+  and upload to `s3://go-builder-data`.
+    - You can find the appropriate the S3 path referenced in
+      `env/windows-arm64/aws/prepare_image.sh`.
+- Re-run packer to build an AMI with your updated Windows image.
+
+### Notes
+
+- `QEMU_EFI.fd` is from the `qemu-efi-aarch64` Debian package, found
+  at `/usr/share/qemu-efi-aarch64/QEMU_EFI.fd`. It can be regenerated
+  with the following command:
+
+  ```shell
+  dd if=/dev/zero of=QEMU_EFI.fd bs=1M count=64
+  dd if=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd of=QEMU_EFI.fd bs=1M count=64 conv=notrunc
+  ```
+
+- `QEMU_VARS.fd` stores saved EFI state when booting a VM. It's
+  generated via the following command:
+
+  ```shell
+  dd if=/dev/zero of=QEMU_VARS.fd bs=1M count=64
+  ```
+
+- The latest virtio driver image can be fetched from:
+  https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/virtio-win.iso
+
+- `win10-arm64.sh` is hard-coded to run with 4 processors instead of
+  the 16 available on an a1.metal instance. Higher numbers of
+  processors are causing a fatal CLOCK_WATCHDOG_TIMEOUT error from
+  interrupt requests not arriving in time. qemu-system-x86_64 has a
+  workaround for this. We're still investigating how to increase this
+  on aarch64.

env/windows-arm64/aws/Makefile

@@ -0,0 +1,18 @@
+# Copyright 2021 The Go Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style
+# license that can be found in the LICENSE file.
+
+env-var-check:
+ifndef AWS_ACCESS_KEY_ID
+	$(error AWS_ACCESS_KEY_ID env var is not set)
+endif
+
+ifndef AWS_SECRET_ACCESS_KEY
+	$(error AWS_SECRET_ACCESS_KEY env var is not set)
+endif
+
+create-aws-image: env-var-check
+	export AWS_MAX_ATTEMPTS=600
+	export AWS_POLL_DELAY_SECONDS=10
+	export PACKER_LOG=1
+	packer build -timestamp-ui packer_image_aws_arm64.json

env/windows-arm64/aws/README.md

@@ -0,0 +1,31 @@
+# AWS Windows ARM64 Builders
+
+## Machines
+
+The AWS builders use the a1 instance types which are arm64 machines.
+The base type used will be a1.metal, which are the cheapest which
+expose KVM support.
+
+## Machine image
+
+Machine images are stored on AWS EBS service as a snapshot. New VMs
+can use the snapshot as an image by providing the AMI ID as the base
+image when a new VM is created.
+
+### Creating a new Ubuntu host image
+
+Requirements:
+
+Two environment variables are required to be set before initiating
+the command: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` should be
+set with the appropriate values.
+
+The [packer](https://www.packer.io) binary should be in `PATH`.
+
+Command:
+
+`make create-aws-image`
+
+or
+
+`AWS_ACCESS_KEY_ID=<id> AWS_SECRET_ACCESS_KEY=<secret> make create-aws-image`

env/windows-arm64/aws/packer_image_aws_arm64.json

@@ -0,0 +1,55 @@
+{
+  "variables": {
+    "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
+    "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
+    "region": "us-east-2"
+  },
+  "builders": [
+    {
+      "type": "amazon-ebs",
+      "iam_instance_profile": "buildetS3ReadOnly",
+      "access_key": "{{user `aws_access_key`}}",
+      "ami_name": "go-linux-arm64-host-{{timestamp}}",
+      "ami_description": "Image for linux-arm64 Go builder hosting windows-arm64",
+      "instance_type": "a1.metal",
+      "region": "{{user `region`}}",
+      "secret_key": "{{user `aws_secret_key`}}",
+      "source_ami": "ami-0b0c8ae527978b689",
+      "decode_authorization_messages": true,
+      "ssh_username": "ubuntu",
+      "tags": {
+        "Name": "Ubuntu",
+        "Created": "{{isotime \"2006-01-02\"}}",
+        "OS": "Ubuntu 20.04 Focal (ARM64)",
+        "Release": "Latest",
+        "Base_AMI_Name": "{{ .SourceAMIName }}",
+        "Extra": "{{ .SourceAMITags.TagName }}",
+        "Description": "{{user `description`}}"
+      },
+      "launch_block_device_mappings": [
+        {
+          "device_name": "/dev/sda1",
+          "volume_size": 50,
+          "volume_type": "gp2",
+          "delete_on_termination": true
+        }
+      ]
+    }
+  ],
+  "provisioners": [
+    {
+      "type": "file",
+      "source": "./win10-arm64.sh",
+      "destination": "/home/ubuntu/win10-arm64.sh"
+    },
+    {
+      "type": "file",
+      "source": "./qemu.service",
+      "destination": "/tmp/qemu.service"
+    },
+    {
+      "type": "shell",
+      "script": "./prepare_image.sh"
+    }
+  ]
+}

env/windows-arm64/aws/prepare_image.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+
+# Copyright 2021 The Go Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style
+# license that can be found in the LICENSE file.
+
+#
+# Installs all dependencies for an Ubuntu Linux ARM64 qemu host.
+#
+
+set -euxo pipefail
+
+TMP_DIR="$(mktemp -d)"
+
+# Retry apt commands until we succeed.
+#
+# Metal instances are still being provisioned when we are first able
+# to connect over SSH. Some parts of apt are locked or not yet
+# present. Retrying a few times seems to do the trick.
+for i in $(seq 1 10); do
+  # Give it a chance to finish before our first try,
+  # and take a break between loops.
+  sleep 1
+  sudo apt-add-repository universe || continue
+
+  sudo apt-get update || continue
+  sudo apt-get upgrade -y || continue
+
+  sudo apt-get install -y apt-transport-https ca-certificates curl gnupg-agent gnupg jq software-properties-common \
+    build-essential ninja-build && break
+done
+
+# QEMU Dependencies
+sudo apt-get install -y git libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev
+
+# QEMU Extras
+sudo apt-get install -y git-email libaio-dev libbluetooth-dev libbrlapi-dev libbz2-dev libcap-dev libcap-ng-dev \
+  libcurl4-gnutls-dev libgtk-3-dev libibverbs-dev libjpeg8-dev libncurses5-dev libnuma-dev librbd-dev librdmacm-dev \
+  libsasl2-dev libsdl1.2-dev libseccomp-dev libsnappy-dev libssh2-1-dev libvde-dev libvdeplug-dev libvte-2.91-dev \
+  libxen-dev liblzo2-dev valgrind xfslibs-dev libnfs-dev libiscsi-dev
+
+# QEMU download & build
+wget https://download.qemu.org/qemu-6.0.0.tar.xz
+tar xJf qemu-6.0.0.tar.xz
+cd qemu-6.0.0
+./configure --target-list=arm-softmmu,aarch64-softmmu
+make -j16
+cd "$HOME"
+
+# S3 CLI
+sudo apt-get install -y aws-shell
+
+# Copy pre-prepared Windows 10 image
+aws s3 sync s3://go-builder-data/win10 "$HOME"/win10
+
+chmod u+x "$HOME/win10-arm64.sh"
+sudo cp /tmp/qemu.service /etc/systemd/user/qemu.service
+sudo systemctl enable /etc/systemd/user/qemu.service
+sudo systemctl start qemu

env/windows-arm64/aws/qemu.service

@@ -0,0 +1,8 @@
+[Unit]
+Description=QEMU
+
+[Service]
+ExecStart=/home/ubuntu/win10-arm64.sh
+
+[Install]
+WantedBy=multi-user.target

env/windows-arm64/aws/win10-arm64.sh

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+# Copyright 2021 The Go Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style
+# license that can be found in the LICENSE file.
+
+/home/ubuntu/qemu-6.0.0/build/aarch64-softmmu/qemu-system-aarch64 \
+  -name "Windows 10 ARM64" \
+  -machine virt \
+  -cpu host \
+  --accel kvm \
+  -smp 4 \
+  -m 8G \
+  -drive file=/home/ubuntu/win10/QEMU_EFI.fd,format=raw,if=pflash,readonly=on \
+  -drive file=/home/ubuntu/win10/QEMU_VARS.fd,format=raw,if=pflash \
+  -device nec-usb-xhci \
+  -device usb-kbd,id=kbd0 \
+  -device usb-mouse,id=tab0 \
+  -device virtio-net,disable-legacy=on,netdev=net0,mac=54:91:05:C5:73:29,addr=08 \
+  -netdev 'user,id=net0,hostfwd=tcp::443-:443,guestfwd=tcp:10.0.2.100:8173-cmd:netcat 169.254.169.254 80' \
+  -device nvme,drive=hdd0,serial=hdd0 \
+  -vnc :3 \
+  -drive file=/home/ubuntu/win10/win10.qcow2,if=none,id=hdd0,cache=writethrough \
+  -drive file=/home/ubuntu/win10/virtio.iso,media=cdrom,if=none,id=drivers,readonly=on \
+  -device usb-storage,drive=drivers \
+  -chardev file,path=/var/log/qemu-serial.log,id=char0 \
+  -serial chardev:char0 \
+  -device ramfb