Skip to content

x/build/env/windows/arm64: windows/arm64 builders are down #57408

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
qmuntal opened this issue Dec 20, 2022 · 18 comments
Closed

x/build/env/windows/arm64: windows/arm64 builders are down #57408

qmuntal opened this issue Dec 20, 2022 · 18 comments
Labels
arch-arm64 Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Milestone
Unreleased

Comments

@qmuntal
Copy link
Member

qmuntal commented Dec 20, 2022

windows\arm64 builders have been unresponsive since December 12. First failing CL was https://go-review.googlesource.com/c/go/+/456835, which doesn't seem related.

Is this a planed-long downtime? Is there an ETA for having them up an running again? I have a couple of CLs stuck due to this issue.

@cagedmantis @heschi @dmitshur
@qmuntal qmuntal added OS-Windows Builders x/build issues (builders, bots, dashboards) arch-arm64 labels Dec 20, 2022
@gopherbot gopherbot added this to the Unreleased milestone Dec 20, 2022
@heschi
Copy link
Contributor

heschi commented Dec 20, 2022

Yeah, unfortunately we've lost our builders and not yet replaced them. @thanm may know more about progress here.

@heschi heschi moved this to In Progress in Go Release Dec 20, 2022
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed NeedsFix The path to resolution is known, but the work has not been done. labels Dec 28, 2022
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/460296 mentions this issue: internal/releasetargets: remove windows-arm64 for now

gopherbot pushed a commit to golang/build that referenced this issue Jan 3, 2023
@heschi @gopherbot
internal/releasetargets: remove windows-arm64 for now
5936c8f 
We haven't gotten the new builder up yet.

Updates golang/go#57408, golang/go#53541

Change-Id: I4f8b1d96d3f7f7a3080532f5e69b2fa0cf6992e0
Reviewed-on: https://go-review.googlesource.com/c/build/+/460296
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@jaredpar
Copy link

jaredpar commented Jan 4, 2023

Is the expectation that these will be brought back when the builders are available?

@thanm
Copy link
Contributor

thanm commented Jan 4, 2023

@jaredpar yes

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/451275 mentions this issue: env/windows-arm64: initial Azure VM support

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/460837 mentions this issue: dashboard: add windows-arm64 back in using azure vms

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/460836 mentions this issue: cmd/buildlet/stage0: add windows/arm64 azure support

gopherbot pushed a commit to golang/build that referenced this issue Jan 5, 2023
@thanm
cmd/buildlet/stage0: add windows/arm64 azure support
7848fa5 
Add support for windows/arm64 azure-based (reverse) builders
using "host-windows11-arm64-azure" host type.

Updates golang/go#57408.

Change-Id: Ia61f711d9456a7fe91995b5157092c01aee0cf48
Reviewed-on: https://go-review.googlesource.com/c/build/+/460836
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Jan 6, 2023
@thanm
env/windows-arm64: initial Azure VM support
029f06c 
Add README / notes and startup powershell scripts for creating and
configuring Windows Arm64 VMs hosted on Azure.

There are two recipes included here, one for creating deployment
reverse builder VMs (with no public IP), and a second recipe for
creating a test/debugging VM (with public IP, ssh enabled, etc) that
can be used for manual testing (the expectation is that debug/test VMs
will be needed only occasionally, and on a short-term temporary basis).

NB: more work/investigation needed on configuring the VMs to reduce
anti-virus overhead; at the moment the all.bat runtime is not as fast
as it could be, due to interference from "Antimalware service
executable" according to task manager.

Updates golang/go#57408.

Change-Id: Ic8f56c42ca882d86cefdda5dd27be846237778c3
Reviewed-on: https://go-review.googlesource.com/c/build/+/451275
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Than McIntosh <thanm@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Jan 6, 2023
@thanm
dashboard: add windows-arm64 back in using azure vms
faea4fa 
Add back windows-arm64 support, using reverse builders based on Azure
VMs. At the moment only windows 11 is available.

Updates golang/go#57408.

Change-Id: Id93adf920226e13f74ec921e9c70765837a9cca8
Reviewed-on: https://go-review.googlesource.com/c/build/+/460837
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/461097 mentions this issue: env/windows-arm64/azure: add stage zero loop script

gopherbot pushed a commit to golang/build that referenced this issue Jan 9, 2023
@thanm
env/windows-arm64/azure: add stage zero loop script
a5c3a09 
Add a wrapper script that loops forever executing the boostrap stage0
buildlet executable (needed for proper reverse builder functionality)
Update startup script to download script during setup and run at login.

Updates golang/go#57408.

Change-Id: Ie77c37462a32ae259e443f2dc9996ab11a631082
Reviewed-on: https://go-review.googlesource.com/c/build/+/461097
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/461255 mentions this issue: internal/releasetargts: add windows-arm64 again

gopherbot pushed a commit to golang/build that referenced this issue Jan 9, 2023
@cagedmantis
internal/releasetargets: add windows-arm64-11 as a release target
180ddaa 
This adds the windows-arm64-11 builder as a release target which was
removed in CL 460296.

Updates golang/go#57408
Updates golang/go#53541

Change-Id: I449859081350a36b05b85a7b87b412d43b19cb27
Reviewed-on: https://go-review.googlesource.com/c/build/+/461255
Run-TryBot: Carlos Amedee <carlos@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@thanm
Copy link
Contributor

thanm commented Jan 10, 2023

Builders are back up. I'm going to keep the issue open for a bit longer in case other problems arise, but for now it looks like things are working.

@qmuntal
Copy link
Member Author

qmuntal commented Jan 10, 2023

Builders are back up. I'm going to keep the issue open for a bit longer in case other problems arise, but for now it looks like things are working.

Good news!

@thanm I'm still not able to trigger the windows-arm64 builders from a CL (by posting TRY=windows-arm64), see for example https://go-review.googlesource.com/c/go/+/454675. Is there something missing?

@thanm
Copy link
Contributor

thanm commented Jan 10, 2023

@qmuntal as far as I know there aren't any remaining issues/problems. We had to do a test fix backport for the release branches, but other than that I think things are working. I'll go ahead and close out the bug, we can reopen later if need be. Thanks.

@thanm thanm closed this as completed Jan 10, 2023
@github-project-automation github-project-automation bot moved this from In Progress to Done in Go Release Jan 10, 2023
@heschi
Copy link
Contributor

heschi commented Jan 10, 2023

@qmuntal I don't see the TRY= comment on that CL? The alias does exist: https://cs.opensource.google/go/x/build/+/master:dashboard/builders.go;l=103;drc=faea4fa6b0c3ed3cd8232c6eb6742d3bcb3f867d.

@qmuntal
Copy link
Member Author

qmuntal commented Jan 10, 2023

@qmuntal I don't see the TRY= comment on that CL? The alias does exist: https://cs.opensource.google/go/x/build/+/master:dashboard/builders.go;l=103;drc=faea4fa6b0c3ed3cd8232c6eb6742d3bcb3f867d.

Here it is: https://go-review.googlesource.com/c/go/+/452995/comments/ba8287b7_ae1d5bac

@heschi
Copy link
Contributor

heschi commented Jan 10, 2023

Oh, it was too old to see. I believe you need to repeat it every time you trigger trybots. https://github.com/golang/go/wiki/SlowBots#pitfalls

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/462015 mentions this issue: env/windows-arm64/azure: fill in a few missing README bits

gopherbot pushed a commit to golang/build that referenced this issue Jan 13, 2023
@thanm
env/windows-arm64/azure: fill in a few missing README bits
8fc660d 
Fill in a few more minor steps needed in the recipe for reverse
builder VM creation/setup (builder key, initial login).

Updates golang/go#57408.

Change-Id: Iba36ef50bb2e1bbf734126aabcec0dcf850f695b
Reviewed-on: https://go-review.googlesource.com/c/build/+/462015
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Run-TryBot: Than McIntosh <thanm@google.com>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/462516 mentions this issue: env/windows-arm64/azure: update instructions

gopherbot pushed a commit to golang/build that referenced this issue Jan 17, 2023
@heschi @gopherbot
env/windows-arm64/azure: update instructions
0801fb3 
- Account for the new production environment
- Use smaller, cheaper machines
- Fix the antivirus instructions

For golang/go#57408.

Change-Id: I219677c570787a907d67763e0d88552ebdc142c4
Reviewed-on: https://go-review.googlesource.com/c/build/+/462516
Reviewed-by: Than McIntosh <thanm@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/462655 mentions this issue: env/windows-arm64/aws: delete

gopherbot pushed a commit to golang/build that referenced this issue Jan 18, 2023
@dmitshur @gopherbot
env/windows-arm64/aws: delete
b263a02 
The Windows ARM64 builders on Azure have replaced the Windows ARM64
builders on AWS by now. Remove unused environment configuration; we
can bring it back or look it up in history if needed.

Updates golang/go#53541.
Updates golang/go#42604.
Updates golang/go#57408.

Change-Id: I080e4f5c53712c0470332a7d26d00c939136e57c
Reviewed-on: https://go-review.googlesource.com/c/build/+/462655
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
@golang golang locked and limited conversation to collaborators Jan 18, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-arm64 Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Projects
Go Release
Archived in project
Milestone
Unreleased
Development

No branches or pull requests
6 participants
@jaredpar @dmitshur @qmuntal @gopherbot @thanm @heschi