macOS 12 images produce VirtualBox machines in gurumeditation state

[saschagrunert]

Description

If we run vagrant up in the Ci, then it fails with:

==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'generic/ubuntu2204' version '4.3.4' is up to date...
==> default: Setting the name of the VM: security-profiles-operator_default_1698229641020_98957
Vagrant is currently configured to create VirtualBox synced folders with
the `SharedFoldersEnableSymlinksCreate` option enabled. If the Vagrant
guest is not trusted, you may want to disable this option. For more
information on this option, please refer to the VirtualBox manual:

  https://www.virtualbox.org/manual/ch04.html#sharedfolders

This option can be disabled globally with an environment variable:

  VAGRANT_DISABLE_VBOXSYMLINKCREATE=1

or on a per folder basis within the Vagrantfile:

  config.vm.synced_folder '/host/path', '/guest/path', SharedFoldersEnableSymlinksCreate: false
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
The guest machine entered an invalid state while waiting for it
to boot. Valid states are 'starting, running'. The machine is in the
'gurumeditation' state. Please verify everything is configured
properly and try again.

If the provider you're using has a GUI that comes with it,
it is often helpful to open that and watch the machine, since the
GUI often has more helpful error messages than Vagrant can retrieve.
For example, if you're using VirtualBox, run `vagrant up` while the
VirtualBox GUI is open.

The primary issue for this error is that the provider you're using
is not properly configured. This is very rarely a Vagrant issue.
make: *** [vagrant-up-ubuntu] Error 1

Used Vagrantfile's:

• https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/hack/ci/Vagrantfile-fedora
• https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/hack/ci/Vagrantfile-ubuntu

Ref: kubernetes-sigs/security-profiles-operator#1936

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

• since runner image version 20230921.1
• version 20230921.4 seemed to work, but 20231017.6 is failing again

Is it regression?

Yes, it worked seamless before

Expected behavior

That the machines come-up successfully.

Actual behavior

The machines stuck in gurumeditation state.

Repro steps

I have no way to reproduce it because I have no access to a macOS 12 machine. It's reproducible in GitHub actions, though.

[erik-bershel]

Hello @saschagrunert!
It seems like we meet here the old bug hashicorp/vagrant#2157
We'll revert VBox update. Will be resolved with the next release.

[saschagrunert]

Hello @saschagrunert! It seems like we meet here the old bug hashicorp/vagrant#2157 We'll revert VBox update. Will be resolved with the next release.

Thank you! I also tried to upgrade VirtualBox to v7 (via brew) but without any luck.

[erik-bershel]

@saschagrunert we didn't found workaround too, which is sad, yeah.

[dereknola]

Given that K3s havs been experience the gurumediation issue for several months, I'm not sure if just the Virtualbox image is the only problem https://github.com/k3s-io/k3s/actions/workflows/install.yaml

[dereknola]

Even with #8731 now committed and Virtualbox back on the old versions, I am still seeing gurumediation errors.

Mac-1699297609190:centos-7 runner$ vagrant up --no-provision
Bringing machine 'install-centos-7' up with 'virtualbox' provider...
==> install-centos-7: Importing base box 'generic/centos7'...
==> install-centos-7: Matching MAC address for NAT networking...
==> install-centos-7: Checking if box 'generic/centos7' version '4.3.6' is up to date...
==> install-centos-7: Setting the name of the VM: centos-7_install-centos-7_1699299721964_58159
==> install-centos-7: Clearing any previously set network interfaces...
==> install-centos-7: Preparing network interfaces based on configuration...
    install-centos-7: Adapter 1: nat
==> install-centos-7: Forwarding ports...
    install-centos-7: 22 (guest) => 2222 (host) (adapter 1)
==> install-centos-7: Running 'pre-boot' VM customizations...
==> install-centos-7: Booting VM...
==> install-centos-7: Waiting for machine to boot. This may take a few minutes...
    install-centos-7: SSH address: 127.0.0.1:2222
    install-centos-7: SSH username: vagrant
    install-centos-7: SSH auth method: private key
The guest machine entered an invalid state while waiting for it
to boot. Valid states are 'starting, running'. The machine is in the
'gurumeditation' state. Please verify everything is configured
properly and try again.

If the provider you're using has a GUI that comes with it,
it is often helpful to open that and watch the machine, since the
GUI often has more helpful error messages than Vagrant can retrieve.
For example, if you're using VirtualBox, run `vagrant up` while the
VirtualBox GUI is open.

The primary issue for this error is that the provider you're using
is not properly configured. This is very rarely a Vagrant issue.

Mac-1699297609190:centos-7 runner$ Vboxmanage --version
6.1.38r153438

Attached is my very simple vagrant file (renamed to .txt because regular Vagrantfiles aren't supported for attachement)
Vagrantfile.txt

[tapasmishra]

We are also experiencing the same issue: https://github.com/magma/magma/actions/runs/6769818735/job/18410707056#step:8:86

[erik-bershel]

Hi @dereknola and @tapasmishra! Fix will be available only after next macOS 12 image rollout. It should be there about end of the very next week but not earlier.

[hummeltech]

This doesn't seem to be a macOS 12 image specific issue, the same gurumeditation issue is also present when using the macOS 13 image (which also began to occur at around the same time.) I am not sure about the status as it pertains to the macOS 11 image though.

[erik-bershel]

@leleliu008 couple more cents about this exact run: https://github.com/universal-ctags/ctags-nightly-build/actions/runs/6965555575/job/18954429674

We invested in upgraded hardware to support macOS 13 and are retiring older hardware, both of these changes impact macOS 12. The older generation Mac hardware was more efficient at nested virtualization. We recommend using larger linux runners for workflows that require nested virtualization. More information can be found here.

[leleliu008]

@erik-bershel Thanks for your advices. I will try the larger runners.

[saschagrunert]

I cannot confirm that the gurumeditation issues went away with runner version 20231115.3: https://github.com/kubernetes-sigs/security-profiles-operator/actions/runs/6978940376/job/18991380271

[leleliu008]

It still occures sometimes, but not that frequent. https://github.com/universal-ctags/ctags-nightly-build/actions/runs/6976821740/job/18986313203

[dmotte]

Just for info: unfortunately both errors are happening to me too:

• gurumeditation: https://github.com/dmotte/vagrant-k3sbox/actions/runs/7000668037/job/19041648685
• Error Type: Disconnect: https://github.com/dmotte/vagrant-podmanbox/actions/runs/7000685685

[hummeltech]

Yeah, it still seems to occasionally pull older runner image versions, I.E. 20230921.1:
https://github.com/dmotte/vagrant-k3sbox/actions/runs/7000668037/job/19041648685#step:1:9

[erik-bershel]

Hey there!
@leleliu008, @dmotte and @hummeltech, there is some amount of macOS-12 agents with the old image, it's being tracked in other issue and we are working on resolving it. #8642
About 'Error Type: Disconnect'. It is almost always about available resources boottleneck and we can recommend only to try optimise workflow. But I'll take a look at the available information about your runs just in case I may find something to fix.

[paulmueller]

I had this problem too and thought it had something to do with #8642. But I am now having promising success rates with reducing the memory and number of CPUs via Vagrantfile:

  # Reduce requirements so everythin runs on GitHub Actions
  config.vm.provider "virtualbox" do |v|
    v.memory = 1024
    v.cpus = 1
  end