Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⚠️ QEMU (homebrew) was broken on Intel: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") ... QEMU has already exited #1742

Closed
AkihiroSuda opened this issue Aug 12, 2023 · 23 comments

Comments

@AkihiroSuda
Copy link
Member

AkihiroSuda commented Aug 12, 2023

Note

Colima users may still see QEMU binary /Users/<USER>/.colima/_wrapper/<HASH>/bin/qemu-system-x86_64 is not properly signed with the latest version of QEMU:

The warning is negligible if the VM is actually working.

The warning should not be printed if you use Lima directly without Colima:

limactl start template://docker

Update (Aug 27, 2023): The issue is solved again in 8.1.0_1 (Homebrew/brew#15903 , Homebrew/homebrew-core#140596 Homebrew/homebrew-core#140643). Run brew reinstall -f --force-bottle qemu to install the updated v8.1.0 bottle.

Update (Aug 23, 2023): This seems to be broken again in v8.1.0 😞 (Homebrew/homebrew-core#140244) . See the Workarounds below.

Update (Aug 14, 2023): The issue is now solved in Homebrew/homebrew-core#139492 .
Run brew reinstall -f --force-bottle qemu to install the updated v8.0.4 bottle.


Homebrew bottle of QEMU v8.0.4 (Intel) is was broken due to a signing issue: Homebrew/homebrew-core#139409

$ limactl start
...
[hostagent] Driver stopped due to error: "signal: abort trap" 
...
[hostagent] QEMU has already exited
...

$ qemu-system-x86_64 -accel hvf
qemu-system-x86_64: -accel hvf: Unknown Error
Abort trap: 6

$ codesign --verify /usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64 
/usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64: invalid signature (code or signature have been modified)
In architecture: x86_64

(The error message can be also [hostagent] Driver stopped due to error: "exit status 255")

Workarounds

Option 1: Downgrade QEMU to v8.0.3

brew uninstall qemu
curl -OSL https://raw.githubusercontent.com/Homebrew/homebrew-core/dc0669eca9479e9eeb495397ba3a7480aaa45c2e/Formula/qemu.rb
brew install ./qemu.rb

Option 2: Install QEMU from the source

brew uninstall qemu
brew install --build-from-source qemu

Option 3: Sign the QEMU binary locally

Lima v0.17.2 shows a prompt to suggest applying this workaround.

cat >entitlements.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF

codesign --sign - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-$(uname -m | sed -e s/arm64/aarch64/)

Thanks to @z0sen for reporting this in abiosoft/colima#777

@AkihiroSuda AkihiroSuda pinned this issue Aug 12, 2023
@AkihiroSuda AkihiroSuda added the area/homebrew Homebrew formula label Aug 12, 2023
@AkihiroSuda AkihiroSuda changed the title ⚠️ QEMU v8.0.4 seems broken: [hostagent] Driver stopped due to error: "exit status 255" ⚠️ QEMU v8.0.4 (homebrew) seems broken: [hostagent] Driver stopped due to error: "exit status 255" Aug 12, 2023
@afbjorklund
Copy link
Member

Lima project probably needs to have an optional QEMU installation tarball (in a different project, like alpine-lima).

@AkihiroSuda AkihiroSuda changed the title ⚠️ QEMU v8.0.4 (homebrew) seems broken: [hostagent] Driver stopped due to error: "exit status 255" ⚠️ QEMU v8.0.4 (homebrew) seems broken: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") Aug 14, 2023
@AkihiroSuda AkihiroSuda changed the title ⚠️ QEMU v8.0.4 (homebrew) seems broken: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") ⚠️ QEMU v8.0.4 (homebrew) is broken: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") ... QEMU has already exited Aug 14, 2023
@AkihiroSuda
Copy link
Member Author

The issue is now solved:

@AkihiroSuda AkihiroSuda changed the title ⚠️ QEMU v8.0.4 (homebrew) is broken: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") ... QEMU has already exited ⚠️ QEMU v8.0.4 (homebrew) was broken: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") ... QEMU has already exited Aug 14, 2023
@AkihiroSuda AkihiroSuda changed the title ⚠️ QEMU v8.0.4 (homebrew) was broken: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") ... QEMU has already exited ⚠️ QEMU v8.0.4 (homebrew) was broken on Intel: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") ... QEMU has already exited Aug 14, 2023
@benmoss
Copy link

benmoss commented Aug 16, 2023

Still seeing this when I run colima start after running brew reinstall -f --force-bottle qemu:

time="2023-08-16T11:22:47-04:00" level=warning msg="QEMU binary "/Users/mossity/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/bin/qemu-system-aarch64" is not properly signed with the "com.apple.security.hypervisor" entitlement" error="binary "/Users/mossity/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/bin/qemu-system-aarch64" seems signed but lacking the "com.apple.security.hypervisor" entitlement"
time="2023-08-16T11:22:47-04:00" level=warning msg="You have to sign the QEMU binary with the "com.apple.security.hypervisor" entitlement manually. See #1742 ."

Not sure if this is a lima or a colima problem. The VM does seem to start fine in spite of the warning.

@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Aug 16, 2023

@benmoss

  • ARM Mac or Intel Mac?
  • Does limactl start (not colima start) print the same warning? If limactl start does not print a warning, it is an issue of colima.
  • Output of codesign --display --entitlements - --xml /opt/homebrew/Cellar/qemu/8.0.4/bin/qemu-system-aarch64 ? (qemu binary path might be different)

@benmoss
Copy link

benmoss commented Aug 16, 2023

  • I'm on ARM
  • limactl start doesn't print any warning, says "QEMU binary "/opt/homebrew/bin/qemu-system-aarch64" seems properly signed with the "com.apple.security.hypervisor" entitlement
  • codesign --display ... shows that it has the hypervisor entitlement

So it looks like it is a colima problem. Feel free to ignore!

@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Aug 23, 2023

@AkihiroSuda AkihiroSuda reopened this Aug 23, 2023
@AkihiroSuda AkihiroSuda changed the title ⚠️ QEMU v8.0.4 (homebrew) was broken on Intel: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") ... QEMU has already exited ⚠️ QEMU (homebrew) is broken on Intel: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") ... QEMU has already exited Aug 23, 2023
@afbjorklund
Copy link
Member

Besides the bloat, this seems like another good reason to not use Homebrew for the QEMU installation.

@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Aug 23, 2023

I guess we should also rush to make vz default?
vz also had a different incompatibility issue though (#1577).

(But when Homebrew fails to sign qemu binaries it may fail to sign lima binaries too, so switching to vz may not improve the situation)

@afbjorklund
Copy link
Member

Adding a check for the entitlement to the brew formula would probably also be a good idea, to stop re-re-regressions from happening again in the future

@AkihiroSuda
Copy link
Member Author

Adding a check for the entitlement to the brew formula would probably also be a good idea, to stop re-re-regressions from happening again in the future

Already done in Lima v0.17.2:

(The Y/n prompt defaults to n when --tty=false though)

GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Aug 23, 2023
The Flux CI step often fails because the `attrs` module cannot be found
on the system. This commit fixes this behaviour by explicitly installing
the module through `pip` in the CI workflow.

As per issue lima-vm/lima#1742, the QEMU 8.1.0 Homebrew package is
broken on Intel architectures. This commit adds an additional step to
uninstall QEMU and recompile it from binary, which fixes the issue.
GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Aug 23, 2023
The Flux CI step often fails because the `attrs` module cannot be found
on the system. This commit fixes this behaviour by explicitly installing
the module through `pip` in the CI workflow.

As per issue lima-vm/lima#1742, the QEMU 8.1.0 Homebrew package is
broken on Intel architectures. This commit adds an additional step to
downgrade QEMU to v8.0.3, which fixes the issue.
GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Aug 23, 2023
The Flux CI step often fails because the `attrs` module cannot be found
on the system. This commit fixes this behaviour by explicitly installing
the module through `pip` in the CI workflow.

As per issue lima-vm/lima#1742, the QEMU 8.1.0 Homebrew package is
broken on Intel architectures. This commit adds an additional step to
downgrade QEMU to v8.0.3, which fixes the issue.
GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Aug 23, 2023
The Flux CI step often fails because the `attrs` module cannot be found
on the system. This commit fixes this behaviour by explicitly installing
the module through `pip` in the CI workflow.

As per issue lima-vm/lima#1742, the QEMU 8.1.0 Homebrew package is
broken on Intel architectures. This commit adds an additional step to
compile QEMU from source, which fixes the issue.
GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Aug 23, 2023
The Flux CI step often fails because the `attrs` module cannot be found
on the system. This commit fixes this behaviour by explicitly installing
the module through `pip` in the CI workflow.

As per issue lima-vm/lima#1742, the QEMU 8.1.0 Homebrew package is
broken on Intel architectures. This commit adds an additional step to
compile QEMU from source, which fixes the issue.
GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Aug 24, 2023
The Flux CI step often fails because the `attrs` module cannot be found
on the system. This commit fixes this behaviour by explicitly installing
the module through `pip` in the CI workflow.

As per issue lima-vm/lima#1742, the QEMU 8.1.0 Homebrew package is
broken on Intel architectures. This commit upgrades the
`douglascamata/setup-docker-macos-action` to version `v1-alpha.9` to
overcome this issue.
GlassOfWhiskey added a commit to alpha-unito/streamflow that referenced this issue Aug 24, 2023
The Flux CI step often fails because the `attrs` module cannot be found
on the system. This commit fixes this behaviour by explicitly installing
the module through `pip` in the CI workflow.

As per issue lima-vm/lima#1742, the QEMU 8.1.0 Homebrew package is
broken on Intel architectures. This commit upgrades the
`douglascamata/setup-docker-macos-action` to version `v1-alpha.9` to
overcome this issue.
GlassOfWhiskey added a commit to alpha-unito/streamflow-postgresql that referenced this issue Aug 24, 2023
As per issue lima-vm/lima#1742, the QEMU 8.1.0 Homebrew package is
broken on Intel architectures. This commit upgrades the
`douglascamata/setup-docker-macos-action` to version `v1-alpha.9`
to overcome this issue.
GlassOfWhiskey added a commit to alpha-unito/streamflow-postgresql that referenced this issue Aug 24, 2023
As per issue lima-vm/lima#1742, the QEMU 8.1.0 Homebrew package is
broken on Intel architectures. This commit upgrades the
`douglascamata/setup-docker-macos-action` to version `v1-alpha.9`
to overcome this issue.
@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Aug 27, 2023

The issue is solved again in 8.1.0_1 (Homebrew/brew#15903 , Homebrew/homebrew-core#140596 Homebrew/homebrew-core#140643).

Run brew reinstall -f --force-bottle qemu to install the updated v8.1.0 bottle.

@janvda
Copy link

janvda commented Aug 30, 2023

It doesn't seem to be fixed for me.
When starting colima it is still saying:

/Users/jan/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64\" is not properly signed 

Since I have this issue I can also no longer build containers.

Here below the complete log

mac-jan:my-question-generator jan$ brew reinstall -f --force-bottle qemu
==> Fetching qemu
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/manifests/8.1.0_1-1
Already downloaded: /Users/jan/Library/Caches/Homebrew/downloads/e9d42585f1662261d504025b0202672ee9fb0633dd8be378c825c484b68ee297--qemu-8.1.0_1-1.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/qemu/blobs/sha256:845671e9625736ab6a15108d369e47f5a6b20b8f6d0e99ba1a3f39d18df1c94d
Already downloaded: /Users/jan/Library/Caches/Homebrew/downloads/bd376e9d023c700e820d08094f41d7ce9e3e8befb13dd644aeb39e91633ad4db--qemu--8.1.0_1.ventura.bottle.1.tar.gz
==> Reinstalling qemu 
==> Pouring qemu--8.1.0_1.ventura.bottle.1.tar.gz
🍺  /usr/local/Cellar/qemu/8.1.0_1: 162 files, 528.8MB
==> Running `brew cleanup qemu`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
mac-jan:my-question-generator jan$ brew install colima
==> Downloading https://formulae.brew.sh/api/formula.jws.json
######################################################################################################################################################################################### 100.0%
==> Downloading https://formulae.brew.sh/api/cask.jws.json
######################################################################################################################################################################################### 100.0%
Warning: colima 0.5.5 is already installed and up-to-date.
To reinstall 0.5.5, run:
  brew reinstall colima
mac-jan:my-question-generator jan$ colima start
INFO[0000] starting colima                              
INFO[0000] runtime: docker                              
INFO[0000] preparing network ...                         context=vm
INFO[0000] starting ...                                  context=vm
> "QEMU binary \"/Users/jan/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64\" is not properly signed with the \"com.apple.security.hypervisor\" entitlement" error="failed to run [codesign --verify /Users/jan/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64]: exit status 1 (out=\"/Users/jan/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64: code object is not signed at all\\nIn architecture: x86_64\\n\")"
> You have to sign the QEMU binary with the "com.apple.security.hypervisor" entitlement manually. See https://github.com/lima-vm/lima/issues/1742 .
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/jan/.lima/colima/serial*.log")
> SSH Local Port: 51472
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"

@AkihiroSuda
Copy link
Member Author

@janvda Using Lima without Colima may work as a workaround:
limactl start template://docker

Removing ~/.colima/_wrapper may work too.

@edw-eqix
Copy link

edw-eqix commented Aug 31, 2023

I was able to work around the issue by manually signing the wrapper link:
codesign --sign - --entitlements entitlements.xml --force /Users/eq/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64

Along with these binaries (which are signed, now, after I trod a path of destruction through all plausible binaries and therefore cannot un-sign to test and see whether signing them is needed):
/usr/local/bin/qemu-system-x86_64
/usr/local/Cellar/qemu/8.1.0_1/bin/qemu-system-x86_64

From my slightly-cleaned-up-for-display error below, something seems to be checking whether the wrapper itself is signed. On my system, the wrapper is a link to /usr/local/bin/colima which is signed. So I'm pretty confused bout this check but things are working.

INFO[0000] starting ...                                  context=vm
> Using the existing instance "colima"
> "QEMU binary
/Users/eq/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64 
is not properly signed with the com.apple.security.hypervisor entitlement" 
error="failed to run [
  codesign --verify /Users/eq/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64
]: exit status 1 (out=
  /Users/eq/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64:
  code object is not signed at all
  In architecture: x86_64
)"
> You have to sign the QEMU binary with the "com.apple.security.hypervisor" entitlement manually.
See https://github.com/lima-vm/lima/issues/1742 .
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/ewehrwein/.lima/colima/serial*.log")
> SSH Local Port: 51848
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
^C

@AkihiroSuda AkihiroSuda changed the title ⚠️ QEMU (homebrew) is broken on Intel: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") ... QEMU has already exited ⚠️ QEMU (homebrew) was broken on Intel: [hostagent] Driver stopped due to error: "signal: abort trap" (or "exit status 255") ... QEMU has already exited Sep 4, 2023
@remidebette
Copy link

Hi,
I just noticed today after coming back from holidays that Colima is broken.
I am on a macbook pro M1. I have applied the same steps as @janvda to no avail.

This issue should be reopened

@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Sep 5, 2023

I am on a macbook pro M1

You are hitting something different (maybe https://gitlab.com/qemu-project/qemu/-/issues/1864 , which can be worked around by brew install --HEAD qemu), as the OP is about Intel.

@janvda
Copy link

janvda commented Sep 5, 2023

have applied the same steps as @janvda to no avail.

FYI The following workaround worked for me on my macbook pro with intel processors:

@connorblack
Copy link

connorblack commented Sep 8, 2023

@edw-eqix Beautiful - this is the current workaround for M1 macbooks until the maintainers release a fix

@RicardoMonteiroSimoes
Copy link

I've just tried it now on my M1 mac, and this is the current output with the most recent brew install --HEAD qemu:

> "QEMU binary \"/Users/risi/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/bin/qemu-system-aarch64\" is not properly signed with the \"com.apple.security.hypervisor\" entitlement" error="binary \"/Users/risi/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/bin/qemu-system-aarch64\" seems signed but lacking the \"com.apple.security.hypervisor\" entitlement"
> You have to sign the QEMU binary with the "com.apple.security.hypervisor" entitlement manually. See https://github.com/lima-vm/lima/issues/1742 .

@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Sep 18, 2023

I've just tried it now on my M1 mac, and this is the current output with the most recent brew install --HEAD qemu:

> "QEMU binary \"/Users/risi/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/bin/qemu-system-aarch64\" is not properly signed with the \"com.apple.security.hypervisor\" entitlement" error="binary \"/Users/risi/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/bin/qemu-system-aarch64\" seems signed but lacking the \"com.apple.security.hypervisor\" entitlement"
> You have to sign the QEMU binary with the "com.apple.security.hypervisor" entitlement manually. See https://github.com/lima-vm/lima/issues/1742 .

The story is different for Colima:

The warning is negligible if the VM is working.

The warning shouldn’t be printed if you use Lima directly without colima (limactl start template://docker)

@RicardoMonteiroSimoes
Copy link

Thanks for the response, my bad, was on the wrong repo :)

@msankhala
Copy link

From my slightly-cleaned-up-for-display error below, something seems to be checking whether the wrapper itself is signed.

After signing the actual binary with the following command:

codesign --sign - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-$(uname -m | sed -e s/arm64/aarch64/)

I also signed the wrapper with something like:

codesign --sign - --entitlements entitlements.xml --force ~/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64

and it worked. Now It shows me message like this:

INFO[0010] starting ...                                  context=vm
> Using the existing instance "colima"
> QEMU binary "/Users/mutant/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64" seems properly signed with the "com.apple.security.hypervisor" entitlement
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/mutant/.lima/colima/serial*.log")
> SSH Local Port: 60997
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"

@AkihiroSuda AkihiroSuda unpinned this issue Nov 15, 2023
@DinoChiesa
Copy link

As above, This hint (Option 3, local signing), worked for me. Intel Mac, Sonoma 14.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants