QEMU support for ARM64

[raffaeler]

I need to debug from Windows some code running inside an ARM64 container (on HyperV / docker which uses QEMU to run ARM code).
The code contains a lot of generated unsafe code using 64 bit pointers that is running well on both Windows x64 and Linux x64.
A first test on the physical device (Ubuntu 64 on Rasberry PI 3) did not succeed without apparent errors, therefore I would like to iterate the development inside a container to ease the development and later the automatic tests phases.

Is that currently supported? What are the limitations?
/cc @richlander

[richlander]

This test case exercises a problem case:

C:\git\dotnet-docker\samples\complexapp>docker run --rm -v %cd%:/app -w /app/tests mcr.microsoft.com/dotnet/core/sdk:3.0-bionic-arm64v8 dotnet test --logger:trx
Unknown QEMU_IFLA_INFO_KIND ipip
Unknown QEMU_IFLA_INFO_KIND ip6tnl

This is the source if you want to try it: https://github.com/dotnet/dotnet-docker/tree/samples/samples/complexapp

[raffaeler]

I did a similar test using your same base image but running interactively:

docker run --rm -ti mcr.microsoft.com/dotnet/core/sdk:3.0-bionic-arm64v8

Then I created a new console app:

# dotnet new console
Unknown QEMU_IFLA_INFO_KIND ipip
Unknown QEMU_IFLA_INFO_KIND ip6tnl
Unknown QEMU_IFLA_INFO_KIND ipip
Unknown QEMU_IFLA_INFO_KIND ip6tnl
The template "Console Application" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /raf/raf.csproj...
Unknown QEMU_IFLA_INFO_KIND ipip
Unknown QEMU_IFLA_INFO_KIND ip6tnl
Unknown QEMU_IFLA_INFO_KIND ipip
Unknown QEMU_IFLA_INFO_KIND ip6tnl
  Restore completed in 1.42 sec for /raf/raf.csproj.


Restore succeeded.

Unknown QEMU_IFLA_INFO_KIND ipip
Unknown QEMU_IFLA_INFO_KIND ip6tnl

And finally dotnet run:

# dotnet run
Unknown QEMU_IFLA_INFO_KIND ipip
Unknown QEMU_IFLA_INFO_KIND ip6tnl
Unknown QEMU_IFLA_INFO_KIND ipip
Unknown QEMU_IFLA_INFO_KIND ip6tnl
Hello World!

It worked but with lots of those weird messages.
How much can I trust the code running there?

[janvorli]

@lpereira do you happen to know what these QEMU messages mean?

[lpereira]

When Qemu is running in user mode, it emulates the CPU and thunks the syscalls to the host. So, if the program running under it tries to make a syscall with something that's not supported by its translation layer, it might give warnings such as these. In case of these particular warnings, it's being generated here; from a cursory glance, it seems to be part of the Netlink translation layer. As you can see from the code, it tries to check if the name is either "bridge" or "tun", and logs anything else.

This doesn't answer @raffaeler's question, but: this isn't an issue with Qemu, but with the program running under it. We'd need to investigate why that program is trying to use Netlink APIs with these parameters. (FWIW, "ipip" is IP in IP and "ip6tnl" is a similar thing, tunneling IPv4 and IPv6 over IPv6: both are IP tunneling protocols, which might give a hint on what to look for.) Another point that might help investigating this: on a quick web search for these strings, other people are reporting the same issues in the same kind of environment (ARM Docker), but unrelated to .NET.

[raffaeler]

Thank you @lpereira
the steps I am doing are exactly the same I published in this thread. The docker client is the latest updated from the stable channel.
Not sure if the local networking configuration of my machine can influence these warnings ... my guess was that the container should only see the virtual interface mapped by the docker client.

[mangod9]

Moving this to future. @AntonLapounov can perhaps determine what is required here.

[richlander]

Last time we investigated this, I believe we determined that the issue was in a dependency (from the Linux package manager). If that's the case, then we're not in a good position to enable the scenario. It is probably worth re-validating that, and then file an issue on the given library. We don't want to be in the position where we have to support a scenario that we cannot control.

@jkotas had well-defined views on this last time we talked about this. I believe I'm mostly parroting his viewpoint.

[AntonLapounov]

According to the comments above, dotnet new and dotnet run commands completed successfully modulo QEMU warnings, which may be ignorable. I cannot comment on potential limitations as this is not one of the scenarios we test.

[richlander]

I don't believe I've had that much success with QEMU. It usually fails more quickly for me.

Stepping back, I think it would be cool to support QEMU but we've had very few requests. Perhaps this will increase as Arm64 becomes more popular. I'd like to wait for more call for this before investing. That said, it would be a fine .NET 7 investigation if folks have cycles for it.

[richlander]

This is relevant: https://gitlab.com/qemu-project/qemu/-/issues/249

[raffaeler]

Thank you @richlander, crossing fingers.
Should they fix this, how would you feel confident to support the scenario?

[richlander]

Great question @raffaeler. I don't have info to answer that, but quite possibly.

[stefandxm]

With NET7 i can now build if i use DOTNET_EnableWriteXorExecute=0 (export DOTNET_EnableWriteXorExecute=0)

[MichalPetryka]

With NET7 i can now build if i use DOTNET_EnableWriteXorExecute=0 (export DOTNET_EnableWriteXorExecute=0)

As I've noted in dotnet/sdk#27190, disabling DOTNET_EnableWriteXorExecute made the SDK go further for me but still crashed. But as also noted there, .Net 6 works fine with QEMU in my case.

[richlander]

I added a support statement about QEMU to clarify this: https://github.com/dotnet/core/blob/main/release-notes/7.0/supported-os.md#qemu

We are not testing QEMU and the fact that the referenced qemu-project issue has not gotten traction is an indication that support might be difficult.

We're working on improving the Docker use case for QEMU so that it won't be needed. We should have that resolved relatively soon.

I'm closing this issue since the original question has been answered. If folks want to continue to explore making QEMU work, that's fine. You can continue to do it on this issue or open a new issue with a descriptive title.

[goncalo-oliveira]

So... what's the reality with .NET and QEMU? .NET 7 (and probably .NET 8) not supported?
Where does that leave anyone needing to build ARM images on AMD and vice versa?
Are the only workarounds 1. Use .NET 6 or 2. Use separate Dockerfile for each arch?

[richlander]

Great timing. We have a solution for this now.

dotnet/dotnet-docker#4388 (comment)

I'll be writing up more about it, but you can try what we've done now.