Switch multi-platform tags to Linux only

[richlander]

Switch multi-platform tags to Linux only

We have published multi-platform images, referencing both Linux and Windows images for many years. For example, the 6.0 tag pulls a Debian Arm64 image on an Arm64 Ubuntu machine and a Nano Server x64 image on a Windows x64 machine (if configured to Windows Containers). For Windows, both an OS and a version are selected, with a matching algorithm, however it is known to have challenges. We believe that this matching algorithm is such a problem that we are planning on removing Windows images from our multi-platform tags, making them Linux only. This change will require the use of Windows-specific tags to use Windows images.

Problem

The problem is that the matching algorithm for containerd doesn't work well when a Windows version is higher than any version in the manifest list. In this scenario, you get the oldest not newest container image. That's not what you'd expect. It also means the final image is often not optimal if you build images on a client environment.

For example, if you are running on Windows 11, you will get a Windows Server 1809 era image, not a newer Windows Server 2022 era image.

Key point: Microsoft guides developers to adopt the latest client OS, but if they do that, then the Windows images they get via multi-platform tags (when using Docker on the client) will be the wrong ones.

This is partially due to the way that Windows Client and Server ship and also due to the containerd algorithm.

This issue doesn't affect Windows base images because those repos don't publish a tag that would include multiple Windows Server versions. In fact, this proposal would result in .NET images being the same as Windows base images in this respect. We're happy to reconsider this decision, but only if the Windows Server team publishes multi-platform tags, effectively expressing that they think that the scenario is important.

.NET multi-platform tags

The following demonstrates (admittedly cryptically) which tags are being discussed/targeted.

$ docker manifest inspect mcr.microsoft.com/dotnet/runtime | grep windows
            "os": "windows",
            "os": "windows",
$ docker manifest inspect mcr.microsoft.com/dotnet/runtime:latest | grep windows
            "os": "windows",
            "os": "windows",
$ docker manifest inspect mcr.microsoft.com/dotnet/runtime:6.0 | grep windows
            "os": "windows",
            "os": "windows",
$ docker manifest inspect mcr.microsoft.com/dotnet/runtime:6.0.1 | grep windows
            "os": "windows",
            "os": "windows",
            "os": "windows",
$ docker manifest inspect mcr.microsoft.com/dotnet/runtime:6.0.13 | grep windows
            "os": "windows",
            "os": "windows",

It is interesting to see that the Windows versions we support with these tags are not constant over time. That's another reason why including Windows versions in these tags is a poor design choice. We're able to fully control the Linux versions we include via these same tags, but have no control over the Windows versions. That's not great for a production-oriented offering.

Options

There are a few options we considered.

1. Status quo, but change guidance for Windows users to use Windows-specific tags.
2. Continue publishing Windows images in SDK multi-platform tags but not runtime ones.
3. Continue publishing Windows images for latest multi-platform tag only, for convenience.
4. Include only one Windows version in multi-platform tags, just like we do for Linux.
5. Remove Windows from all multi-platform tags.

We ended up choosing the last option.

We rejected the second option since it would often result in pulling the Windows layer twice. It's kindof the worst of all options.

We rejected the third option since it odd and also doesn't map to what the Windows Server team does.

Windows Server repos don't have a latest tag.

>docker pull mcr.microsoft.com/windows/nanoserver
Using default tag: latest
Error response from daemon: manifest for mcr.microsoft.com/windows/nanoserver:latest not found: manifest unknown: manifest tagged by "latest" is not found

We rejected the fourth option since users would need to reason about Windows container version compatibility when pulling our multi-platforms tags.

Some folks would see this, but when pulling just 8.0.

C:\Users\annie>docker pull mcr.microsoft.com/dotnet/runtime:6.0-nanoserver-ltsc2022
6.0-nanoserver-ltsc2022: Pulling from dotnet/runtime
a Windows version 10.0.20348-based image is incompatible with a 10.0.19045 host

Unfortunately, with this plan, Windows users will get an even less user-friendly error message.

We can demonstrate that with our existing Linux-only multi-platform tags.

>docker pull mcr.microsoft.com/dotnet/runtime:6.0-alpine
6.0-alpine: Pulling from dotnet/runtime
no matching manifest for windows/amd64 10.0.22621 in the manifest list entries

Plan

Update .NET 8 multi-platform images to be Linux-only. It's not really "Linux-only". Where we have multi-platform tags today that mix Windows and Linux together, the Linux distro is always Debian. That means that the latest, major.minor, and major.minor.patch tags will be Debian-only going forward.

We will update our guidance on the tags that Windows users should be using. It will also result in Nano Server and Windows Server Core being more symmetric. They will both require the use of OS and version-specific tags.

This change is breaking for upgrades. Windows users will not be able to simply update their Dockerfiles from 6.0 to 8.0, for example. They will need to switch to one of the following:

FROM mcr.microsoft.com/dotnet/sdk:8.0-nanoserver-ltsc2022
FROM mcr.microsoft.com/dotnet/sdk:8.0-nanoserver-1809

Three part versions can also be used, like mcr.microsoft.com/dotnet/sdk:8.0.1-nanoserver-ltsc2022.

Users can similarly adopt Windows Server Core.

FROM mcr.microsoft.com/dotnet/sdk:8.0-windowsservercore-ltsc2019
FROM mcr.microsoft.com/dotnet/sdk:8.0-windowsservercore-ltsc2022

Note that 8.0 tags are currently published with as 8.0-preview for all tags for the Preview period.

For example, Nano Server tags are currently published as:

• mcr.microsoft.com/dotnet/sdk:8.0-preview-nanoserver-ltsc2022
• mcr.microsoft.com/dotnet/sdk:8.0-preview-nanoserver-1809

We will update our samples per our updated guidance.

[mthalman]

A consequence of this plan that is important to realize is that developers won't be able to define a simple Dockerfile that can be used to produce both a Linux- and Windows-based image.

This is an example of a Dockerfile that is capable of being used cross-platform:

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS build
WORKDIR /source

COPY *.csproj .
RUN dotnet restore --use-current-runtime

COPY . .
RUN dotnet publish -c Release -o /app --use-current-runtime --self-contained false --no-restore

FROM mcr.microsoft.com/dotnet/runtime:7.0
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["dotnet", "dotnetapp.dll"]

You can use this Dockerfile to produce a Linux image or a Windows image, with no modifications to the content.

That same Dockerfile wouldn't work in the context of .NET 8 under this plan. Instead, you would have two options:

1. Define separate Dockerfiles that are Linux- and Windows-specific:

Linux:

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /source

COPY *.csproj .
RUN dotnet restore --use-current-runtime

COPY . .
RUN dotnet publish -c Release -o /app --use-current-runtime --self-contained false --no-restore

FROM mcr.microsoft.com/dotnet/runtime:8.0
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["dotnet", "dotnetapp.dll"]

Windows:

FROM mcr.microsoft.com/dotnet/sdk:8.0-nanoserver-ltsc2022 AS build
WORKDIR /source

COPY *.csproj .
RUN dotnet restore --use-current-runtime

COPY . .
RUN dotnet publish -c Release -o /app --use-current-runtime --self-contained false --no-restore

FROM mcr.microsoft.com/dotnet/runtime:8.0-nanoserver-ltsc2022
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["dotnet", "dotnetapp.dll"]

2. Define an ARG that lets you dynamically pass in the image tag that you want to target.

ARG TAG=8.0
FROM mcr.microsoft.com/dotnet/sdk:${TAG} AS build
WORKDIR /source

COPY *.csproj .
RUN dotnet restore --use-current-runtime

COPY . .
RUN dotnet publish -c Release -o /app --use-current-runtime --self-contained false --no-restore

FROM mcr.microsoft.com/dotnet/runtime:${TAG}
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["dotnet", "dotnetapp.dll"]

With the example Dockerfile shown above, it targets Linux by default but it can dynamically be configured to target Windows in the build command: docker build --build-arg TAG=nanoserver-ltsc2022 .

If there's any need to include a RUN instruction in the Dockerfile, it almost certainly will be an OS-specific Dockerfile. But I'm interested to know how common a scenario having a cross-platform Dockerfile is for people.

[richlander]

We should start by updating our guidance for Windows Container tags, for both .NET Core and .NET Framework.

[tfenster]

Wouldn't it make more sense to fix the underlying containerd bug instead of removing convenient functionality that helps especially developers new to the topic to get started quickly and without friction?

[richlander]

We have no reason to believe that the containderd issue will get resolved soon. We're happy to reconsider this change once that issue is resolved.

[mthalman]

This needs to be implemented at least by 8.0 Preview 4 to align with VS container tooling that needs to compensate for this change.

[richlander]

This change has been made.

C:\Users\rlander>docker pull mcr.microsoft.com/dotnet/runtime:7.0
7.0: Pulling from dotnet/runtime
Digest: sha256:c8e3a70be53ae450e193069c0e56fb716f8a1dc54375914c5e42bf969ff66754
Status: Image is up to date for mcr.microsoft.com/dotnet/runtime:7.0
mcr.microsoft.com/dotnet/runtime:7.0

C:\Users\rlander>docker pull mcr.microsoft.com/dotnet/nightly/runtime:8.0-preview
8.0-preview: Pulling from dotnet/nightly/runtime
no matching manifest for windows/amd64 10.0.22621 in the manifest list entries

C:\Users\rlander>docker pull mcr.microsoft.com/dotnet/nightly/runtime:8.0-preview-nanoserver-ltsc2022
8.0-preview-nanoserver-ltsc2022: Pulling from dotnet/nightly/runtime
Digest: sha256:25a75553788542ca19457b9d7db1b7cc607c5433d7da31c204bd8db24142473b
Status: Image is up to date for mcr.microsoft.com/dotnet/nightly/runtime:8.0-preview-nanoserver-ltsc2022
mcr.microsoft.com/dotnet/nightly/runtime:8.0-preview-nanoserver-ltsc2022

C:\Users\rlander>docker pull mcr.microsoft.com/dotnet/nightly/runtime:8.0-preview-nanoserver-1809
8.0-preview-nanoserver-1809: Pulling from dotnet/nightly/runtime
Digest: sha256:5ed81c273870f2c3827ffe901fe184681555ec4894a20873b87d6662a40a701d
Status: Image is up to date for mcr.microsoft.com/dotnet/nightly/runtime:8.0-preview-nanoserver-1809
mcr.microsoft.com/dotnet/nightly/runtime:8.0-preview-nanoserver-1809

The new approach encourages (actually forces) intentional targeting decisions.

C:\Users\rlander>docker run --rm mcr.microsoft.com/dotnet/runtime:7.0 cmd /c "ver"

Microsoft Windows [Version 10.0.17763.4131]

C:\Users\rlander>docker run --rm mcr.microsoft.com/dotnet/nightly/runtime:8.0-preview-nanoserver-1809 cmd /c "ver"

Microsoft Windows [Version 10.0.17763.4131]

C:\Users\rlander>docker run --rm mcr.microsoft.com/dotnet/nightly/runtime:8.0-preview-nanoserver-ltsc2022 cmd /c "ver"

Microsoft Windows [Version 10.0.20348.1607]

[mthalman]

Documented breaking change at dotnet/docs#35955