Should we expect Windows container base images labelled 21H1/10.0.19043.X?

[TBBle]

In a discussion related to adamrehn/ue4-docker#164, we realised with growing concern that there doesn't seem to be hide-nor-hair of either a Windows Server SAC 21H1 release, and neither 21H1 nor 10.0.19043.985 are listed in the tags on https://hub.docker.com/_/microsoft-windows-servercore

That said, I also noticed that the Windows Server container update history has always marked the feature-enablement releases (1909, 20H2) with the same kernel as their base release (1903, 2004), which matches my existing intuition that those containers would be cross-compatible, but conflicts with the Windows container version compatibility docs regarding process isolation.

So for those use-cases where it's useful to have a Windows 10 client with the recommended release (currently 21H1), even if there's no matching Windows Server SAC release to go to production with, do any of the following apply?

• ❌ Process-isolation of 2004 and 20H2 images on 21H1 is supported, and the cross-version compatibility tables were simply overlooked?
• ❌ There will be a 21H1 set of container images in the June updates, and it's only missing now because the May update of the container images for 10.0.1904n.985 was a week before Windows 10 21H1 was released?
• ❌ Is there a Windows Server SAC 21H1 release in the pipeline that will bring 10.0.19043.X container images, and we just haven't heard much about it because everyone's focused on (excited for) Windows Server LTSC 2022?

Edit: None of the three dot-points above have proved out, in retrospect.

[slonopotamus]

I want to add that Windows base image also still doesn't exist for 21H1 after a month since its release.

[TBBle]

I think the "Windows" base image, is still technically Windows Server, just with more DLLs, rather than being a Windows 10 (Client) container image.

That might be why it's being replaced with the new "Windows Server" container image, since that will be less confusing. I suspect they had the same build source, "Windows Server with Desktop Experience".

I saw mention the other day that there won't be a 21H1 SAC release.

So I guess 21H1 is the point where Windows 10 and Windows Server diverge now, rather than 21H2/LTSC 2022 as I would have predicted. I don't know if there's any plans to reconverge them or not... I feel like the 20H2/21H1 feature update releases took quite a few people by surprise, so it's an open field for predictions now.

[TBBle]

@adamrehn inadvertently tested a Windows Server 20H2-based container image on a Windows 10 21H1, and it indeed failed to run in Process Isolation mode.

[adamrehn]

@TBBle I had also assumed that mcr.microsoft.com/windows was derived from Windows Server, but it actually reports itself as a client version of Windows 10 when querying the system information, and the announcement for the new Windows Server image variant confirms this:

Some of the constraints are by design because that Windows container image is built from a full Windows Client edition and enabled to run on Windows Server.

There also doesn't seem to be any suggestion that the new container image will replace the old one, but rather that both will coexist as available options for developers who are weighing image size against application compatibility.

[adamrehn]

The above information of course makes it all the more puzzling that there isn't a Windows 10 version 21H1 container image available, especially given that 20H2 images fail to run in process isolation mode under 21H1 host systems.

[TBBle]

Ah, I had missed that note that the Windows image was derived from the Client edition.

The reason I assume the Windows Server image replaces the Windows image, is that the Windows insider image tags stop at build 20324, and the Windows Server insider image tags start at build 20329, weeks before the latter was formally announced with build 20344 (the last Insider build before Windows Server 2022 Preview locked down).

It also makes sense to me to drop a Windows 10-based image, since there was no Windows 10 release from that series for installation after 20279, the next Insider Preview was a 21xxx release from a different branch, so this is basically keeping a client build alive for a fairly niche use-case that should be covered by the Windows Server image. There won't be a Windows 10 release matching the kernel for Windows Server 2022 LTSC, so this isn't just a temporary "Insider-only" situation anymore.

[adamrehn]

The one concern I would have about the Windows Server base image replacing the Windows client base image is whether the former will have SAC releases like the latter has up until now. For both Windows Server 2016 and 2019, the Desktop Experience component was exclusive to LTSC releases, and the new Windows Server image is built with Desktop Experience as a key component. I've not seen any indication yet as to whether Windows Server 2022 will reflect its predecessors in this regard (Server Core and Desktop Experience configurations available for LTSC, only Server Core available for SAC.)

[BumbrT]

Any update on this? Can i launch windows containers with process isolation on w10 pro 21H1 laptop?
Downloaded 20H2 container and got " The container operating system does not match the host operating system." error.

[adamrehn]

@BumbrT as far as we can tell, there's currently no way to run process-isolated containers under Windows 10 version 21H1. If your machine has an Intel CPU then you can run 20H2 in a Hyper-V virtual machine with nested virtualisation enabled in order to use both process isolation mode and Hyper-V isolation mode as needed. If your machine has an AMD CPU like mine does then the best workaround I've found is to boot a copy of 20H2 from a VHDX file using Native Boot, which as a bare metal installation also has the benefit of being able to run GPU accelerated Windows containers. If you're interested in doing this, it's pretty easy to configure:

• Obtain an installation ISO for Windows 10 version 20H2 (either using Rufus or through your Microsoft account if you have a Visual Studio subscription or a volume license that includes software assurance.)

• Use my Windows Insider VHDX Boot script to prepare a VHDX file and add a boot entry for it, modifying the variables (and optionally the boot entry description string) to point to the Windows 10 version 20H2 ISO file rather than an Insider Preview ISO.

[gerwim]

Still no updates from Microsoft? @weijuans-msft?

[doctorpangloss]

This is okay. @weijuans-msft removing the feature update is the simplest workaround.

[weijuans-msft]

The next Windows Server release is Windows Server 2022, an LTSC. There is no Windows Server version 21H1 or any container image of that version planned.

Curious why folks need to run process isolated containers on Windows 10. Process isolated containers are officially only supported on Windows Server.

You could try Windows 11 with Windows Server 2022 container images with process isolated containers.

[TBBle]

Windows Server 2022 Preview is build 20348, Windows 11 Insider Preview is 22000. Will the latter run the former process-isolated?

Also, there are no Windows Server 2022 Preview container images for build 20348, the most recent builds on Docker Hub are for the last Windows Server 2022 Insider Preview, build 20344.

There are probably a lot of use-cases for running process-isolated containers on Windows 10. My primary ones are:

• Iterating on large-scale container software (like ue4-docker) on my development environment.
  • Hyper-V isolation is blocked here because docker build doesn't support --cpu-count on Windows, and with the default (2 core) available, an 8-hour build in process isolation becomes more than 24 hours.
  • My desktop machine is significantly more powerful than a server I would be allocated if I needed to use Windows Server to iterate on the software, unless I pull one of the actual build-pipeline servers which slows everyone else's builds down.
• Using utility containers (e.g. ue4-docker, again) on my desktop machine. I don't want to need to copy my work to a Windows Server machine, run the container, and then copy the results back every time.
  • I don't do this much, mostly when working on the above.
• Actually working on the container infrastructure itself as a hobby project. I don't have a license for Windows Server personally, so Windows 10 is the only platform where I can work on integrating WCOW features into containerd, or contribute changes to hcsshim, etc.
  • Without process isolation, I would be building and testing a code-path that is significantly different than what is intended for production use.
  • Docker's WCOW integration tests, for example, don't support Hyper-V isolation currently. Neither does Kubernetes currently.

[adamrehn]

@doctorpangloss the NVIDIA drivers are already WDDM 2.7 compliant, and offscreen rendering with DirectX works just fine in GPU accelerated Windows containers. If you copy the right files to System32 at runtime you can even use other APIs such as CUDA and NVENC. See these articles for details:

• https://unrealcontainers.com/blog/offscreen-rendering-in-windows-containers/
• https://unrealcontainers.com/blog/enabling-vendor-specific-graphics-apis-in-windows-containers/

With the code from those articles, I've been able to successfully run Unreal Engine applications inside Windows containers (with raytracing enabled) and stream the output to a web browser over WebRTC using the Unreal Engine Pixel Streaming system. Testing this functionality has been my primary use case for process-isolated Windows containers outside of the ue4-docker development use cases that @TBBle already mentioned above.

[gerwim]

Curious why folks need to run process isolated containers on Windows 10.

We develop .NET applications (full framework, not .NET 5 (yet)) and use process isolation to run the application. Without process isolation, there's a significant overhead while developing. Using process isolation is much better performant and way more stable.

Process isolated containers are officially only supported on Windows Server.

According to the MS docs (https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/hyperv-container#isolation-examples), process isolation on Windows 10 is only for development / testing and production workloads should only be deployed on Windows Server (which makes sense).

[slonopotamus]

Curious why folks need to run process isolated containers on Windows 10.

+1 to #117 (comment).

We develop containers on our desktop machines. Process isolation is faster and has smaller memory overhead. Process isolation supports device sharing unlike Hyper-V isolation.

[mika-fischer]

Also chiming in to say that not having 21H1 images hurts us since we use them for application development (i.e. we cannot upgrade the machines, and had to downgrade the ones that already updated).

Hyper-V is not a viable alternative due to it needing significantly more resources (i.e. it's a lot slower) and some issues with the version of MS build tools that we're using (dotnet/msbuild#3360).

It's very disappointing that MS seemingly discontinued this without much fuss. It can't be that hard to provide these images...

[marius-luca-87]

Curious why folks need to run process isolated containers on Windows 10.

One clear advantage of having process isolation for containers on a development machine is RAM usage.
A running Hyper-V can require blocking a lot of RAM for the container (the maximum RAM needs to be known when starting the container).
With process isolation a developer can start as many containers as necessary since the RAM will be shared with the host.
In process isolation you only incur the RAM cost when starting tasks in containers versus having it reserved all the time with Hyper-V witch brings a lot more flexibility.

[TBBle]

It's very disappointing that MS seemingly discontinued this without much fuss. It can't be that hard to provide these images...

I suspect it might be actually hard to provide the servercore and nanoserver images, because there was no Windows Server SAC release to build them from.

I agree that it should have been messaged better (or messaged at all...) so that alternative measures could be taken earlier. The missing Windows Server 2022 Preview images makes me wonder what MS's own Containers team are using for their development flow.

Also, an addendum for my previous answer: My development box at home doesn't qualify for Windows 11, as I have a 5th-Generation i7, and Windows 11 requires 7th Generation (or 8th?). As it is, this issue is holding me back from the 21H1 upgrade, and I wouldn't upgrade to Windows 11 unless I knew there was a process-isolation-compatible Windows base image set available to me after the upgrade.

[TBBle]

Well, I see that Windows Server SAC is dead, and with Windows Server Insider Previews also (temporarily) gone, and no container base image for Windows Server 2022 Preview, the answer to this question is "No, and unless you grabbed the Insider ISOs already, you can't use Windows Server for this either".

[adamrehn]

Interesting. Although Azure Stack HCI is stated to be the container-focussed replacement for SAC, it looks like that doesn't have a release newer than 20H2 at the moment either: https://docs.microsoft.com/en-us/azure-stack/hci/release-information

[TBBle]

Azure Stack HCI 20H2 is build 17784, i.e. it's an iteration on Windows Server 2019 (build 17763) underneath but was released in H2 2020; release name is the only relation it has to Windows 10 20H2 or Windows Server SAC 20H2, aka build 19042.

The Azure Stack HCI 21H2 Public Preview is build 20348, i.e. Windows Server 2022. So at least to start with, it should be process-isolation-compatible with Windows Server 2022, assuming neither Server 2022 Preview nor Azure Stack HCI 21H2 Preview see a build bump before leaving Preview.

So it'll be interesting to see if Azure Stack HCI sees SAC-frequency build updates or not. It'd be weird if it did, as that requires either making cross-build process isolation feasible and supported, or releasing new base images every six months to go with the new SAC release of Azure Stack HCI, without a Windows Server SAC release to go with them. I guess that's feasible, if MS're not updating the new Windows Server image for Azure Stack HCI, just Server Core and (hopefully...) Nanoserver.

Despite the SAC announcement pointing at Azure Stack HCI for containers, Azure Stack HCI docs don't seem to discuss containers, except in the context of AKS-on-HCI, and that doesn't yet acknowledge the Azure Stack HCI 21H2 Public Preview. I've never looked at HCI, so I don't know if it's just going unmentioned that you could run DockerMSFTProvider like you would on Windows Server if that's the use-case you have.

Anyway, if Azure Stack HCI is going to be semi-annually updated, then we'll be back to where we were before dropping SAC (i.e. a Server LTSC release or two plus some number of more-frequently-released targets to support when building images for WCOW), except we lose the ability to develop on Windows 10.

[TBBle]

no container base image for Windows Server 2022 Preview

Container base images for the Windows Server 2022 Preview have been published to the "Insider" pages on Docker Hub now.

[vrapolinario]

Any updates to Windows 10 will continue to be on the same build (19043), so this applies to any future update to Windows 10.

[TBBle]

To be sure I understand the blog post completely, is this accurate:

• the intention is that for any host kernel version 20348 or later, either Client (Windows 11) or Server (Windows Server 2022), process isolation will be supported for 20348-based images; and
• container base images will be released with Windows Server LTSC releases, but will be usable in process isolation on kernels newer than that Windows Server release.

If I'm following correctly, this would mean an Azure Stack HCI release after 21H2 (i.e. with a kernel newer than 20348) can also be expected to run ltsc2022 base images in process isolation correctly?

The other thing I'm concluding from this is that the ltsc2022 base image is supported for 5 or 5+5 years, and this include being able to run in process isolation on newer kernels, so in theory rather than needing a slew of base images (five or six today, but with SAC ended and LTSC 2016 expiring soon, it was already trending towards two), we can just manage ltsc2022 images for the next 5 or 10 years, and Windows Server 2025 would still run those perfectly well, and it's only Windows Server 2028 where we might need to use a new image base. And ltsc2019 if required, as a legacy of the "before times". ^_^

That'll be a nice cost-of-play reduction for Windows containers, particularly for open-source where you have to either limit your users' options, or maintain a wide coverage of support. It'll help decouple migrations, allowing upgrading of, e.g., host systems in a build cluster, and validating them while the same images keep running on the existing, working hardware.

Obviously I'm making up "2025" and "2028" there, but those numbers illustrate one more question, if the above holds: Is the ltsc2022 image going to be supported by 5 years of kernels, or 10 years? Does "extended support" mean "It works with kernels released in that time", or just "it keeps working with kernels released while it was mainstream"?

Or perhaps I am conflating things, and the upper limit of "kernel versions that can host ltsc2022 images in process isolation" will be defined some other way.

[slonopotamus]

Windows Server 2025 would still run those perfectly well

Err... Why do you think that? I'm under impression that ltsc2022 images are not going to be supported by Windows Server after 2022:

customers will be able to run their Windows Server 2022 container images with either process or Hyper-V isolation on any build of Windows Server 2022 or Windows 11.

The only improvement this new scheme has over what we had in the past is that now Microsoft promises not to break ABI in minor updates (like Feb 2020 issue).

I do not count as improvement the fact that ltsc2022 images can be run on Windows 11 with process isolation. We could run containers with process isolation on client Windows since 1809 and it is a regression that 21H1/21H2 doesn't have a corresponding image that it can use for process isolation.

[TBBle]

I was going from

Customers won’t need to worry whether their container image will run on a newer build of Windows Server 2022 or Windows 11 host.

I think I had parsed "newer build of Windows Server 2022" as "newer build of Windows Server". There won't be a newer build of Windows Server 2022, unless there are other significant release cadence changes coming, it'll be 20348 forever, while Windows 11 I expect will see newer builds after 22000 or whatever ships in October, like Windows 10 saw a new build every six months.

So yeah, I might have read that overeagerly... We never had the worry about LTSC 2019 container images running on newer builds of LTSC 2019, that was a supported use case (but not for LTSC 2016). So I guess I assumed that as a new promise, we'd be getting more than we already had.

Also, it seems a little arbitrary if you can run 20348-based containers on 22000 and 25456 (Windows 11 25H1), but not 25659 (LTSC 2025) (Invented build numbers, obviously). And the spirit of the whole paragraph seemed to me to be that the intention is to support down-version container images between Server releases, and we got Windows 11 hosting as a side-effect or preview of that, so that it's rock-solid once it becomes a 10-year-supported feature in LTSC 2025.

But that was really why I was checking my understanding, because I might have been reading too much into it.

I consider the Feb 2020 issue to be different, because that wasn't (AFAIK) a deliberate ABI breakage, but it was noticed after the fact that the security patch update had broken the ABI.

I definitely count "Can run older kernels in process isolation" as an improvement, that's a pretty big shift from where we were since LTSC 2019. I don't think it's fair to say it's not an improvement, because some other thing regressed.

[slonopotamus]

it seems a little arbitrary if you can run 20348-based containers on 22000 and 25456 (Windows 11 25H1), but not 25659 (LTSC 2025) (Invented build numbers, obviously).

It also seems arbitrary that you can run 20348 on 21H1 but can't on 20H2. What's so different between them that makes 21H1 capable, but 20H2 uncapable? Anyway, I believe that if MS wanted to say "good news folks, now we have backward kernel compatibility and you'll be able to run your ltsc2022 containers on Windows Server 2022 and later + Windows 11 and later" they could say that in a much clearer way.

I consider the Feb 2020 issue to be different, because that wasn't (AFAIK) a deliberate ABI breakage, but it was noticed after the fact that the security patch update had broken the ABI.

I mean, now MS would ensure that kernel<->userspace ABI remains stable across patches. Before this promise, they were not expected to care about it.

[adamrehn]

Although the inability to use process isolation mode under Windows 10 version 21H1 has been frustrating, I'm far more interested in exploring the compatibility story going forward. (The temporary forwards-compatibility for Hyper-V isolation mode under Windows 10 version 21H1 is also really neither here nor there, other than to serve as a reminder that Microsoft is perfectly capable of enabling this scenario if they so choose.)

Much like @TBBle, I interpreted the blog post as stating that backwards-compatibility for process isolation mode would span across LTSC releases, and I'm very interested in hearing whether this is indeed the case.

[TBBle]

It also seems arbitrary that you can run 20348 on 21H1 but can't on 20H2. What's so different between them that makes 21H1 capable, but 20H2 uncapable?

Could be that 21H1 includes a later build of the HCS system as part of the feature update, but I'd be surprised if that was the case. It's possible, this branch of Windows (Iron) was in active development before Windows 10 21H1 shipped, even though 21H1's kernel is based on an older windows branch. And we know MS run multi-year roadmaps, so it's not unbelievable that this blog post's content has been planned since late 2020.

That said, I would guess that it is arbitrary, in order to limit the affected scope, since it's only a temporary measure. Coming out-of-the-blue like this (no mention during the Server 2020 preview cycle that I'm aware of), makes it feel arbitrary, though.

That said, a colleague of mine said last week that he'd thought Windows Server 2022 was going to be able to run Server 2019 images in process isolation. He later said he'd misunderstood something, but perhaps I wrongly assumed what he had misunderstood, and he had heard about this improvement somewhere else.

So yeah, lots of speculation. This is probably the wrong place for that...

I mean, now MS would ensure that kernel<->userspace ABI remains stable across patches. Before this promise, they were not expected to care about it.

They had a stable ABI across patches as a supported use-case in LTSC 2019 as well, at least for older-container-image and newer-host. And that's still true here, since the feature they've added is "Your existing container images will work on future host builds", so newer-container on older-host isn't any-more supported than it was.

[mtola]

I confirm that all of this is a very severe regression and a very surprising change (?) in policy on Microsoft's part.

From the start we have docker images for each version of Windows to make things run properly in process isolation mode (as hyper-v mode isn't always suitable).

We thus have:

• 1809/ltsc2019
• 1903
• 1909
• 2004
• 20H2

We have now:

• ltsc2022

In the future we will therefore also need:

• 21H1 (we have been waiting for this since May...)
• 21H2 (if this version ever exists ?)
• and of course all new future versions of Windows 10/11

This is very important especially for an operating system which requires mandatory updates that we can only pause them for 4/5 weeks...

Finally I will end with the fact that Microsoft knows how to generate Docker images for each version of Windows, so why since May we still do not have version 21H1 of the Docker Windows Server Core image ?

[isanych]

@mtola, SAC server is dead, last version was 20H2, if you want process isolation in windows 10, 20H2 is latest version you can use. If you want something fresher you'll have to use Server 2022 or Windows 11 - and they will use ltsc2022 containers for many years.

[vrapolinario]

Coming to the conversation late (morning here on Pacific time here).

We will publish a blog post with more details on the down-level support with process isolation soon.
In the meantime, let me clarify a few things:

• There won't another SAC release for Server. The last one was 20H2. The latest release is WS2022 which is LTSC.
• Windows Client (10 and 11) follows a different approach than Server. Windows 10 and 11 will continue to get updates, always on the same build number. So Windows 10 won't be able to run WS2022 images. We have created images temporarily for WS2022 to run on Win10, but that's with hyper-v isolation.
• ltsc2022 images will be supported on the next releases of Windows 11 and Windows Server. That means the next LTSC release of Windows Server (I won't give it a name/number, but it was referenced above as 2025) will run ltsc2022 images with process isolation. We can't guarantee at this point the the release after that will run ltsc2022.
• Regardless of the above. If you stay on Windows Server 2022 or move one container host version up, we will support the ltsc2022 image for 5+5 years.
• 5+5 years means: 5 mainstream years with security + feature update. 5 years of extended support with security updates.

Hope that clarifies the question a bit. But please let us know if you still have questions.

[roozbehid-ic]

Just discovered this. So there is no plan to fix Windows 10 20H2 to be able to run inprocess dockers?
Does this mean IT departments should not allow update to latest Windows version so developers can still use dockers?

[isanych]

20H2 is fine, 21H1 is in trouble, but going forward looks like Windows 11 would be the version you want to use with docker.

[roozbehid-ic]

Yeah my mistake, I meant 21H1.
And it seems also Windows 10 21H2 does not solve this problem too. The problem is (or maybe Microsoft intentionally wanted this) many enterprise laptops already in use by developers are not Windows 11 compatible. So hardware changes are required. Technically it means either new hardware or don't use latest Windows 10. Kind of seems odd.

[QuAzI]

Windows 10 21H2 also is in trouble. LTSC release without any chance to have normal containers perfomance

[mtola]

ltsc2022 image works perfectly in "process mode" with win11-21h2, and if I understood things correctly it will also work in the future with win11-22h1, win11-22h2, win11-23h1, etc...

[QuAzI]

For some people it is impossible to change OS each time when somebody forget about actual LTSC release.

[mtola]

For some people it is impossible to change OS each time when somebody forget about actual LTSC release.

I totally understand this and I agree 100%...

[arionl]

FWIW, today was the day I figured I'd try Windows Containers in isolation mode and was completely blocked by this issue. I'm running the current stable and supported release of Windows 10 (21H2). Really bummed that I have no option besides spinning up a new VM somewhere or upgrading my daily driver/development system to Windows 11..

[slonopotamus]

upgrading my daily driver/development system to Windows 11

That doesn't work either, see #197

[robindegen]

Curious why folks need to run process isolated containers on Windows 10. Process isolated containers are officially only supported on Windows Server.

For local development. The windows 10 machine is already in a VM. I can't have another HyperV on top of that. So I use native containers. I have been running my local builds for quite a while this way. I can't upgrade to windows 11 because it simply reports my system isn't compatible.

It's a little strange to me that microsoft implements something like this and then just drops support for it. It's all part of the plan to force people to upgrade to window 11 I guess... but its not very developer friendly in the slightest.