Incident: NuGet Restore Issues on Debian Family Linux Distros

[JonDouglas]

On the morning of Wednesday, January 27, at 12:00 PM UTC, an incident occurred that caused NuGet package restore operations to fail in some Debian Family Linux environments. We were alerted to restores failing a few hours later by users on GitHub and put together an incident response team. Over the next few days, we shared mitigation options for users, published patched .NET Docker images, and reached out to Debian maintainers to resolve the certificate issues for the majority of users. The following post covers the root cause, guidance for resolving failing package restores, and status updates per distro.

Incident Summary

Users started receiving error messages when running dotnet restore on some Debian Family Linux distros. The immediate cause was a Microsoft-owned certificate expiring. The certificate was one that Microsoft had previously used (until ~ November 1, 2020) for author-signing NuGet packages. This certificate expiration event was announced and planned for, and the intended uneventful rollover to validate packages using the timestamp went as expected for most users.

Approximately two years prior, browser and operating system vendors reached a consensus proposal that began a gradual process to distrust Symantec certificates. The intended outcome was that these certificates would no longer be trusted for TLS (used for web browsing and other network connections). The certificates were not slated for removal for code integrity purposes. These Symantec certificates are part of the trust chain for the timestamp on the affected NuGet packages.

Debian and Ubuntu maintainers published an update (for at least some distro versions) that distrusted the Symantec certificate entirely. This change was later rolled back, but patched packages were not available on the package repositories prior to the Microsoft NuGet author signing certificate expiring. This situation resulted in a perfect storm that led to this incident. If the certificate had expired a week or two later, then this incident likely would not have happened.

Prior to the Microsoft NuGet author signing certificate expiring, the timestamp was not used to validate the package signature. Once the signing certificate expired, NuGet relies on the timestamp to validate packages; however, NuGet rejected the timestamping certificate as the Symantec certificate was no longer trusted.

Affected Environments

.NET 5.0 SDK on the following Debian Family Linux distros -

Debian

• Debian 11 ("bullseye"): Not Impacted ✔
• Debian 10 ("buster"): Resolved ✔
• Debian 9 ("stretch"): Resolved ✔

🎉As of February 6th, 2021 Debian 10.8 was released including the fix for ca-certificates🎉

Debian .NET Docker Images

• Debian 10 Images: Resolved ✔
  • January 27: Patched images released with a *-ca-patch-buster-slim suffix to tags
  • January 30: Updated images released for the standard release tags ✔

Ubuntu

• Ubuntu 20.10 (Groovy Gorilla): Resolved ✔
• Ubuntu 20.04 (Focal Fossa): Not Impacted ✔
• Ubuntu 18.04 (Bionic Beaver): Not Impacted ✔

Other Linux Distros

• We do not currently have any reports of non-Debian Family Linux distros being impacted by this issue.

Azure App Service

• App Service and App Service's Built-In Build Service known as Oryx has issued a response.

Response

Due to the nature of the underlying cause, steps to remedy the issue require the highest degree of caution. The CA certificate store plays a central role in security, and changes that introduce a certificate that is currently distrusted by the OS, that ignore certificate checking, or that can corrupt the certificate store can introduce security vulnerabilities to the machine.

After determining the root cause of the issue, we turned to developing mitigation approaches that we could offer users experiencing this NuGet restore error. Given the nature of the issue, we applied Microsoft security standards to any potential solution with the understanding that we would only offer approaches that conformed to those standards.

We decided to start with mitigating this problem in the Docker container images that we publish for Debian. These images are a good starting point for three reasons: they have a lot of usage, .NET usage in Debian is primarily via containers, and containers are disposable (have a more well-defined lifespan).

We created and published a Debian 10-based .NET image that imported the certificate store from Debian 11, which had a version of the package with the desired certificate behavior. However, Debian 11, and all of the packages in its feed, are currently still in testing and not recommended for use in production environments.

We were presented with a (somewhat) tough choice. We could push the patched images using our convential tags, like 5.0, and then users would naturally pull those images without any changes on their side. In addition, users would pull these images, and get back to a working system, without the need to read our advisory posts. That option was very attractive, and would normally have been the one we chose. Unfortuntately, we couldn't in good conscience pull in a test version of ca-certificates into a production image that users would pull without any knowledge that we had done so, or of its implications. This choice also wouldn't have aligned with Microsoft security practices. Instead, we published the images with an unconventional tagging scheme that users would have to manually opt into, after (hopefully) weighing their own choices and concerns.

A new tag was chosen: 5.0.102-ca-patch-buster-slim (and architecture-specific tag variants). It was intended to be descriptive and unfamiliar, to make it obvious that it was different and should not be used for long.

Next, we had to decide what to do for other uses of Debian, such as in virtual machines and on raw-metal. We decided that there was no specific guidance we could offer. We felt good about what we'd done for container images, but that's because they are disposable and immutable. That means the scope of modifying the certificate store in containers is more limited. We could not reason about modifing the certificate store in persistent installations of Debian. As a result, we offered no guidance there. We were conscious that any changes we recommended had to provide a workable fix without introducing long-term security risks for our users. For example, installing the future version from Debian 11 would prevent VM's from receiving future security updates. We noticed that some people realized that the pattern implemented in our Dockerfiles was applicable to other environments and did so. We couldn't recommend that, but it was workable if you understood the implications and knew what you were doing. If you did make these changes, our instructions for resetting your version of ca-certificates can be found later in this post.

Thanks to the efforts of the Debian maintainer for the ca-certificates package, the patched version of the affected package was released to production while we were working on these mitigations. It takes several days for a new package update to reach the official package repositories, which makes it available to end users. In the meantime, we provided a method for users to validate and install this pre-release update from the Debian pool. This updated Debian 10 enabled us to offer a Debian 10 rather than Debian 11-based solution. While the package still wasn't yet an official production package, and required an unfamiliar series of commands to use, it had much better characteristics than the pattern used in our Dockerfiles.

Waking up this morning -- on January 30th -- we found that the updated ca-certificates package was officially available for Debian 10. We tested it and it resolved all issues. We published new container images, using our conventional 5.0 and other floating tags. If you changed your Dockerfiles to use the temporary tags, you can revert that change, and go back to the previous tags you were using. You do not have to proactively re-build and re-publish your applications with the new images if you do not want to. If you are currently on the *-ca-patch-buster-slim tag, you will need to revert to the standard tags to receive updates after the next SDK release. If you haven't made any changes to your Dockerfiles, you will get updated/resolved images when you next pull.

Other Debian users (not containers) are recommended to update the ca-certificates package. More detailed instructions are provided later in the post.

Users of Ubuntu 20.10 (Groovy Gorilla) have experienced the same issues found on Debian 10, and our team is reaching out to the Ubuntu security team to bring the Debian patch to Ubuntu. We are also determining if Debian 9 has these same challenges.

Mitigation

Debian 10 ("buster")

As of January 30, the patched version of ca-certificates is available from Debian's package repository.

If you did not modify your ca-certificates package in response to the Symantec certificate removal, you can perform a standard upgrade of your packages:

apt update && apt upgrade

Patched Using Pre-Release ca-certificates Version 20200601~deb10u2 (Guidance from January 29)

For users who followed our guidance issued on January 29 to install the pre-release package, version "20200601~deb10u2," you do not need to take further action. Your system should receive future updates to the ca-certificates package from Debian. You can update your system if you'd like, but you should already have the latest ca-certificates package based on your previous manual installation of that package.

Modified CA Certificate Store

If you took actions to modify your CA certificate store or your ca-certificates package version, the changes can be reverted with the following steps. This is necessary for your machine to receive future updates.

apt-get remove ca-certificates
apt-get purge ca-certificates
apt-get update
apt-get install -f ca-certificates=20200601~deb10u2

Debian 9

We have reached out to the Debian ca-certificates maintainers and have asked their team to consider applying the same mitigation steps followed by their Debian 10 ("buster") package.

✔ This has since been resolved.

Ubuntu 20.10 (Groovy Gorilla)

We have reached out to the Ubuntu Security maintainers and asked their team to consider applying the same mitigation steps followed by Debian to restore the ca-certificates package.

✔ This has since been resolved.

Stay Updated

• We are continuing to investigate this issue and will let you know more as we find out.
• We'll continue to update the discussion at Incident: NuGet Restore Issues on Debian Family Linux Distros Home#10504.
• Original GitHub Issue for discussion NuGet restore failing on .NET 5 for Debian-based OSes (CA trust changed) Home#10491
• We will also continue posting updates on NuGet Status and on Twitter @NuGet.

[rconard]

For .NET Docker users, we have new guidance.

Current Guidance

As of January 30, an updated stable version of the ca-certificates package is available and included in the .NET Docker Debian 10 (Buster) sdk images using the standard tags. Use of the ca-patch tags described in the previous guidance is no longer necessary to resolve the NuGet restore issue. If you made use of the ca-patch tags, you can switch back to one of the below tags you had previously used. Those ca-patch tags will continue to be maintained until the 5.0.103 SDK ships, in accordance with the support policy for .NET Docker images.

Recommended Tags for Debian 10 (Buster) sdk

Tags	Dockerfile	OS Version
5.0.102-buster-slim-amd64, 5.0-buster-slim-amd64, 5.0.102-buster-slim, 5.0-buster-slim, 5.0.102, 5.0	Dockerfile	Debian 10
5.0.102-buster-slim-arm64v8, 5.0-buster-slim-arm64v8, 5.0.102-buster-slim, 5.0-buster-slim, 5.0.102, 5.0	Dockerfile	Debian 10
5.0.102-buster-slim-arm32v7, 5.0-buster-slim-arm32v7, 5.0.102-buster-slim, 5.0-buster-slim, 5.0.102, 5.0	Dockerfile	Debian 10

Previous Guidance

We had previously published new images that mitigate these issues by using a pre-release version of the Debian ca-certificates package.

These new tags were created for dotnet/sdk to use as a workaround:

Tags	Dockerfile	OS Version
5.0.102-ca-patch-buster-slim-amd64, 5.0.102-ca-patch-buster-slim	Dockerfile	Debian 10
5.0.102-ca-patch-buster-slim-arm64v8, 5.0.102-ca-patch-buster-slim	Dockerfile	Debian 10
5.0.102-ca-patch-buster-slim-arm32v7, 5.0.102-ca-patch-buster-slim	Dockerfile	Debian 10

[rconard]

Note: We have posted mitigation strategies for Debian 10 Buster in the original post above.

[rconard]

Following the release of an updated ca-certificates package for Debian 10 ("buster"), we have posted additional instructions on how to install the updated package. We have also provided additional information on the root cause of the issue.

Please see the post above.

We have also added a new NuGet Home Issue for discussion.