.NET Core Linux Feeds -- Now available

[richlander]

.NET Core Linux Feeds

Update @ 11:30AM PST: Feeds are now available. The remainder of the text is unchanged.

The .NET Core teams provides a set of Linux feeds for a variety of distros. We support the following package managers: APT, Yum, DNF, and ZYpp. We rely on a Microsoft-only service provided by the Azure organization for the actual underlying feeds infrastructure.

We are seeing widespread reports of errors (with strange messages) when people are pulling .NET Core packages via APT. Both Ubuntu, Debian use APT. We also discovered the problem ourselves. We are not seeing any issues with the other package managers listed above.

We are working closely with the team that maintains the underlying infrastructure. They are working on it with the highest priority. We have a "severity 1" ticket with them.

We will use this issue for tracking and reporting until the issue is resolved, and updated when we have new information.

We are very sorry that this is happening. After resolution, we will look for new safeguards to prevent this situation in future.

Reports we are tracking:

• Debian 9 Hash Sum Mismatch #3994
• Can't install .NET Core on Ubuntu, Debian from the Linux package repository: "File has unexpected size (23655 != 156065)" #3988
• Unable to locate package dotnet-sdk-3.1 on Ubunto 18.04.3 #3993
• Failed to fetch due to Hash sum mismatch microsoft/vscode#86858
• https://github.saobby.my.eu.orgmunity/t5/GitHub-Actions/ubuntu-latest-Apt-repository-list-issues/m-p/41191#M4527

[thorsig]

My 10 cents: The Visual Studio Code team did overwrite the InRelease file (and more?) for every repository.

[dagood]

I just got an update from the Linux repo admins: they've reverted the recent changes to the underlying service that caused this problem in the first place, and they're now working on fixing up the individual feeds so that they can be used again.

[vazgen6]

Please speed up the fix, been waiting for it for about 16 hours now, I built a new pc and can't install vscode nor .net.
A company like microsoft should be more responsible and test before publishing it to the public.
With all my respect, this is not professional.

[dagood]

The feeds are starting to work again. I'm getting successful apt-get update for Ubuntu 19.04 and Debian 9, and I'm in contact with one of the devs who is working on Debian 10 now, and they'll then refresh all the feeds to make sure none are missed.

[thorsig]

Successful for Ubuntu-18.04 LTS.

[dagood]

Debian 10 is now working for me (and others in #3988)! I'm trying out more feeds now to try to find any gaps.

[dagood]

The problem has still been happening with the VS Code feed. They're working on fixing that one as well. It just now started working for me, but it may take up to 15 minutes from now for the fix to sync across the world.

[dagood]

It looks like we're finally in the clear!

I haven't seen any new reports after the repo admins confirmed the feeds were fixed, and it's been 15 minutes since VS Code feed was fixed, so we expect that fix to be available to everyone now. There are some reports of success on that thread as well (microsoft/vscode#86858).

[richlander]

Please share any new reports of issues that you see. We want to ensure that we really are "in the clear".

[vazgen6]

Works on Mint (based on Ubuntu Bionic)

[dleeapho]

I'm in touch with the Microsoft repo admins and will work with them on a post-mortem of this issue. I'll work with the repo admins on both responsiveness and prevention.

[richlander]

Closing this issue now. The incident appears to have passed.

We're sorry this happened, and we are investigating the cause. We don't want this to happen again.

[harshavardhana]

Seen again today

17
Fetched 4003 B in 6s (603 B/s)
4918
Reading package lists...
4919
[91mE: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/dists/xenial/main/binary-amd64/Packages  Writing more data than expected (622228 > 606754)
4920
E: Some index files failed to download. They have been ignored, or old ones used instead.
4921
[0mThe command '/bin/sh -c apt-get --yes update && apt-get --yes upgrade &&     apt-get --yes --quiet install wget jq curl git dnsmasq &&     cd /mint && /mint/release.sh' returned a non-zero code: 100
4922
error running ci task for minio/minio:d5ce8f7f19592f16c405fe673e4bd658f6171ed3: exit status 100

[randallswf]

Both the 18.04 and 19.04 repos are down.
Failed to fetch https://packages.microsoft.com/ubuntu/18.04/prod/dists/bionic/main/binary-amd64/Packages.bz2 File has unexpected size (88725 != 85265). Mirror sync in progress? [IP: 13.91.48.226 443]

[baronfel]

Can agree with both. repro is a dockerfile:

FROM mcr.microsoft.com/dotnet/core/sdk:3.1

RUN wget -qO- https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg \
    && mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/ \
    && wget -q https://packages.microsoft.com/config/debian/10/prod.list \
    && mv prod.list /etc/apt/sources.list.d/microsoft-prod.list \
    && chown root:root /etc/apt/trusted.gpg.d/microsoft.asc.gpg \
    && chown root:root /etc/apt/sources.list.d/microsoft-prod.list \
    && apt-get update \
    && apt-get install aspnetcore-runtime-2.2

[dagood]

We have a few new issues open to track the recent problems:
#4121 "yum install dotnet-sdk-3.0 fails on centos7"
#4122 "Linux package repositories are temporarily down (Debian and RPM packages)"

Still in initial stages of investigation.

[cyberatz]

https://packages.microsoft.com/debian/10/prod/dists/buster/main/binary-amd64/Packages.bz2 File has unexpected size

Error is occurring again.

[dagood]

@cyberatz The new instance of this problem is tracked at #4864.

[mikevo13]

Just encountered the same issue today 7/23/2020. Ticket opened to Microsoft support.

[KarFan-FS]

Just got this error:
"Failed to fetch https://packages.microsoft.com/debian/9/prod/pool/main/u/unixodbc/unixodbc-dev_2.3.7_amd64.deb 404 Not Found [IP: 13.79.173.94 443]"

Steps in my dockerfile:
RUN apt-get update &&\ apt-get -yq update
RUN apt-get install -y python3-dev && \ apt-get install -y g++ && \ apt-get install -y unixodbc-dev

[dagood]

When you see this happening, please open a new issue (feel free to include a link back to this issue as a reference) or look for an issue currently tracking it--commenting on a closed issue isn't likely to ping the right person.

The current problems are tracked by #6381.