Many jobs are failing with a 401 response

[tannergooding]

The CI failures look like:

.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.22255.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) A call to an Azure DevOps api returned 401, which may indicate a bad 'System.AccessToken' value.

Please Check the 'Make secrets available to builds of forks' in the pipeline pull request validation trigger settings.
We have evaluated the security considerations of this setting and have determined that it is fine to use for our public PR validation builds.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[tannergooding]

CC. @dotnet/runtime-infrastructure and @dotnet/dnceng

This may be related to #69829, so CC. @agocke

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

The CI failures look like:

.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.22255.2/tools/azure-pipelines/AzurePipelines.MultiQueue.targets(16,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) A call to an Azure DevOps api returned 401, which may indicate a bad 'System.AccessToken' value.

Please Check the 'Make secrets available to builds of forks' in the pipeline pull request validation trigger settings.
We have evaluated the security considerations of this setting and have determined that it is fine to use for our public PR validation builds.

Author:	tannergooding
Assignees:	-
Labels:	area-Infrastructure, untriaged
Milestone:	-

[premun]

Hi @tannergooding,

This seems to be a fallout of a security setting we were asked to change yesterday for our pipelines ("Make secrets available to builds of forks"). I have already reverted this setting in many pipelines but not sure if all.

We will have a better remediation for this problem soon.

[tannergooding]

Thanks! There is a decent list of failures from 6 hours ago here: https://github.com/dotnet/runtime/pull/69756/checks?check_run_id=6606234716

[tannergooding]

I'm seeing new failures now:

Failed to retrieve information about 'Microsoft.NETCore.App.Ref' from remote source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/d1622942-d16f-48e5-bc83-96f4539e7601/nuget/v3/flat2/microsoft.netcore.app.ref/index.json'.

Not sure if these are related or deserve their own issue.

[MattGal]

See this FR discussion - AzDO is/was having trouble with public feeds; we're on it.

[BruceForstall]

I believe this has been fixed now (some AzDO feature was reverted)?

[tannergooding]

I believe so as well. My PRs are at least passing now.

[BruceForstall]

@MattGal Is this fixed? we're still seeing it, e.g.

https://dev.azure.com/dnceng/public/_build/results?buildId=1799534&view=logs&j=9d34e523-f5d6-52dc-46f2-0a66cb13e494&t=5aa7bffb-7d91-5080-bb39-aa229a02b9d3

cc @amanasifkhalid

[riarenas]

Fixing this requires an active change to pipelines. Please re-enable the pipeline to allow usage of secrets from builds from forks in any pipelines where you still see this issue.

[MattGal]

I'm seeing new failures now:

Failed to retrieve information about 'Microsoft.NETCore.App.Ref' from remote source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/d1622942-d16f-48e5-bc83-96f4539e7601/nuget/v3/flat2/microsoft.netcore.app.ref/index.json'.

Not sure if these are related or deserve their own issue.

@BruceForstall apologies for mixing the two things, I was responding to this one which was a feature rollout by AzDO. If you're seeing the top comment in this issue still, do exactly what @riarenas says.

[BruceForstall]

@riarenas Thanks; I updated the pipeline configuration, and things seem to be working.

@MattGal Yeah, I had assumed from discussion here that some AzDO configuration had been rolled back and things would "just work" afterwards.

[ViktorHofer]

I assume this can be closed again now? Feel free to reopen if you still see the issue somewhere.