[Bug]: Error: NU1403 Package content hash validation failed for NETStandard.Library.2.0.3. The package is different than the last restore.

[Kurt-von-Laven]

NuGet Product Used

MSBuild.exe

Product Version

MSBuild 17.1.0+ae57d105c, Visual Studio 17.1.0

Worked before?

MSBuild 17.0.6, Visual Studio 17.0.6

Impact

I'm unable to use this version

Repro Steps & Context

To clarify, the impact of this issue is that I can not build our UWP app on any version of any tool without removing the lock file, which is prohibitively risky since it may be catching and preventing a live attack on the Microsoft ecosystem.

Repro steps:

1. Use Visual Studio 17.1.0 to create a default UWP project with a lock file expecting the following hash for NETStandard.Library.2.0.3: "st47PosZSHrjECdjeIzZQbzivYBJFv6P2nv4cj2ypdI204DO+vZ7l5raGMiX4eXMJ53RfOIg+/s4DHVZ54Nu2A==". This can be achieved by introducing a direct dependency on Microsoft.NETCore.UniversalWindowsPlatform.6.2.13 and adding the following to your .csproj:

<PropertyGroup>
  <RestoreProjectStyle>PackageReference</RestoreProjectStyle>
  <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>
</PropertyGroup>

You will probably then need to modify the hash in the lock file (packages.lock.json) to the value provided above.

2. Build the UWP project in Visual Studio 17.1.0.

Expected result:

The project builds without errors.

Actual result:

The project builds successfully, but reports error NU1403 because of a lock file hash mismatch in NETStandard.Library.2.0.3. The error was first observed on February 22nd, 2022 at 18:33 UTC. The same project built successfully without reporting any errors prior to this, and the error has never been encountered before. Nothing was changed between the last successful build and this one, and clearing the NuGet cache had no effect. The error can be reproduced locally on MSBuild 17.1.0 and in CI on MSBuild 17.0.6, so it seems very unlikely that the issue is related to the release of MSBuild 17.1.0.

Verbose Logs

Here is the relevant portion of the diagnostic output of MSBuild 17.1.0+ae57d105c:


...
All packages and projects are compatible with UAP,Version=v10.0.17763 (win10-x86-aot). (TaskId:30)
Package content hash validation failed for NETStandard.Library.2.0.3. Expected: st47PosZSHrjECdjeIzZQbzivYBJFv6P2nv4cj2ypdI204DO+vZ7l5raGMiX4eXMJ53RfOIg+/s4DHVZ54Nu2A== Actual: 548M6mnBSJWxsIlkQHfbzoYxpiYFXZZSL00p4GHYv8PkiqFBnnT68mW5mGEsA/ch9fDO9GkPgkFQpWiXZN7mAQ== (TaskId:30)
14:12:32.118
...

The full diagnostic log unfortunately reveals too much proprietary information to disclose publicly (including the name of every single file in the repository, for example), but I suspect the remainder of the contents will not be of much assistance in this particular case.

[nkolev92]

Hey @Kurt-von-Laven,

As mentioned in the other issue, the problem here is that there's probably a package that was somehow preinstalled on your machine that doesn't match the same package that would've been installed if say you got the package from nuget.org.

So there's actually a discrepancy, and our first step is to understand how the discrepancy happened (usually by some preinstalled packages), and then once we know how it happened, we can proceed fixing or correcting whatever it may be that caused this difference.

Can you please find the obj folder of the project you're having issues with and open the assets file.

In there, you'll find a prop called packageFolders.

On my machine it can be something like:

  "packageFolders": {
    "C:\\Users\\Roki2\\.nuget\\packages\\": {},
    "C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\NuGetPackages": {},
    "C:\\Program Files (x86)\\Microsoft\\Xamarin\\NuGet\\": {},
    "C:\\Program Files (x86)\\Microsoft SDKs\\UWPNuGetPackages\\": {}
  },

Then please go into each one of those folders and find the netstandard.library equivalent folder, and please upload the:
.nupkg.metadata and netstandard.library.2.0.3.nupkg.sha512 files from the folders where it exists.

Please note that the ordering in the packageFolders object matters, so please make sure you preserve that when you paste that information.

Thanks!

[Kurt-von-Laven]

Thank you, @nkolev92! This has been super illuminating, and the results are as I imagine you expected. Here is the relevant excerpt from project.assets.json:

"packageFolders": {
  "%USERPROFILE%\\.nuget\\packages\\": {},
  "C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\NuGetPackages": {},
  "C:\\Program Files (x86)\\Microsoft SDKs\\UWPNuGetPackages\\": {}
},

The only copy of these files was in C:\\Program Files (x86)\Microsoft SDKs\UWPNuGetPackages\. The pertinent .nupkg.metadata contains:

{
  "version": 1,
  "contentHash": "548M6mnBSJWxsIlkQHfbzoYxpiYFXZZSL00p4GHYv8PkiqFBnnT68mW5mGEsA/ch9fDO9GkPgkFQpWiXZN7mAQ=="
}

The contents of netstandard.library.2.0.3.nupkg.sha512 are: 548M6mnBSJWxsIlkQHfbzoYxpiYFXZZSL00p4GHYv8PkiqFBnnT68mW5mGEsA/ch9fDO9GkPgkFQpWiXZN7mAQ==

I created a nuget.config, added <DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder> to our .csproj, and ran dotnet nuget locals all --clear && git clean -xfd as described by #7921 (comment), but that doesn't appear to have been sufficient to trigger the UWP libraries to be downloaded from nuget.org as I don't see any new packages in %USERPROFILE%\.nuget\packages\. The original error remains. If I delete the lock file, then of course the error disappears, and the lock file is recreated expecting 548M6mnBSJWxsIlkQHfbzoYxpiYFXZZSL00p4GHYv8PkiqFBnnT68mW5mGEsA/ch9fDO9GkPgkFQpWiXZN7mAQ==, but I still don't see any new packages in %USERPROFILE%\.nuget\packages\, so it's still using the fallback folder. What am I overlooking?

[nkolev92]

Turns out the UWPNuGetPackages source is getting added in a way that it's not easy to disable right now.

It's being set through RestoreAdditionalProjectFallbackFolders, and it's being set in

C:\Program Files\Microsoft Visual Studio\2022\Preview\MSBuild\Microsoft\WindowsXaml\v17.0\8.21\Microsoft.Windows.UI.Xaml.CSharp.targets

or the equivalent on your machine, and it's bringing a version of the library that has incorrect metadata., which should in reality be: st47PosZSHrjECdjeIzZQbzivYBJFv6P2nv4cj2ypdI204DO+vZ7l5raGMiX4eXMJ53RfOIg+/s4DHVZ54Nu2A==

Unfortunately given that it's in the targets, it needs to be done after that.

Can you try creating a Directory.Build.targets with the following content?

<Project>
    <PropertyGroup>
        <RestoreAdditionalProjectFallbackFolders></RestoreAdditionalProjectFallbackFolders>
    </PropertyGroup>
</Project>

If that works, I'll try to see if the fallback folder can be fixed up in a VS update.

[Kurt-von-Laven]

What a great find! That forced NuGet to pull down all of the UWP libraries from nuget.org. As desired, within %USERPROFILE%\.nuget\packages, .nupkg.metadata contains:

{
  "version": 2,
  "contentHash": "st47PosZSHrjECdjeIzZQbzivYBJFv6P2nv4cj2ypdI204DO+vZ7l5raGMiX4eXMJ53RfOIg+/s4DHVZ54Nu2A==",
  "source": "https://api.nuget.org/v3/index.json"
}

However, netstandard.library.2.0.3.nupkg.sha512 from nuget.org contains: 548M6mnBSJWxsIlkQHfbzoYxpiYFXZZSL00p4GHYv8PkiqFBnnT68mW5mGEsA/ch9fDO9GkPgkFQpWiXZN7mAQ==!

Evidently NuGet looks at the former, because it no longer complains about NETStandard.Library.2.0.3. However, it now complains about Microsoft.NETCore.Platforms.2.1.0! NuGet expected GmkKfoyerqmsHMn7OZj0AKpcBabD+GaafqphvX2Mw406IwiJRy1pKcKqdCfKJfYmkRyJ6+e+RaUylgdJoDa1jQ==, but got ok+RPAtESz/9MUXeIEz6Lv5XAGQsaNmEYXMsgVALj4D7kqC8gveKWXWXbufLySR2fWrwZf8smyN5RmHu0e4BHA==.

Predictably, the pertinent .nupkg.metadata contains:

{
  "version": 2,
  "contentHash": "ok+RPAtESz/9MUXeIEz6Lv5XAGQsaNmEYXMsgVALj4D7kqC8gveKWXWXbufLySR2fWrwZf8smyN5RmHu0e4BHA==",
  "source": "https://api.nuget.org/v3/index.json"
}

However, microsoft.netcore.platforms.2.1.0.nupkg.sha512 contains GmkKfoyerqmsHMn7OZj0AKpcBabD+GaafqphvX2Mw406IwiJRy1pKcKqdCfKJfYmkRyJ6+e+RaUylgdJoDa1jQ==!

[nkolev92]

What does the .nupkg.metadata contain?
That's what used for lock file scenario.
It basically accounts for the difference between signed and unsigned packages.

[Kurt-von-Laven]

Good to know. The exact contents of the .nupkg.metadata for each of the pertinent packages are listed above. To recap, after downloading all of the UWP dependencies from nuget.org NETStandard.Library.2.0.3's is right, but Microsoft.NETCore.Platforms.2.1.0's is wrong (at least relative to the value it was at before whatever changed). Is it possible that an SDK update unintentionally modified the hash of NETStandard.Library.2.0.3 recently, and the hash of Microsoft.NETCore.Platforms.2.1.0 in the SDK fallback folders has been wrong for years (in other words, the issue predates the creation of our lock file)?

[nkolev92]

@Kurt-von-Laven

You need to run restore with /p:RestoreForceEvaluate="true" to update the lock file.

Can you try that?

[Kurt-von-Laven]

Updating the lock file of course succeeds by modifying the expected hash of Microsoft.NETCore.Platforms.2.1.0 to ok+RPAtESz/9MUXeIEz6Lv5XAGQsaNmEYXMsgVALj4D7kqC8gveKWXWXbufLySR2fWrwZf8smyN5RmHu0e4BHA==. The question is why, and whether ok+RPAtESz/9MUXeIEz6Lv5XAGQsaNmEYXMsgVALj4D7kqC8gveKWXWXbufLySR2fWrwZf8smyN5RmHu0e4BHA== or GmkKfoyerqmsHMn7OZj0AKpcBabD+GaafqphvX2Mw406IwiJRy1pKcKqdCfKJfYmkRyJ6+e+RaUylgdJoDa1jQ== is the correct hash? If the former, the fallback folder has been providing the wrong hash for years. If the latter, nuget.org (or at least what my computer thinks came from nuget.org) is providing the wrong hash currently, but I don't know how to distinguish these cases. Does that make sense?

[nkolev92]

It's the fallback folder that's been providing the hash incorrectly.

[Kurt-von-Laven]

Thank you for confirming! Should I open a separate issue to get the hashes of both of these packages corrected in the fallback folder? (If so, where?)

[nkolev92]

Can you please file an issue at developercommunity about fixing the UWPNuGetPackages fallback folder.

I'll work on finding the best owner for that.

[erdembayar]

@Kurt-von-Laven
May I close this issue as considering it's resolved? We don't own packages mentioned here.

[Kurt-von-Laven]

Apologies for the delay; just got back from a trip. Yes, I am closing this issue in favor of https://developercommunity.visualstudio.com/t/NETStandardLibrary203-Ships-With-Wro/1688568 and https://developercommunity.visualstudio.com/t/MicrosoftNETCorePlatforms210-Ships/1688567. Thank you for helping us get unblocked so quickly!

[alexswan10k]

Just wanted to make a note that I had a similar problem with FSharp.Core 6.0.5 today on the build server, with exactly this problem - - mismatching hashes. It looks like .NET SDK 6.0.400 was dropped yesterday, which is exactly when it started failing. Forcing the build server back to 6.0.303 fixed the issue, so presumably, the hashing algorithm is behaving differently.

[nwithan8]

Just wanted to make a note that I had a similar problem with FSharp.Core 6.0.5 today on the build server, with exactly this problem - - mismatching hashes. It looks like .NET SDK 6.0.400 was dropped yesterday, which is exactly when it started failing. Forcing the build server back to 6.0.303 fixed the issue, so presumably, the hashing algorithm is behaving differently.

Seconding this. Rolling back to FSharp.Core 6.0.3 worked for me.