Locked mode not working as hash is calculated incorrectly for previously cached packages

[alefranz]

Details about Problem

I've been struggling with the new "locked mode" of dotnet restore since release and I've finally been able to create repro steps.

The problems seems related to using different version of the .NET Core SDK on the same machine, in particular versions before the locked mode introduction alongside the new ones.
As the NuGet Package Cache is shared between all versions, if the package as been cached by an old version of dotnet.exe/nuget, it looks like the new .nupkg.metadata get created with an incorrect signature.

Example:
Newtonsoft.Json 12.0.1 has signature:

{
  "version": 1,
  "contentHash": "jmVyoEyk0In8r+AObYQyFKVFm7uSRzE0XSHSbEtBJcZDMV6DqJoyB4FLcHwprPVhAh826so0db3DIKXVnpGoPA=="
}

however if it was already in the Nuget Package Cache, when using the new dotnet.exe/nuget it get this signature

{
  "version": 1,
  "contentHash": "pBR3wCgYWZGiaZDYP+HHYnalVnPJlpP1q55qvVb+adrDHmFMDc1NAKio61xTwftK3Pw5h7TZJPJEEVMd6ty8rg=="
}

which obviously cause the locked mode feature to fail.

This behaviour seems consistent in different Windows versions and machines.
Not tested on Linux/MaxOs.

I believe it is a common scenario to use different versions of the SDK on a dev machine as well on a build agent (not everyone use disposable docker agents yet) e.g. via global.json in different projects

Happy to help investigate more but I will need someone to point me in the right direction.

Thank you

Product Versions

NuGet product used: dotnet.exe

Version pre locked mode used:
❯ dotnet --version
2.1.403
❯ dotnet nuget --version
NuGet Command Line
4.8.1.0

Version with locked mode used:
❯ dotnet --version
2.2.102
❯ dotnet nuget --version
NuGet Command Line
4.9.2.0

OS version: Win10 17763

Worked before? No

Detailed repro steps so we can see the same problem

1. git clone https://github.com/alefranz/DotnetRestoreLockedMode
2. Delete package from Nuget package cache in %userprofile%.nuget\packages\newtonsoft.json\12.0.1
3. cd DotnetRestoreLockedMode\NetSdk22
4. dotnet restore --force --force-evaluate
5. dotnet restore --force --locked-mode pass!
6. Delete package from Nuget package cache in %userprofile%.nuget\packages\newtonsoft.json\12.0.1
7. cd ..\NetSdk21
8. dotnet restore --force
9. cd ..\NetSdk22
10. dotnet restore --force --locked-mode fail! :(

Sample Project

https://github.com/alefranz/DotnetRestoreLockedMode

Note the global.json is at the project level for simplicity, but it doesn't make any difference having different solutions.

[rrelyea]

/cc: @anangaur

[rrelyea]

@jainaashish - I think we meant to put this in 4.9.3, not 4.9.x - when we discussed yesterday, right?

[jainaashish]

we discussed #7667 and #7646 to be in 4.9.3. We didn't discuss this one but from the description it seems the issue is packages cache having wrong package in there which gets wrong hash in lock file. And the current solution for this is to clean packages cache or that specific package so I'm afraid we can address this for 4.9.3 or even 4.9.x

[anangaur]

issue is packages cache having wrong package in there which gets wrong hash in lock file. A

@jainaashish I dont think the package is wrong. I see different SHA being generated when the package is downloaded from nuget.org vs. when the package is already there but .nupkg.metadata file is not there and is generated in-situ. For Newtonsoft.json 12.0.1, when downloaded from internet (nuget.org), the SHA is jmVyoEyk0In8r+AObYQyFKVFm7uSRzE0XSHSbEtBJcZDMV6DqJoyB4FLcHwprPVhAh826so0db3DIKXVnpGoPA==

but when generated, it is: pBR3wCgYWZGiaZDYP+HHYnalVnPJlpP1q55qvVb+adrDHmFMDc1NAKio61xTwftK3Pw5h7TZJPJEEVMd6ty8rg==

[jainaashish]

this is weird, I'll look into this. Does this also happen with other packages as well or other version of Newtonsoft.json?

[anangaur]

Yes. Same with all packages. When a package is freshly downloaded, the contentHash in .nuspec.metadata file matches with the hash stored in .nupkg.SHA512 file. Now if you delete the .nuspec.metadata file and restore again, the newly generated .nuspec.metadata file has a different hash than what was before. Seems wrong to me.

[jainaashish]

this is happening with dotnet restore... so assuming something is fishy with xplat path. Works fine inside VS.

[anangaur]

I guess we should test out all the paths including NuGet.exe as well. The priority of fixing this issue is high IMO.

[alefranz]

yes, confirming this happens with dotnet restore and for all packages.
@anangaur thanks for appreciating the importance of this issue.
@jainaashish if you can point me in the right direction I'll be happy to try to have a look over the weekend

[jainaashish]

Root cause of the issue is while writing the .nupkg.metadata file of a downloaded package, it mistakenly writes PackageHash instead of ContentHash but packages.lock.json file has ContentHash so it fails the SHA validation at next restore. So this isn't specific to xplat instead specific for VS 15.9.

This issue has already been fixed in dev16 preview bits so I'll cherry-pick the commit into 4.9.3 branch as well.

[jainaashish]

Cherry-picked to release-4.9.3-rtm branch with 2d7c84030935330c0e5f1889bed825ff1ec50cc2

[Flavien]

Where is that bug fixed? I'm using VS2019 Preview 2 (NuGet 5.0.0), and I'm still experiencing that problem.

[jainaashish]

I can't repro it with VS2019 Preview2. Can you share your complete repro steps? and also the dotnet version details (run dotnet --info)

[Flavien]

Here is what I am doing:

• I add <RestorePackagesWithLockFile>true</RestorePackagesWithLockFile> to my project to generate the lock file
• Then I delete my fallback Folders (C:\Users\User\.nuget\packages and C:\Program Files\dotnet\sdk\NuGetFallbackFolder)
• I rebuild the project to regenerate the lock file
• All the SHA512 values are different in the new lock file.

dotnet --info:

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview-009812
 Commit:    e3abf6e935

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.0.100-preview-009812\

Host (useful for support):
  Version: 3.0.0-preview-27122-01
  Commit:  00c5c8bc40

.NET Core SDKs installed:
  1.1.0 [C:\Program Files\dotnet\sdk]
  2.0.3 [C:\Program Files\dotnet\sdk]
  2.1.402 [C:\Program Files\dotnet\sdk]
  2.1.500 [C:\Program Files\dotnet\sdk]
  2.1.600-preview-009426 [C:\Program Files\dotnet\sdk]
  2.1.600-preview-009472 [C:\Program Files\dotnet\sdk]
  2.2.101 [C:\Program Files\dotnet\sdk]
  3.0.100-preview-009812 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview-18579-0056 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview-27122-01 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-alpha-27128-4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[jainaashish]

Thanks for details, but I still don't have repro steps to reproduce your issue locally. I need details about project, packages, did you already had a lock file when you restored, does it throw hash mis-match error at the end of restore, etc... So It will be helpful if you can list down the exact repro steps one by one and if possible, share a sample project.

Also, are you running within VS or dotnet?

[Flavien]

I use VS. Restore works but generates a different SHA512 the second time, which is a red flag. And then restore fails in my CI build, which uses dotnet restore.

Is there a version of dotnet where this is supposed to be fixed?

I can try to build a minimal repro.

[bergbria]

I've created a repro using nuget.exe 4.9.3 and dotnet sdk 3.0.100-preview-010184. I don't know if it's identical to the issue @Flavien saw but it's impacting my use case.

@jainaashish, is the issue just that your fix hasn't made it into the dotnet distribution yet?

Unzip the below and run repro.bat to reproduce the failure.

lockFile_hashValidationErrors.zip

[jainaashish]

@jainaashish, is the issue just that your fix hasn't made it into the dotnet distribution yet?

Yes, is hasn't made into dotnet yet, it will be available in next dotnet preview/release.

Can you repro it with only nuget.exe 4.9.3?

[anangaur]

As these make into the .NET SDKs, we will update our table in the release notes:

[bergbria]

Can you repro it with only nuget.exe 4.9.3?

No. If I replace all the dotnet restore calls with nuget restore (and enable lock files in the csproj) the errors do not manifest.

This isn't a complete solution, though, since nuget.exe is missing some of the args from dotnet.exe (e.g. --lock-file-path and --locked-mode).

[madelson]

@jainaashish I'm also running into this. I just updated to VS 2017 15.9.6 since according to https://docs.microsoft.com/en-us/nuget/release-notes/nuget-4.9-rtm this should include NuGet 4.9.3 which fixes this. However, I still see this wrong hash issue. I also see it when we run MSBuild

Which specific versions should be expected to be correct here?

[jainaashish]

@madelson Are you using VS or dotnet CLI to repro your issue? VS 2017 15.9.6 includes NuGet 4.9.3 bits only for VS but it still doesn't update dotnet SDK. So make sure you're using VS to repro your issue or wait for the next dotnet SDK update.

[Flavien]

Same problem here. I'm using .NET Core SDK 2.2.104, which is supposed to have the 4.9.3 bits, and I am still getting the wrong hash error.

Is there some cache we need to clear before generating the lock file to ensure the incorrect .nupkg.metadata files are removed?

[anangaur]

/cc: @rrelyea

[anangaur]

@Flavien
Yes, you will have clean up the global packages folder once.

[Flavien]

I've already tried that and that didn't make a difference. I am still getting the sha512 validation error.

[anangaur]

Thats weird. /cc: @nkolev92

[nkolev92]

@Flavien

Do you have a minimal repro you can provide for us?

I want to eliminate sha validation issues that are potentially unrelated to the above.

[alefranz]

@Flavien from one of you previous comments it looks like you are experiencing an issue in CI.
You have to make sure your agent will not try to build any other project with RestorePackagesWithLockFile enabled but using an old sdk affetced by the bug (e.g. specified via global.json) after you cleanup your nuget fallback folder, otherwise you will be getting back the corrupted value in the cache.

[nexussays]

I'm also consistently seeing this with a UWP library and Microsoft.NETCore.Platforms. When I do nuget locals all -clear and then regenerate package.lock.json I get this hash:

"Microsoft.NETCore.Platforms": {
        "type": "Transitive",
        "resolved": "2.1.0",
        "contentHash": "GmkKfoyerqmsHMn7OZj0AKpcBabD+GaafqphvX2Mw406IwiJRy1pKcKqdCfKJfYmkRyJ6+e+RaUylgdJoDa1jQ=="
      }

If I were to re-run the build and regenerate the lock files again I would get this hash:

"Microsoft.NETCore.Platforms": {
        "type": "Transitive",
        "resolved": "2.1.0",
        "contentHash": "ok+RPAtESz/9MUXeIEz6Lv5XAGQsaNmEYXMsgVALj4D7kqC8gveKWXWXbufLySR2fWrwZf8smyN5RmHu0e4BHA=="
      }

The upshot is that every time I add/remove/update packages this value will be set to the incorrect one, causing the CI build to fail.

I'm presuming this is coming from a bad cache somewhere (what else could it be?), but I have <DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder> in Directory.Build.props.

What am I missing?

[nkolev92]

@nexussays
Try using the latest version of the client first.

Look at sources and track down where that package could be coming from. Your assets file with contain sources, fallback folders and related (I can help if you can't find them)

Keep in mind a few things.

Fallback folders are preferred over sources. Local sources are preferred over http.

If I were to re-run the build and regenerate the lock files again I would get this hash:

Do you clean the global packages folder or something here?

[nexussays]

@nkolev92 Thanks for the suggestions!

Try using the latest version of the client first.

Using nuget.exe 5.2.0.6090 in dev environment and on the CI server.

Look at sources and track down where that package could be coming from. Your assets file with contain sources, fallback folders and related (I can help if you can't find them)

It's a transitive dependency from UWP/UAP. I'm targeting 10.0.18362.0

Fallback folders are preferred over sources. Local sources are preferred over http.

So if there's no way to prevent the cache from altering the checksum how can I disable the use of caches other than <DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>?

[nexussays]

@nkolev92

Do you clean the global packages folder or something here?

When I clear nuget caches the hash is GmkKfoyerqmsHMn7OZj0AKpcBabD+GaafqphvX2Mw406IwiJRy1pKcKqdCfKJfYmkRyJ6+e+RaUylgdJoDa1jQ== when NuGet is (I presume) pulling it from one of its gazillion different caches, the hash is ok+RPAtESz/9MUXeIEz6Lv5XAGQsaNmEYXMsgVALj4D7kqC8gveKWXWXbufLySR2fWrwZf8smyN5RmHu0e4BHA== which will either cause the build to fail locally or to have this incorrect hash written to package.lock.json (i.e., if there are updates) which will then cause the CI server to fail.

[nkolev92]

NuGet is (I presume) pulling it from one of its gazillion different caches, the hash is

NuGet has an http-cache that lasts 30 mins, however I've rarely seen that as a problem. Once something is in the global packages folder, that's the final and permanent install location, no changes are ever made here.

DisableImplicitNuGetFallbackFolder might not apply to UWP projects because that's a .NET Core SDK switch, not a NuGet one.
Fallback folders can be added to your project through configs as well.

[Kurt-von-Laven]

I came across the same issue, @nexussays. See #11610 (comment). Did you ultimately find a resolution?

[zivkan]

Try <DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder> in your csproj (in a PropertyGroup)

Also: <fallbackPackageFolders><clear/></fallbackPackageFolders> in your nuget.config: https://github.com/NuGet/NuGet.Client/blob/7a5b0aafd767d105014c995378a1dbb71a5d86bd/NuGet.Config#L80-L82

If you have any .NET Core 1.x or .NET Core 2.x SDKs installed, uninstall them (you can still target older versions of netcoreapp with a newer SDK). On a clean machine with only in-support versions of the .NET SDK installed, you should no longer have any (dotnet install directory)\sdk\NuGetFallbackFolder on the machine, but deleting it without also removing whatever is telling NuGet to use that folder will probably result in errors when NuGet can't find the folder. You can however delete all the packages in that NuGetFallbackFolder directory.

[Kurt-von-Laven]

Thank you! I knew about DisableImplicitNuGetFallbackFolder, but not fallbackPackageFolders. I believe both of these are for .NET Core, but since we are making a UWP app, we instead have to create a Directory.Build.targets with:

<?xml version="1.0" encoding="utf-8" ?>
<Project>
    <PropertyGroup>
        <RestoreAdditionalProjectFallbackFolders></RestoreAdditionalProjectFallbackFolders>
    </PropertyGroup>
</Project>

The oldest version of .NET Core I have installed is 3.1. We intend to port to .NET Core though, so I'm sure this information will come in handy in the future! Re-reading your comment above, I realized that I actually observe the opposite of what you reported. ok+RPAtESz/9MUXeIEz6Lv5XAGQsaNmEYXMsgVALj4D7kqC8gveKWXWXbufLySR2fWrwZf8smyN5RmHu0e4BHA== is the hash when NuGet isn't pulling from the fallback folder, and GmkKfoyerqmsHMn7OZj0AKpcBabD+GaafqphvX2Mw406IwiJRy1pKcKqdCfKJfYmkRyJ6+e+RaUylgdJoDa1jQ== is the hash when it is. I'm not sure which is the correct hash.