FSharp.Core version 6.0.5 has multiple content hashes.

[JustinWilkinson]

Please provide a succinct description of the issue.
Version 6.0.5 of the FSharp.Core library seems to have two different contents depending on where it is delivered from.
The version bundled in the SDK (found under C:\Program Files\dotnet\sdk\6.0.400\FSharp\library-packs stems from this commit.

The version published on NuGet.org stems from this commit.

In our builds, we make use of packages.lock.json and global.json files to ensure repeatable builds across different machines.

We make use of the below in a Directory.Build.props file to ensure hashes are the same, and this has worked historically up until the 6.0.400 SDK release.

<RestorePackagesWithLockFile>true</RestorePackagesWithLockFile>
<DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>

The version on NuGet.org predates the version in the SDK, so the content hash on local builds seems to update but our pipelines are failing with the error:

error NU1403: Package content hash validation failed for FSharp.Core.6.0.5. The package is different than the last restore.

Given the hashes (and the content) are indeed different, this is expected, what is unexpected is the delivery of two distinct packages with the same version number.

Repro steps

1. Create a project that references FSharp.Core version 6.0.5, enable lock files and restore.
2. Install the 6.0.400 SDK.
3. Restore packages again.

Expected behavior
The content of the FSharp.Core package (version 6.0.5) delivered by the 6.0.400 SDK should match that on NuGet.org.

Actual behavior

The content of the FSharp.Core package (version 6.0.5) delivered by the 6.0.400 SDK is different to that which is delivered on NuGet.org.

Known workarounds

We have downloaded the package from NuGet.org and assigned it version 6.0.5.1 in our own feed to force it to pick up a new version.

• Operating system - Windows
• .NET Runtime kind - .NET 6.
• Editing Tools - Visual Studio version 17.3

[vzarytovskii]

@KevinRansom it seems that it is because we didn't bump a version

[cmeeren]

All our builds have started failing with this error now. Are there any easier/better workarounds than what @JustinWilkinson described (pushing the package from NuGet to a private feed and incrementing the version)? We'd have to do this for hundreds of projects. (Turning off lock files is not an option at the moment.)

[cmeeren]

I tried pinning the .NET SDK version to 6.0.303 using global.json, but I get the same result. (I verified that the CI restore task is using SDK Version: 6.0.303.) Since this bug was introduced with 6.0.400, I thought that this should work, but I have probably misunderstood something. Can anyone explain why it does not help pinning the SDK to pre-6.0.400 on CI?

[hcoona]

I also meet this issue

[cmeeren]

Pinning the .NET SDK to 6.0.302 worked. 6.0.303 did not work.

[vzarytovskii]

I think it is an issue with that we didn't bump the version when we were getting the localization changes in, and it went to SDK but not to the NuGet.

[vzarytovskii]

Is removing FSharp.Core (6.0.5) from local nuget cache and having <DisableImplicitLibraryPacksFolder>true</DisableImplicitLibraryPacksFolder> helping the build?

This should disable library-packs and it will be using FSharp.Core from the NuGet.

[bddckr]

Is removing FSharp.Core (6.0.5) from local nuget cache and having <DisableImplicitLibraryPacksFolder>true</DisableImplicitLibraryPacksFolder> helping the build?

Removing that version from the local cache does indeed help. For my team, this reliably pulls down the version found on NuGet.org. Note, however: We are using DisableImplicitNuGetFallbackFolder and NOT DisableImplicitLibraryPacksFolder. I have not tried the latter, as the former works fine for us, as long as the previously cached version is removed once.

So the steps are

1. Ensure your projects use <DisableImplicitNuGetFallbackFolder>true</DisableImplicitNuGetFallbackFolder>.
2. Delete the locally cached copy of 6.0.5 out of your .nuget folder, i.e. delete the folder C:\Users\<your windows user>\.nuget\packages\fsharp.core\6.0.5 on Windows.
3. Restore packages and update your lock files. This will write the other contentHash into the lock files.
4. Make those new lock files available to the rest of the team.
5. Ask every team member to wipe their cached copy followed by a restore as well.
6. If your CI caches restored packages, make sure to wipe the cached copy there as well.

[cmeeren]

A couple of questions:

1. What exactly is DisableImplicitLibraryPacksFolder? Web searches turn up nothing useful about this. I'm therefore hesitant to use it (especially since I am now using the 6.0.302 SDK as a simple and reliable workaround).
2. In the steps posted by @bddckr above, would it cause problems if some projects/solutions on your machine did not use DisableImplicitLibraryPacksFolder? Would they then try to place the package from the SDK in the NuGet cache (and win if you opened them first)?

[bddckr]

Apologies, I copied the wrong property and confused myself.

My team is using DisableImplicitNuGetFallbackFolder like the original poster said. This works fine - as long as every team member ensures to delete the cached version that has the "wrong" hash. We do NOT use DisableImplicitLibraryPacksFolder.

I've edited my earlier message to fix that.

[bddckr]

Both properties are seemingly very closely related for the same thing - the extra search paths pointing to files not sourced from NuGet.org (or your other configured feeds):
https://github.com/dotnet/sdk/blob/a30e465a2e2ea4e2550f319a2dc088daaafe5649/src/Tasks/Microsoft.NET.Build.Tasks/targets/Microsoft.NET.NuGetOfflineCache.targets#L20-L29

I'm thinking about enabling both moving forward. I believe DisableImplicitLibraryPacksFolder should be set to true for the same reason the recommendation is to enable DisableImplicitNuGetFallbackFolder:
https://github.com/dotnet/reproducible-builds/blob/d6b3923a0908249a8d6416a1e2d02c51e53b6cce/Documentation/Reproducible-MSBuild/Techniques/DisableImplicitNuGetFallbackFolder.md

[cmeeren]

My team is using DisableImplicitNuGetFallbackFolder like the original poster said. This works fine - as long as every team member ensures to delete the cached version that has the "wrong" hash. We do NOT use DisableImplicitLibraryPacksFolder.

Not sure what you mean by "works fine". I have long used DisableImplicitNuGetFallbackFolder (and not DisableImplicitLibraryPacksFolder), deleting the NuGet cache did not change the content hash in the lock files after dotnet restore --force-evaluate. Everything builds fine locally (either way), but the CI build fails.

I'm thinking about enabling both moving forward. I believe DisableImplicitLibraryPacksFolder should be set to true for the same reason the recommendation is to enable DisableImplicitNuGetFallbackFolder:

Makes sense. Would be great to have someone else confirm this.

[bddckr]

Not sure what you mean by "works fine". I have long used DisableImplicitNuGetFallbackFolder (and not DisableImplicitLibraryPacksFolder), deleting the NuGet cache did not change the content hash in the lock files after dotnet restore --force-evaluate. Everything builds fine locally (either way), but the CI build fails.

Hmmm. I just set up a fresh Windows VM (for something unrelated) and tried it there as well. You're right. It keeps using the library-packs folder. I must have gotten my results wrong when I was testing DisableImplicitLibraryPacksFolder earlier as well.

Finally, to confirm (sorry for the back and forth): If I'm setting both DisableImplicitNuGetFallbackFolder and DisableImplicitLibraryPacksFolder to true, the original content hash is used - the hash I got from before the 6.0.400 SDK was installed. As such, the lock file stays the same, and I don't have to ask my team members to wipe their cached copy. I simply need to ensure they are on the latest project state - using <DisableImplicitLibraryPacksFolder>true</DisableImplicitLibraryPacksFolder>.

tl;dr: vzarytovskii's suggestion above works.

[cmeeren]

Happy to hear it.

Would be great to get confirmation from at least one other knowledgeable person that using DisableImplicitLibraryPacksFolder is a general good idea for the same reasons as DisableImplicitNuGetFallbackFolder.

[vzarytovskii]

Happy to hear it.

Would be great to get confirmation from at least one other knowledgeable person that using DisableImplicitLibraryPacksFolder is a general good idea for the same reasons as DisableImplicitNuGetFallbackFolder.

It's fine to use it. It was made for scenarios like this - when there's need to turn off implicit packs import.

[cmeeren]

Thanks! Should it then perhaps be added alongside https://github.com/dotnet/reproducible-builds/blob/main/Documentation/Reproducible-MSBuild/Techniques/DisableImplicitNuGetFallbackFolder.md?

[baronfel]

I'm a maintainer on that project and would gladly accept a contribution outlining how and why to disable this property. Happy to help wordsmith it too!

[dsyme]

@vzarytovskii What's the status, is it point in time and can be closed, or do we need to fix it? Thanks

[vzarytovskii]

Yeah, this should be closed I think, nothing we can do for this particular version. Should be fixed for future ones (I will also have a CI gate in place to check it won't happen again).

[JustinWilkinson]

@vzarytovskii - not sure if the CI gate's worked.

We'd swapped to using DisableImplicitLibraryPacksFolder in most cases but a couple were missed and we've detected this issue again for version 6.0.6 distributed in the 6.0.402 SDK has this same issue, in that the commit hash is different to the one on NuGet
@dsyme

[vzarytovskii]

@vzarytovskii - not sure if the CI gate's worked.

We'd swapped to using DisableImplicitLibraryPacksFolder in most cases but a couple were missed and we've detected this issue again for version 6.0.6 distributed in the 6.0.402 SDK has this same issue, in that the commit hash is different to the one on NuGet
@dsyme

CI gate is not in place yet.

@KevinRansom we again have different hashes for the package in nuget and SDK

Not sure why. It seems like we publish a different version frow what we have in SDK?

Is it because we didn't manage to insert a certain things in VS but did in SDK?

[vzarytovskii]

Sigh..., it seems it is, in fact, caused by the differences in how insertions work for VS and SDK, and the package was published from VS pipeline, which has one less insertion.

[vzarytovskii]

We'll probably need to change the flow - how and from where do we publish nugets.

[KevinRansom]

sdk publishes them, and given that, I have no idea how they could mess that up.

[KevinRansom]

Okay, I believe I know what this is. It highlighted a bug in the sdk publish process that caused them to not update the package in the upload cache. I believe this release is the one they discovered it with and it will be fixed in the next release. I will follow up with @mmitche later today and report back if necessary.

[KevinRansom]

Okay ...

In SDK 6.0.401 we shipped FSharp.Core 6.0.6. We backported a change for the 6.0.402 SDK but failed to rev the nuget package version. When we shipped the 6.0.402 SDK the upload to nuget obviously failed because it already existed.

We have a mechanism to verify that this doesn't happen moving forward, however, I am not certain that it has been switched on, although I believe that I heard that it was switched on recently, because it happens during our insertion process. I will work with @vzarytovskii to ensure that it is definitely turned on.
I will also update the 17.3 branch to rev the fsharp.core version number to 6.0.7 so that should there be a future release of the dotnet SDK it will ship an update FSharp.Core nuget package.

[petteripoyhtari]

We have now currently this same issue with the 7.0.0 package and we have trouble to pass our Azure ci/cd pipeline builds through. Some machines will get the same hash and other won't.

Could you change the package version number when you change the package contents / code, please.

Maybe I need to open yet another issue with the 7.0.0 version...