build(pipeline): Add pipeline package caching

[lpraz]

Add CacheBeta task to azure-pipelines.yml to enable NuGet package caching
and hopefully speed up CI builds.
Enable creation of packages.lock.json for projects built in pipeline to
enable above.

Resolves issue #667. So far the only cache task that was added is immediately after the dotnet build in the "Run Sonar Analysis" job, let me know if this misses out on any packages.

[rouke-broersma]

The run sonar stage is only executed for non-fork builds so it won't actually be triggered by your pr. The deploy integration test stage makes more sense I think.

[rouke-broersma]

Now that I think about it, all stages probably should have a cache task as they are all new agents.

Also your current implementation never has a cache update so new package versions will not be cached because of the following part of the docs:

Note
Caches are immutable, meaning that once a cache is created, its contents cannot be changed. See Can I clear a cache? in the Q & A section for additional details.

[lpraz]

Now that I think about it, all stages probably should have a cache task as they are all new agents.

Does this include the Sonar stage?

[rouke-broersma]

Yes, the sonar stage can use the cache if it's not a fork pr.

Also disregard the cache update comment, I did not realize a filepath hit on the key would actually hash the file contents and use that as the key.

We might want to have a mechanism to periodically clean the packages dir so the cache doesn't grow bigger and bigger for long lasting branches like master. The master cache will end up containing all package updates which can add up over time and actually increase build time due to cache download.

[rouke-broersma]

I would also like libman to be cached, since the cdn we use is unreliable and results in failed builds. The libman file as well as the destination dir is found here: https://github.com/stryker-mutator/stryker-net/blob/master/src/Stryker.Core/Stryker.Core/libman.json

[lpraz]

I would also like libman to be cached

Do you know of any documentation for libman? I've only been able to find anything about clearing the cache, rather than specifying where it should be kept. Disregard, did some more reading of the documentation I could find, and things make more sense to me now.

[lpraz]

Currently running into NU1403 (Package content hash validation failed) errors from Azure Pipelines for three packages:

• runtime.native.System.IO.Compression.4.3.0
• runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0
• runtime.ubuntu.14.04-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0
  Tried making the cache key OS-specific (since all of the packages are native), but to no avail. Is there anything else missing from my implementation?

[rouke-broersma]

We seem to be hitting this issue: microsoft/azure-pipelines-tasks#11449
Which to me suggests this is an issue with nuget. There are workarounds here:

NuGet/Home#7921 (comment) <- don't add the nowarn, I don't think that is neccesary

[rouke-broersma]

Hey @lpraz

It looks like $(Pipeline.Workspace)/.librarymanager is not the correct location for the libman cache. If I look at this pr on libman: https://github.com/aspnet/LibraryManager/pull/527/files it seems the location depends on the OS.. And it also seems like they are changing the cache location soon. Perhaps for now remove the libman caching so I can merge your PR and you or we can add the libman caching back when the libman cache location PR has been changed. Sounds good?

[rouke-broersma]

The cache location should become $(LocalAppData)/.librarymanager/cache in the future.
We could maybe fix it with the current cache location with something like this:

variables:
  LibManCacheWindows: $(USERPROFILE)/.librarymanager/cache
  LibManCacheNonWindows: $(HOME)/.librarymanager/cache

- task: CacheBeta@0
  displayName: 'Cache Library Manager libraries windows'
  inputs:
    key: libman | **/libman.json
    path: $(LibManCacheWindows) 
  condition: eq(variables['Agent.OS'], 'Windows_NT')
- task: CacheBeta@0
  displayName: 'Cache Library Manager libraries non windows'
  inputs:
    key: libman | **/libman.json
    path: $(LibManCacheNonWindows) 
  condition: ne(variables['Agent.OS'], 'Windows_NT')

[lpraz]

Tried making the suggested changes, still getting this:
libman.json : error LIB002: The "mutation-testing-elements@1.1.1" library could not be resolved by the "unpkg" provider [d:\a\1\s\src\Stryker.Core\Stryker.Core\Stryker.Core.csproj]
Should I remove/comment out the libman caching entirely for now?

[rouke-broersma]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[rouke-broersma]

That is an expected error and is one of the reasons we want to have pipeline caching! Unpkg is very unreliable causing the build to fail.

Libman caching seems to work now. We do have another issue with nuget caching.. I'll look into it today.

[rouke-broersma]

Thank you for your PR! With your starting point I managed to make it work. I was waiting on this feature very much so I went ahead and created a new branch with your changes and fixed it from there. That means that this PR is no longer needed so I'll be closing it.

If you were participating in hacktober, I do not believe closing this PR should have any impact on your t-shirt eligibility so don't worry about that.

Your help is much appreciated :)

[lpraz]

Sorry for the wait, I wasnt able to get to work on my PR in the past few days. Good to hear you were able to get it sorted out, and thanks! I don't have much experience with using YAML files for Azure Pipelines, so thanks for helping me out as well!

[rouke-broersma]

Not a problem at all, I was just too excited to use this to wait :d