[ExtensionsMetadataGenerator] Improve configuration for cleaning build output

[brettsam]

Today we have an "all or nothing" approach to cleaning your build output and disabling this involves a (currently) undocumented MSBuild setting: _FunctionsSkipCleanOutput. We're seeing a handful of customers run into this -- they move forward to a newer version of an assembly that we're cleaning away.

This often fails with Could not load file or assembly... exceptions during runtime.

I think we need to:

1. Make this setting not as "internal" (remove the underscore) and document this... I'd assume here: https://docs.microsoft.com/en-us/azure/azure-functions/functions-develop-vs
2. How fancy we can get with the cleanup and still have it be maintainable? Can we only delete if the version is to an exact match to what we have in https://github.com/Azure/azure-functions-host/blob/dev/tools/ExtensionsMetadataGenerator/src/ExtensionsMetadataGenerator/runtimeassemblies.txt?
3. If we can't do this smart enough with number 2, allow you use MSBuild items to selectively exclude or include assemblies. That way you can still benefit from the cleaning but get things running.

Some related issues that have come up (I'm resolving all of those to this one):

• Could not load file or assembly System.IdentityModel.Tokens.Jwt, Version=5.6.0.0 azure-functions-vs-build-sdk#397
• version 3.0.4 removes required dependencies azure-functions-vs-build-sdk#412
• Unable to start V3 Azure Function : Could not load file or assembly Azure-Functions#1518 (comment)
• Unable to load assembly after upgrading Microsoft.NET.Sdk.Functions to v1.0.33+ Azure-Functions#1525

[brettsam]

@fabiocav -- I'll sync with you on what we can do here.

[ramondeklein]

I'll share my workaround from #1525 for people who run into this issue:

It seems that the RemoveRuntimeDependencies task removes this assemblies (source). It doesn't run when the _FunctionsSkipCleanOutput variable is set to true. Adding the following line effectively disables this task and effectively disables the cleaning completely.

  <PropertyGroup>
    <_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput>
  </PropertyGroup>

The only downside is larger deployments...

[fabiocav]

Moving this as a stretch item for sprint 76. We'll need to at least define the work to be done.

[jzabroski]

Hi,

I am helping maintain RazorLight and am also the co-maintainer of FluentMigrator. To date, I have had zero interest in Azure Functions, but users keep clamoring for it (congratulations on the popularity!). My understanding is its the only practical option for serverless computing with .NET, and, as such, meets a need for .NET customers, but we're still working through what those exact needs are as it relates to modular, pluggable, hot-swappable computing.

That said, here is my broad take on things based on bugs that have been submitted to me:

1. Azure Functions needs to fundamentally re-think how it does modularity. Nate McMaster has done some proof-of-concept of what's possible with .NET Core AssemblyLoadContext viz a viz his DotNetCorePlugins github project, but there are still some deeper problems that issues like these reveal.
2. There are ancillary modularity issues like SQL Profiling that are not handled well from a modularity standpoint.
3. Until .NET Core 3.0, some of these modularity issues were not even solvable, like hot-swapping code. Nate recently updated his ASP.NET Core samples for DotNetCorePlugins to work with .NET Core 3.0 and even did a hot-swap example at my request. However, Nate also communicated to me that some modularity challenges aren't solvable at all with DotNetCorePlugins. It was a vague remark, but I imagine he is referring to things like SQL Profiling checkbox on the Azure Functions configuration screen, which, as best as I can tell from poking around, forces a specific version of a specific assembly to be loaded into the default AssemblyLoadContext, which is just magical yuck / Nickelodeon Gak in the sense that there is no way for me to prevent you from doing this and causing modularity problems.

I don't know if a sprint really solves these problems I observe. I personally will continue to not use Azure Functions. However, it would be nice if we could work together and have a script I can request my customers submit to me/you to troubleshoot these problems. I feel so bad for my users that they're on Azure Functions and I can't help them because of the black box nature.

Separately, I'm not into the weeds on what this RemoveRuntimeDependencies task is doing, BUT: I think you need to adopt a Tombstone design pattern instead of wiping a directory clean. Maybe there's an alternative I am not considering, but blowing away a directory with active open file handles seems like a really bad approach to hotswapping code. The Tombstone design pattern was invented in Smalltalk to deal precisely with hot swapping code. In this case, incoming requests to Azure Functions need to assign a pointer to a deployment directory. This design pattern is what Ruby on Rails has used for years with Capistrano deployment tool. Deployments are just directories, and the cap command-line tool just switches the virtual directory. This makes rolling back a failed deployment as easy as reverting any external db schema changes and running cap deploy:rollback -s previous_release={verison.number.to.rollback.to}. You can have a background task clean up Tombstones with no active requests. Ideally, do it in such a way that the .NET Garbage collector is that background task (clever)! There's probably some nuances here, but I think Nate has already solved some of those for you with his ASP.NET Core hot-swap example.

I hope this is taken as come constructive criticism. I've debated for months whether to push any feedback to Azure Functions, but I've probably now received over 100 requests for help with Azure Functions and am tired of it. I need better ways to support open source software customers.

[StephenCleary]

@jzabroski RemoveRuntimeDependencies is a build-time task; it doesn't have anything to do with hot-swapping or updates. The bug in this issue is when the function app uses a newer version of an assembly that the functions host also uses. Near the end of the build, RemoveRuntimeDependencies removes the assemblies that are in the functions host from the function app's output directory, and then the function app fails because it needs the newer assemblies and not the versions included in the function host.

There's a whole other question about whether the host should be providing any assemblies at all, and whether it would be cleaner to have an out-of-proc implementation, but then you run into problems supporting the strongly-typed C# binding, which is really nice to have.

BTW, thanks for RazorLight! I use it in one of my (non-Azure Functions) projects.

[jzabroski]

@StephenCleary I see. Here is my confusion, then: How can you RemoveRuntimeDependencies if there is a run-time setting called FUNCTIONS_EXTENSION_VERSION that can only be set to ~3 for "latest 3.x version"? How do you know what runtime dependencies you have if your runtime version is a moving target? If your users are compiling against 3.0.4 dependencies but Azure Functions fabric is running it with 3.1 the day 3.1 is officially released, users will perpetually be broken. It just seems like an unpredictable deployment model?

This is not a small point. Improving configuration for cleaning build output (title of this issue) DOES NOT solve the problem. It only provides a bandaid, one that gets ripped off every time Azure Functions does a minor .NET version upgrade and sucks in unwanted dependencies. I ran into similar problems when trying to figure out the following problem: How should FluentMigrator support .NET Core 3.x? Originally, I tried implementing the pragmatic wisdom of Nick Guerrera (the designer of the targeting pack system) and his advice didn't work so well and I landed on fluentmigrator/fluentmigrator#1178 instead, which is to regression test every LTS release of .NET via xunit, and support every LTS release of .NET via .NET CLI Tool and other nuget packages.

The crux of the problem is that everything depends on what's in your .deps.json file, so you really can't get away with opting in to .NET Core's default setting of <RollForward>Minor</RollForward>, because you're opting into potential breaking changes and those breaking changes get hardcoded into the .deps.json file you generate. We can create all kinds of baroque workarounds for .deps.json, but they're workarounds, and we already have so many of them, I'd rather we not add another. Just look at the code for OrchardCMS or RazorLight for all the different knobs we now have for building projects. All these workarounds have almost no structurally meaningful content. They're just hash-bang operators that do some magic to bin and pack and publish.

Another way to think of this problem is the one I pushed to Nick Guerrera and Dan Plaisted: Prove to me your design works, by implementing a DotNet CLI tool called TaskRunner that recursively runs open-ended Tasks given a TaskKey. If you can do that, you have an open, pluggable system. But we already know you can't do that with Azure Functions because you can't control the exact version of the runtime. Ceteris paribus, it won't work. It can't. We all know how to break this system and under what rules it can never work, yet, amazingly, Microsoft customers want it to work and expect it to work, and come to open source software library writers and ask us why our code won't work for something that is illogical.

P.S. Most of the credit for RazorLight goes to Ivan Balan, a brilliant engineer. I just couldn't stand to see the project die, given the alternatives all lacked IntelliSense and type safety from Visual Studio.

[StephenCleary]

Just for clarification, I don't work for MS. I'm just an AF user (and a bit of a fanboy). Watching this issue since I've been affected by it.

How do you know what runtime dependencies you have if your runtime version is a moving target?

Yes. This is a problem.

As an function app writer, I stay up-to-date, with a GitHub Action that updates all my dependencies nightly. This seems to be working well enough, and will do so until breaking changes occur (like this issue, which was added a few Functions.Sdk patch-versions ago). I minimize my library usage to minimize the number of breaking changes. I would expect most Functions apps are very simple and only depend on a handful of libraries.

As a library author, I'm not sure what the best approach would be. Currently my libraries all just take dependencies of "whatever version was current at the time I wrote them", which is less than ideal. I've been thinking of auto-updating all of them, too with autogenerated semver updates, but that can cause a lot of downstream churn. I have no idea what the best answer is there.

[jzabroski]

I would expect most Functions apps are very simple and only depend on a handful of libraries.

While I can accept the idea that the larger your dependency graph, the more likely you'll have dependency-related issues, it seems more fruitful to try to compile the list of problems and why they occur, one-by-one. I once challenged a Microsoft engineer, who claimed to have hit many of these problems himself, to list just some of them, and he politely refused.

Here, I am listing one of the problems: You cannot use .NET Core's dependency loading subsystem, which takes a .deps.json file and runtime.settings.json file, and expect it to work with the default RollForward Minor version functionality. It's supposed to work in a semver 2.0.0 model, but for practical reasons, it's very hard to align all ".NET In-Box" APIs in such a way that 3.0 to 3.1 doesn't contain any breaking changes. There are additional problems using ".NET Out-of-box" APIs in such ways, too, but they're hard for different reasons.

So, back to my original point. Configuring Build Output solves nothing. (Yes, I got it wrong in my original reply by assuming that Azure Functions had some kind of "runtime compilation" feature similar to ASP.NET Classic's incremental compilation of web pages. I just am not an active user of Azure Functions, and I thought, based on a user description of one of the symptoms happening without any deployment changes by them, that the code was compiled. In reality, it sounds like Azure Functions can upgrade the "latest runtime" whenever it wants (I believe it calls this feature "Trigger Syncing") but the underlying code remains the same statically built libraries. This actually seems much worse, to be honest, because you basically don't let MSBuild do its job which is to build a .deps.json file for you and a runtime.settings.json to create the correct virtual environment.)

[ramondeklein]

As an function app writer, I stay up-to-date, with a GitHub Action that updates all my dependencies nightly. This seems to be working well enough, and will do so until breaking changes occur (like this issue, which was added a few Functions.Sdk patch-versions ago). I minimize my library usage to minimize the number of breaking changes. I would expect most Functions apps are very simple and only depend on a handful of libraries.

@StephenCleary I am a big fan of using Paket. It's especially useful for multi-project solutions, but because it also keeps track of all dependencies, it much better in ensuring buillds are always using the same package versions and that all projects in your solution use the same version.

PS: Loved your book 👍

[lucavgobbi]

The current implementation enables full cleanup or no cleanup using _FunctionsSkipCleanOutput.

I think it would be viable to use the ../obj/project.assets.json file to exclude any DLLs that are listed under the dependencies key from being removed.

What do you think?

[miqm]

For me the problem was that Microsoft forgot (sic!) to publish updated runtime dependencies list (see Azure/azure-functions-vs-build-sdk#422) when aspnetcore3.1 was released.

As Azure Functions run on aspnet core and our DLL is part of that executable, some dependencies would need to be aligned. However the challenge here is to keep host assemblies and build sdk assembly list aligned, so they do not duplicate.

If you need full control of assemblies - go with docker. Using Azure Functions is nice but it has some downsides that we just need to accept for now that what we develop is not our application but we build a dynamic part of Azure Functions App.

My temporary solution to the problem was to add a target that runs after the cleanup and restores the dlls I need:

    <ItemGroup>
        <MissingAssembly Include="[dllNameHere]" Version="[dllVersionHere]" />
    </ItemGroup>
    <Target Name="CopyMissingAssemblies" AfterTargets="_FunctionsBuildCleanOutput">
        <Message Text="CopyMissingAssemblies" />
        <Copy SourceFiles="@(MissingAssembly->'$(NuGetPackageRoot)\%(Identity)\%(Version)\lib\$(TargetFramework)\%(Identity).dll')" DestinationFolder="$(OutDir)\bin" />
    </Target>
    <Target Name="CopyMissingAssembliesToPublish" AfterTargets="_FunctionsPublishCleanOutput">
        <Message Text="CopyMissingAssembliesToPublish" />
        <Copy SourceFiles="@(MissingAssembly->'$(OutDir)\bin\%(Identity).dll')" DestinationFolder="$(PublishDir)\bin" />
    </Target>

[cachai2]

@cachai2 @anthonychu

[jzabroski]

If you need full control of assemblies - go with docker. Using Azure Functions is nice but it has some downsides that we just need to accept for now that what we develop is not our application but we build a dynamic part of Azure Functions App.

Correct. The same more or less goes for Azure App Service.

[wjchristenson2]

Just upgraded to 5.0 and it appears the RemoveRuntimeDependencies task was the culprit for System.Private.CoreLib: Could not load file or assembly 'Microsoft.Extensions.Logging.Abstractions, Version=5.0.0.0 in our functions app.

<_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput>

Disabling the task fixed it.

[anthonychu]

@brettsam @fabiocav Could you please take a look? Also saw this error in this tweet.

[dsghi]

Just upgraded to 5.0 and it appears the RemoveRuntimeDependencies task was the culprit for System.Private.CoreLib: Could not load file or assembly 'Microsoft.Extensions.Logging.Abstractions, Version=5.0.0.0 in our functions app.

<_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput>

Disabling the task fixed it.

I have this error "Could not load file or assembly 'Microsoft.Extensions.Logging.Abstractions, Version=5.0.0.0" if I update Microsoft.Extensions.Configuration.UserSecrets from 3.1.6 to 5.0.0 - My probably dumb questions, where do I make the change for this Skip Clean Output?

[AndyMDoyle]

@dsghi, drop this in to your csproj file and it should prevent the assemblies being cleaned.

<PropertyGroup>
    <_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput>
</PropertyGroup>

the <_FunctionsSkipCleanOutput> line can be placed in an existing property group though too.

[dsghi]

Thank you for the quick response @AndyMDoyle, interestingly that did not solve the problem for me. We're just going to roll back the package updates and wait until things get sorted out. We don't absolutely need to be on the latest versions at the moment.