FunctionsStartup not being called after runtime update in Azure, causing DI failure

[hallidev]

I had recently updated my Azure Functions from v3-preview to v3 after the .NET Core 3.1 GA announcement. Everything was running fine for development locally and in Azure.

At almost exactly 10:30PM EST on Friday (2019-12-13), all of my functions in development and production environments (two different resource groups) started failing.

Everything still runs fine locally.

The reason is that FunctionsStartup is no longer being run. I have no means by which to fix this as these functions were developed from day 1 in .NET Core 3 and have netcoreapp3.1 dependencies I can't change. I have no way to rollback these functions to v2.

My functions app on Azure reads:
Runtime version: 3.0.12939.0 (~3)

Local development reads:
Azure Functions Core Tools (3.0.2009 Commit hash: 77395527a4e9c28da8400dcfd1a450f4e0d0c36c)
Function Runtime Version: 3.0.12930.0

Let me reiterate - this was working after upgrading from v3-preview to v3. It stopped working after something was deployed by the Azure team

This is incredibly frustrating as a breaking changed was simply introduced by someone at Microsoft.

Investigative information

Please provide the following:

• Timestamp: 2019-12-16T20:08:06.480
• Function App version (1.0 or 2.0): 3.0.12939.0
• Function App name: CavFunctions-Dev
• Function name(s) (as appropriate): all functions
• Invocation ID:
• Region: East US

Repro steps

Register something in a DI Container like so:

[assembly: FunctionsStartup(typeof(MyApp.Functions.Startup))]
namespace MyApp.Functions
{
    public class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            // Register things in DI container
            builder.Services.AddScoped<Greeter>();
        }
    }
}

Use it like so:

    public class MyFunction
    {
        private readonly Greeter _greeter;

        public MyFunction(Greeter greeter)
        {
            _greeter = greeter
        }

        // Every 5 minutes
        [FunctionName(nameof(MyFunction))]
        public async Task Run([TimerTrigger("0 0/5 * * * *", RunOnStartup = true)]TimerInfo myTimer, CancellationToken cancellationToken)
        {
            await _greeter.Greet();
        }
    }

Expected behavior

Constructor injection works properly

Actual behavior

FunctionsStartup ignored, Greeter isn't registered in DI container, results in exception:

Unable to resolve service for type 'Greeter' while attempting to activate 'MyApp.MyFunction'. System.InvalidOperationException: at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService (Microsoft.Extensions.DependencyInjection.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at lambda_method (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)

Known workarounds

#5401 (comment)

[hallidev]

After another round of troubleshooting, I've determined that I'm experiencing this exact issue:

Azure/Azure-Functions#972
also
#3386

The only way I was able to get the functions running was to use the workarounds in both of the following comments:

#3386 (comment)
Azure/Azure-Functions#972 (comment)

Adding Directory.Build.targets didn't work on its own and manually adding extensions.json also didn't work on its own, but together I get the proper extensions.json in the publish folder:

C:\Projects\MyApp\MyApp.Functions\obj\Release\netcoreapp3.1\win-x64\PubTmp\Out\bin

I'm not going to bother opening another issue, but in case anyone reads this, I get OutOfMemoryExceptions after every publish which magically seems to resolve itself after about 30 minutes. It made troubleshooting this issue a true nightmare, and honestly this whole experience makes Azure Functions feel like an early beta / toy. I'm glad this is for a personal project as I'll be avoiding functions like the plague from here on out. This issue is the worst of the ones I've run into, but it's not nearly the first. The idea is cool, but I'll check back in a year or two when it's hopefully ready for production.

[Ludogo]

Hello
i had same problem on local machine, Startup class was not called anymore.
I had to delete

"extensionBundle": {
        "id": "Microsoft.Azure.Functions.ExtensionBundle",
        "version": "[1.*, 2.0.0)"

from host.json file to make Startup class be called again.
Hope this helps !

[wtt235-bh]

Hello
i had same problem on local machine, Startup class was not called anymore.
I had to delete

"extensionBundle": {
        "id": "Microsoft.Azure.Functions.ExtensionBundle",
        "version": "[1.*, 2.0.0)"

from host.json file to make Startup class be called again.
Hope this helps !

And that's in the built solution, correct?

[Ludogo]

Hello
i had same problem on local machine, Startup class was not called anymore.
I had to delete

"extensionBundle": {
        "id": "Microsoft.Azure.Functions.ExtensionBundle",
        "version": "[1.*, 2.0.0)"

from host.json file to make Startup class be called again.
Hope this helps !

And that's in the built solution, correct?

Yes, i've checked file history, and it was created by Visual Studio Code's Azure Function extension.

[fabiocav]

@soninaren this seems like an issue with the bundles selection process and the templates, which shouldn't apply to pre-compiled scenarios.

@Ludogo can you confirm the steps you've followed to create the Function project?

[soninaren]

I created a new C# precompiled project via VS Code, the host.json did not have bundle configuration in it. I tried adding several templates to it as well to make sure any additional action does not change host.json configuration. @Ludogo let us know if you have the repro steps for project creation.

[kzryzstof]

I got the same issue as well. The workaround was to downgrade Microsoft.NET.Sdk.Functions from 3.0.2 to 1.0.31.

[wtt235-bh]

I'm still having this problem - where the extensionBundle property in the host.json file is being populated incorrectly.
Here is my console log output when I debug the function app locally:

Azure Functions Core Tools (3.0.2009 Commit hash: 77395527a4e9c28da8400dcfd1a450f4e0d0c36c)
Function Runtime Version: 3.0.12930.0
[1/6/2020 4:54:49 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:java
[1/6/2020 4:54:49 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:node
[1/6/2020 4:54:49 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:powershell
[1/6/2020 4:54:49 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:python
[1/6/2020 4:54:49 PM] Building host: startup suppressed: 'False', configuration suppressed: 'False', startup operation id: '235a865c-ebf3-477b-b5f4-9976c66bf3fc'
[1/6/2020 4:54:49 PM] Reading host configuration file 'C:\...\bin\Debug\netcoreapp3.1\host.json'
[1/6/2020 4:54:49 PM] Host configuration file read:
[1/6/2020 4:54:49 PM] {
[1/6/2020 4:54:49 PM]   "version": "2.0",
[1/6/2020 4:54:49 PM]   "extensionBundle": {
[1/6/2020 4:54:49 PM]     "id": "Microsoft.Azure.Functions.ExtensionBundle",
[1/6/2020 4:54:49 PM]     "version": "[1.*, 2.0.0)"
[1/6/2020 4:54:49 PM]   }
[1/6/2020 4:54:49 PM] }
[1/6/2020 4:54:49 PM] Reading functions metadata
[1/6/2020 4:54:49 PM] 2 functions found
[1/6/2020 4:54:49 PM] Looking for extension bundle Microsoft.Azure.Functions.ExtensionBundle at C:\...\AppData\Local\Temp\Functions\ExtensionBundles\Microsoft.Azure.Functions.ExtensionBundle
[1/6/2020 4:54:49 PM] Found a matching extension bundle at C:\...\AppData\Local\Temp\Functions\ExtensionBundles\Microsoft.Azure.Functions.ExtensionBundle\1.1.1
[1/6/2020 4:54:49 PM] Fetching information on versions of extension bundle Microsoft.Azure.Functions.ExtensionBundle available on https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/index.json
[1/6/2020 4:54:50 PM] Skipping bundle download since it already exists at path C:\...\AppData\Local\Temp\Functions\ExtensionBundles\Microsoft.Azure.Functions.ExtensionBundle\1.1.1
[1/6/2020 4:54:50 PM] Loading Extention bundle from C:\...\AppData\Local\Temp\Functions\ExtensionBundles\Microsoft.Azure.Functions.ExtensionBundle\1.1.1
[1/6/2020 4:54:50 PM] Loading startup extension 'ServiceBus'
[1/6/2020 4:54:50 PM] Loaded extension 'ServiceBus' (3.2.0.0)
[1/6/2020 4:54:50 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:java
[1/6/2020 4:54:50 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:node
[1/6/2020 4:54:50 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:powershell
[1/6/2020 4:54:50 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:python
[1/6/2020 4:54:50 PM] Initializing Warmup Extension.
[1/6/2020 4:54:50 PM] Initializing Host. OperationId: '235a865c-ebf3-477b-b5f4-9976c66bf3fc'.
[1/6/2020 4:54:50 PM] Host initialization: ConsecutiveErrors=0, StartupCount=1, OperationId=235a865c-ebf3-477b-b5f4-9976c66bf3fc

To test, I've placed a breakpoint in the Startup.Configure() method and it isn't being hit during debug.

[fabiocav]

@wtt235-bh based on the logs above (thank you for sharing!), it looks like you may have extension bundles configured, which would cause the issue you're describing. Please remove the extension bundles configuration in host.json and try again.

[wtt235-bh]

@wtt235-bh based on the logs above (thank you for sharing!), it looks like you may have extension bundles configured, which would cause the issue you're describing. Please remove the extension bundles configuration in host.json and try again.

The host.json file i'm describing is the (global) one that's produced in the build (post compile), and not the one that's associated with each function (pre compile).
Essentially, I am not explicitly setting the "extensionBundle" property anywhere in my solution.

Just in case this is important - my installation of visual studio is updated to the latest version.

[fabiocav]

@wtt235-bh somehow, it does seem like that has been added to your host.json file. Could you confirm that it is there and that removing the setting resolves the issue?

[wtt235-bh]

Rebuilding the solution an manually removing the "extensionBundle" property before running the debugger allows the Startup class to execute.

Just to clarify something I said earlier, the global host.json file doesn't initially exist until after running the function app locally. So I have to run it, then modify the host.json file, the run it again to get it to work.

[fabiocav]

Explicitly adding a host.json file to your solution (and set its property to copy to the output folder) should address this issue.

@soninaren a new issue should be created to track this specific scenario with pre-compiled functions.

[wtt235-bh]

Explicitly adding a host.json file to your solution (and set its property to copy to the output folder) should address this issue.

That works. After manually adding a global host.json file (with no extensionBundle), the startup executes.

[VitorRigoni]

I already had a global host.json file with no extensionBundle and it was still failing. This is what fixed for me:

I got the same issue as well. The workaround was to downgrade Microsoft.NET.Sdk.Functions from 3.0.2 to 1.0.31.

[fabiocav]

@VitorRigoni looks like your issue was slightly different and you had a reference to 3.x when running on 2.0

[leboeuf]

@fabiocav I'm having the same issue, when you say "running on 2.0" which value are you referring to ?

• AzureFunctionsVersion in the csproj (mine is v3)
• FUNCTIONS_EXTENSION_VERSION in appsettings.json (mine is ~3)
• version in host.json (mine is 2.0 -- should it be 3.0?)

I also created extensions.json with the following content:

{
  "extensions": [
    {
      "name": "Startup",
      "typeName": "MyAssembly.Startup, MyAssembly, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"
    }
  ]
}

Downgrading Microsoft.NET.Sdk.Functions from 3.0.2 to 1.0.31 didn't fix it for me.

[fabiocav]

@leboeuf that was specifically about what @VitorRigoni was reporting above.

The version in host.json should remain as 2.0

Do you have the same issue mentioned above where the extension bundle configuration is defined in your host.json file?

[leboeuf]

@fabiocav I do not have an extension bundle in host.json, here is the whole file:

{
    "version": "2.0"
}

Is the extensions.json file mandatory for the Startup class to be called in Azure? It works locally without.

[leboeuf]

I got it working, my issue was that I was overwriting the configuration with this line:

builder.Services.AddSingleton<IConfiguration>(configuration);

This caused me to falsely believe that Startup wasn't called. I fixed it by removing the line and replacing it with something similar to #4464 (comment).

[Ludogo]

@fabiocav i had followed documentation available at https://docs.microsoft.com/fr-fr/azure/azure-functions/functions-develop-vs-code?tabs=csharp on 23th october with the current version of function SDK at this time.
I was able to see the difference on host.json file when i've created a new azure function project using the same documentation but with more up to date package on 19th december.

[fabiocav]

@Ludogo so, to clarify, the updated version doesn't have that reference, correct?

[Ludogo]

@Ludogo so, to clarify, the updated version doesn't have that reference, correct?

No it doesn't. So i've deleted extension bundle declaration and all works fine :)

[paonekumar]

@fabiocav it worked for me after deleting extensionBundle, but I am seeing performance issues after deleting extensionBundle from host.json.
host.json failing
{
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[1.*, 2.0.0)"
},
"singleton": {
"listenerLockPeriod": "00:00:15"
}
}

This is working, but I am seeing performance issues.

{
"version": "2.0",
"WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT": 1,
"singleton": {
"listenerLockPeriod": "00:00:15"
}

[benmiller86]

I had this issue today with a brand new functions project, the Startup.Configure(...) method was not being called at any point.

Visual Studio: 16.5.0 Preview 5.0
Azure Functions & Web Jobs Tools: 16.5.236.49856
Microsoft.NET.Sdk.Functions: 3.0.4
TargetFramework: netcoreapp3.1
AzureFunctionsVersion: v3

Downgrading Microsoft.NET.Sdk.Functions to 3.0.3 solved this, and the Startup.Configure(...) method is now called on startup as expected.

[danielegiallonardo]

I had this issue today with a brand new functions project, the Startup.Configure(...) method was not being called at any point.

Visual Studio: 16.5.0 Preview 5.0
Azure Functions & Web Jobs Tools: 16.5.236.49856
Microsoft.NET.Sdk.Functions: 3.0.4
TargetFramework: netcoreapp3.1
AzureFunctionsVersion: v3

Downgrading Microsoft.NET.Sdk.Functions to 3.0.3 solved this, and the Startup.Configure(...) method is now called on startup as expected.

Same thing just happened to me, no binding registration was found, I had to downgrade to 3.0.3 to have Startup.Configure(...) called again.

[HansOlavS]

I had the exact same issue as @benmiller86. I'm pretty sure 3.0.4 introduced a bug so FunctionsStartup/WebJobsStartup won't trigger any more.

[xsebas21]

I had this issue today with a brand new functions project, the Startup.Configure(...) method was not being called at any point.

Visual Studio: 16.5.0 Preview 5.0
Azure Functions & Web Jobs Tools: 16.5.236.49856
Microsoft.NET.Sdk.Functions: 3.0.4
TargetFramework: netcoreapp3.1
AzureFunctionsVersion: v3

Downgrading Microsoft.NET.Sdk.Functions to 3.0.3 solved this, and the Startup.Configure(...) method is now called on startup as expected.

Same here, downgrading to 3.0.3 solved the issue for me.

[coder-candace]

I can also confirm that it is an issue with Microsoft.NET.Sdk.Functions version 3.0.4. Issues was resolved once I downgraded back to 3.0.3 .

[henrikgering]

I have experienced this issue and the downgrade from 3.0.4 to 3.0.3 also resolved it.

[Alberto2686]

Downgrading to 3.0.3 fixed our functions as well. Too bad I found this thread after solving and not 6 hours ago...

[StefH]

@Alberto2686 Can you provide a working example project ?
I still have issues with v3 and netcore3.1, so I switched back to netstandard20 / v2

[Alberto2686]

@StefH I can't at the moment but I can suggest 2 checks:

1. Target Core 3.0 not Core 3.1 (and propagate the change to your dependencies of course)

2. In your functions csproj check that AzureFunctionsVersion is set to "v3" (without quotes)

Let me know it this helps, otherwise I can provide you a repo tomorrow

[TroyWitthoeft]

I've made a repo to reproduce @benmiller86 's bug. https://github.com/TroyWitthoeft/AzureFunctionsHostStartupBug

If you clone that down and debug, you'll notice the Startup.cs is not hit. If we update the csproj file and change the Microsoft.NET.Sdk.Functions version from 3.04 to either 3.0.3 or 3.0.5, then the Startup.cs file is again hit.

Project uses Functions v3, dotnet core 3.1
Local environment = VSCode 1.42.1 func core tools = 3.0.2245

[fabiocav]

The SDK issue has been resolved in 3.0.5. If you’re running into a problem after moving to 3.0.4, please just upgrade.

[StefH]

Does this apply for all types of Azure Functions?
Also for running Azure Functions on a Linux App Service plan, because that's my setup.
See also Azure/Azure-Functions#1518

[IanKemp]

The SDK issue has been resolved in 3.0.5. If you’re running into a problem after moving to 3.0.4, please just upgrade.

This is absolutely pathetic from Microsoft.

How is it possible that one of the world's largest software companies managed to push an update that is completely, fundamentally broken in the simplest way possible? Do you not have any tests of any sort for this software?

And why, for the love of all that is holy, have you not pulled the broken and useless 3.0.4 from NuGet? With a corresponding blog post to notify people to avoid that version? Or even an issue pinned on this repo?

Unbelievable. Just unbelievable.

[TroyWitthoeft]

@IanKemp I hear you. We burned some time on this bug too. It hurt. OTOH, this is a open source project. It's free. They accepts PRs. If you've got the talent, help them out and add some test coverage for this? We'd all benefit.

[IanKemp]

@IanKemp I hear you. We burned some time on this bug too. It hurt. OTOH, this is a open source project. It's free. They accepts PRs. If you've got the talent, help them out and add some test coverage for this? We'd all benefit.

That would require me to gift some of my own time to Microsoft. Considering that time is money, and that I've already paid for Azure, what you're effectively suggesting is that I pay Microsoft to write the tests that they, as a multibillion-dollar company, already should have in place as basic due diligence. Or to put it another way, Microsoft gets for free something they would normally have paid an employee for.

You can probably guess how I feel about doing that.

If we were talking about OSS from a nonprofit, my reaction would be quite different. But not only is this a for-profit company, and an extremely profitable one at that; we are literally paying for this functionality from said company. Open-source does not abrogate Microsoft of responsibility for basic quality, and it most certainly does not mean that we the paying end users should be willing to fall into the trap of doing Microsoft's work for it.

[hallidev]

@IanKemp This particular piece of functionality seems to break with every other release. I'm still using the workaround noted just below my original post because when I tried to take it out, I still couldn't manage to get my Startup method called.

I share your frustration, because like you I'm paying a hefty Azure bill every month. I don't put Azure Functions in the same OSS league as I do something like MediatR where the author gets very little (nothing?) for the work.

As it stands I have a few timer functions in production, but beyond that I'll check back on how this is doing in a year or two.

[StefH]

I still get an error when running a V3 function on a Linux App Service.
When deploying the same code (3.0.3 or 3.0.5) on Windows App Service, it works fine.

Anyone else noticed this same behaviour also?

See also Azure/Azure-Functions#1518

[fabiocav]

@StefH if you're still seeing this issue with the latest version of the SDK, can you please open a separate issue with the details requested on the template so we can take a closer look?

Closing this as the issue has been resolved in 3.0.5.

Also tracking the work to create additional validation steps in the coming spring: Azure/azure-functions-vs-build-sdk#404