FunctionsStartup not being called after runtime update in Azure, causing DI failure

[hallidev]

I had recently updated my Azure Functions from v3-preview to v3 after the .NET Core 3.1 GA announcement. Everything was running fine for development locally and in Azure.

At almost exactly 10:30PM EST on Friday (2019-12-13), all of my functions in development and production environments (two different resource groups) started failing.

Everything still runs fine locally.

The reason is that FunctionsStartup is no longer being run. I have no means by which to fix this as these functions were developed from day 1 in .NET Core 3 and have netcoreapp3.1 dependencies I can't change. I have no way to rollback these functions to v2.

My functions app on Azure reads:
Runtime version: 3.0.12939.0 (~3)

Local development reads:
Azure Functions Core Tools (3.0.2009 Commit hash: 77395527a4e9c28da8400dcfd1a450f4e0d0c36c)
Function Runtime Version: 3.0.12930.0

Let me reiterate - this was working after upgrading from v3-preview to v3. It stopped working after something was deployed by the Azure team

This is incredibly frustrating as a breaking changed was simply introduced by someone at Microsoft.

Investigative information

Please provide the following:

• Timestamp: 2019-12-16T20:08:06.480
• Function App version (1.0 or 2.0): 3.0.12939.0
• Function App name: CavFunctions-Dev
• Function name(s) (as appropriate): all functions
• Invocation ID:
• Region: East US

Repro steps

Register something in a DI Container like so:

[assembly: FunctionsStartup(typeof(MyApp.Functions.Startup))]
namespace MyApp.Functions
{
    public class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            // Register things in DI container
            builder.Services.AddScoped<Greeter>();
        }
    }
}

Use it like so:

    public class MyFunction
    {
        private readonly Greeter _greeter;

        public MyFunction(Greeter greeter)
        {
            _greeter = greeter
        }

        // Every 5 minutes
        [FunctionName(nameof(MyFunction))]
        public async Task Run([TimerTrigger("0 0/5 * * * *", RunOnStartup = true)]TimerInfo myTimer, CancellationToken cancellationToken)
        {
            await _greeter.Greet();
        }
    }

Expected behavior

Constructor injection works properly

Actual behavior

FunctionsStartup ignored, Greeter isn't registered in DI container, results in exception:

Unable to resolve service for type 'Greeter' while attempting to activate 'MyApp.MyFunction'. System.InvalidOperationException: at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService (Microsoft.Extensions.DependencyInjection.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at lambda_method (Anonymously Hosted DynamicMethods Assembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)

Known workarounds

#5401 (comment)

[hallidev]

After another round of troubleshooting, I've determined that I'm experiencing this exact issue:

Azure/Azure-Functions#972
also
#3386

The only way I was able to get the functions running was to use the workarounds in both of the following comments:

#3386 (comment)
Azure/Azure-Functions#972 (comment)

Adding Directory.Build.targets didn't work on its own and manually adding extensions.json also didn't work on its own, but together I get the proper extensions.json in the publish folder:

C:\Projects\MyApp\MyApp.Functions\obj\Release\netcoreapp3.1\win-x64\PubTmp\Out\bin

I'm not going to bother opening another issue, but in case anyone reads this, I get OutOfMemoryExceptions after every publish which magically seems to resolve itself after about 30 minutes. It made troubleshooting this issue a true nightmare, and honestly this whole experience makes Azure Functions feel like an early beta / toy. I'm glad this is for a personal project as I'll be avoiding functions like the plague from here on out. This issue is the worst of the ones I've run into, but it's not nearly the first. The idea is cool, but I'll check back in a year or two when it's hopefully ready for production.

[Ludogo]

Hello
i had same problem on local machine, Startup class was not called anymore.
I had to delete

"extensionBundle": {
        "id": "Microsoft.Azure.Functions.ExtensionBundle",
        "version": "[1.*, 2.0.0)"

from host.json file to make Startup class be called again.
Hope this helps !

[wtt235-bh]

Hello
i had same problem on local machine, Startup class was not called anymore.
I had to delete

"extensionBundle": {
        "id": "Microsoft.Azure.Functions.ExtensionBundle",
        "version": "[1.*, 2.0.0)"

from host.json file to make Startup class be called again.
Hope this helps !

And that's in the built solution, correct?

[Ludogo]

Hello
i had same problem on local machine, Startup class was not called anymore.
I had to delete

"extensionBundle": {
        "id": "Microsoft.Azure.Functions.ExtensionBundle",
        "version": "[1.*, 2.0.0)"

from host.json file to make Startup class be called again.
Hope this helps !

And that's in the built solution, correct?

Yes, i've checked file history, and it was created by Visual Studio Code's Azure Function extension.

[fabiocav]

@soninaren this seems like an issue with the bundles selection process and the templates, which shouldn't apply to pre-compiled scenarios.

@Ludogo can you confirm the steps you've followed to create the Function project?

[soninaren]

I created a new C# precompiled project via VS Code, the host.json did not have bundle configuration in it. I tried adding several templates to it as well to make sure any additional action does not change host.json configuration. @Ludogo let us know if you have the repro steps for project creation.

[kzryzstof]

I got the same issue as well. The workaround was to downgrade Microsoft.NET.Sdk.Functions from 3.0.2 to 1.0.31.

[wtt235-bh]

I'm still having this problem - where the extensionBundle property in the host.json file is being populated incorrectly.
Here is my console log output when I debug the function app locally:

Azure Functions Core Tools (3.0.2009 Commit hash: 77395527a4e9c28da8400dcfd1a450f4e0d0c36c)
Function Runtime Version: 3.0.12930.0
[1/6/2020 4:54:49 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:java
[1/6/2020 4:54:49 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:node
[1/6/2020 4:54:49 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:powershell
[1/6/2020 4:54:49 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:python
[1/6/2020 4:54:49 PM] Building host: startup suppressed: 'False', configuration suppressed: 'False', startup operation id: '235a865c-ebf3-477b-b5f4-9976c66bf3fc'
[1/6/2020 4:54:49 PM] Reading host configuration file 'C:\...\bin\Debug\netcoreapp3.1\host.json'
[1/6/2020 4:54:49 PM] Host configuration file read:
[1/6/2020 4:54:49 PM] {
[1/6/2020 4:54:49 PM]   "version": "2.0",
[1/6/2020 4:54:49 PM]   "extensionBundle": {
[1/6/2020 4:54:49 PM]     "id": "Microsoft.Azure.Functions.ExtensionBundle",
[1/6/2020 4:54:49 PM]     "version": "[1.*, 2.0.0)"
[1/6/2020 4:54:49 PM]   }
[1/6/2020 4:54:49 PM] }
[1/6/2020 4:54:49 PM] Reading functions metadata
[1/6/2020 4:54:49 PM] 2 functions found
[1/6/2020 4:54:49 PM] Looking for extension bundle Microsoft.Azure.Functions.ExtensionBundle at C:\...\AppData\Local\Temp\Functions\ExtensionBundles\Microsoft.Azure.Functions.ExtensionBundle
[1/6/2020 4:54:49 PM] Found a matching extension bundle at C:\...\AppData\Local\Temp\Functions\ExtensionBundles\Microsoft.Azure.Functions.ExtensionBundle\1.1.1
[1/6/2020 4:54:49 PM] Fetching information on versions of extension bundle Microsoft.Azure.Functions.ExtensionBundle available on https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle/index.json
[1/6/2020 4:54:50 PM] Skipping bundle download since it already exists at path C:\...\AppData\Local\Temp\Functions\ExtensionBundles\Microsoft.Azure.Functions.ExtensionBundle\1.1.1
[1/6/2020 4:54:50 PM] Loading Extention bundle from C:\...\AppData\Local\Temp\Functions\ExtensionBundles\Microsoft.Azure.Functions.ExtensionBundle\1.1.1
[1/6/2020 4:54:50 PM] Loading startup extension 'ServiceBus'
[1/6/2020 4:54:50 PM] Loaded extension 'ServiceBus' (3.2.0.0)
[1/6/2020 4:54:50 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:java
[1/6/2020 4:54:50 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:node
[1/6/2020 4:54:50 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:powershell
[1/6/2020 4:54:50 PM] FUNCTIONS_WORKER_RUNTIME set to dotnet. Skipping WorkerConfig for language:python
[1/6/2020 4:54:50 PM] Initializing Warmup Extension.
[1/6/2020 4:54:50 PM] Initializing Host. OperationId: '235a865c-ebf3-477b-b5f4-9976c66bf3fc'.
[1/6/2020 4:54:50 PM] Host initialization: ConsecutiveErrors=0, StartupCount=1, OperationId=235a865c-ebf3-477b-b5f4-9976c66bf3fc

To test, I've placed a breakpoint in the Startup.Configure() method and it isn't being hit during debug.

[fabiocav]

@wtt235-bh based on the logs above (thank you for sharing!), it looks like you may have extension bundles configured, which would cause the issue you're describing. Please remove the extension bundles configuration in host.json and try again.

[wtt235-bh]

@wtt235-bh based on the logs above (thank you for sharing!), it looks like you may have extension bundles configured, which would cause the issue you're describing. Please remove the extension bundles configuration in host.json and try again.

The host.json file i'm describing is the (global) one that's produced in the build (post compile), and not the one that's associated with each function (pre compile).
Essentially, I am not explicitly setting the "extensionBundle" property anywhere in my solution.

Just in case this is important - my installation of visual studio is updated to the latest version.

[fabiocav]

@wtt235-bh somehow, it does seem like that has been added to your host.json file. Could you confirm that it is there and that removing the setting resolves the issue?