[Discussion] Platform upgrade for Azure Functions v2 apps coming in October 2020

[anthonychu]

Discussion issue for: Azure/app-service-announcements#266

[shibayan]

Is it correct to understand that the Azure Functions Runtime version will remain v2 (maybe v2.1?) and the target .NET Core version will change to 3.1 (netcoreapp3.1)?

[anthonychu]

@shibayan The runtime will be upgraded to V3. If the app does not use any APIs that have changed between .NET Core 2.2 and .NET Core 3.1, it should continue to run on V3 without modification.

[shibayan]

@anthonychu Thanks!
Existing Azure Functions have FUNCTION_RUNTIME_VERSION set to ~2. Is that done by you changing it to ~3?
I will migrate to v3, but I am worried about the difference when I am creating a Function using IaC (Terraform).

[tomkerkhove]

This brings the question - Should .NET use the language server model like Python et al to avoid this in the future?

[ishepherd]

@anthonychu Will FUNCTIONS_V2_COMPATIBILITY_MODE setting still be available?
(And will it be automatically applied for these affected apps)?

[anthonychu]

@anthonychu Will FUNCTIONS_V2_COMPATIBILITY_MODE setting still be available?
(And will it be automatically applied for these affected apps)?

Yes. How this will work is the runtime will get updated to V3. FUNCTIONS_EXTENSION_VERSION will remain at ~2. This causes the app will run on the V3 runtime in V2 compat mode (setting FUNCTIONS_EXTENSION_VERSION to ~2 has the same effect as setting FUNCTIONS_V2_COMPATIBILITY_MODE to true).

As long as your .NET apps are not using APIs that changed between .NET Core 2.2 and 3.1 the upgrade should be seamless. We do encourage you to perform this upgrade yourself and test your application ahead of the automatic upgrade. You can do this by setting FUNCTIONS_EXTENSION_VERSION to ~3 and (if necessary) FUNCTIONS_V2_COMPATIBILITY_MODE to true.

[anthonychu]

This brings the question - Should .NET use the language server model like Python et al to avoid this in the future?

Yes. This is a direction we are actively investigating for future .NET versions.

[konstantinvlasenko]

Hello everyone!

We are getting this alert in portal for Functions v2. This is related to the anouncement. Correct?

• Can you also clarify fate of Function v1 as well. Would they be affected?

[pragnagopa]

cc @fabiocav

@konstantinvlasenko - yes, the advisory is related to this announcement. V1 apps are not affected by this.

[DanielLarsenNZ]

Hi @anthonychu, can you provide more details about what exactly is being upgraded in the platform? As in what is being added, upgraded and removed.

• Will v2 Framework runtimes be removed from D:\Program Files (x86)\SiteExtensions\Functions (if the customer does not opt-out)?
• Is Functions v2 really running on .NET Core 2.2 (as the wiki states)? This was EOL in December 2019
• Will any .NET Core 2 versions be removed from the Platform?
• Will App Service Environments also be upgraded?

[johndowns]

Also @anthonychu, as best I can tell, this is the state of play - can you please confirm?

• If the FUNCTIONS_EXTENSION_VERSION is set to ~2, it will move to Functions runtime v3 but in v2 compatibility mode. It will target .NET Core 3.1. It will receive patches.
• If the FUNCTIONS_EXTENSION_VERSION is set to ~2.0, it will stay on Functions runtime v2. It will target .NET Core 2.2. It will not be patched.
• If the FUNCTIONS_EXTENSION_VERSION is set to ~1 (or presumably ~1.0), it will stay on Functions runtime v1. This upgrade does not impact Functions V1 apps.
• If the FUNCTIONS_EXTENSION_VERSION is not present, it will get the latest major version - as of today that is Functions runtime v3.

If this is accurate, it might be helpful to put this somewhere explicit (like on the original post) to be super clear, as the post is a little ambiguous about what's happening.

[fowl2]

Much clearer, thanks @johndowns.

Presumably for Functions runtime v1 the .NET Framework will be patched with the rest of the Azure App Service - and even upgraded to 4.8 when that happens.

[pragnagopa]

@johndowns - No changes to V1 apps. We will be updating our documentation with upgrade instructions.

@DanielLarsenNZ -

Will v2 Framework runtimes be removed from D:\Program Files (x86)\SiteExtensions\Functions (if the customer does not opt-out)?

V2 runtime will exist but will only updated if critical fixes are needed.

Is Functions v2 really running on .NET Core 2.2

Yes, as of now Functions V2 apps are running on .Net core 2.2

Will any .NET Core 2 versions be removed from the Platform

No. Existing versions will remain but no more updates.

Will App Service Environments also be upgraded

Yes. This announcement includes all skus and ASEs

[harisubramaniam1]

Unfortunately, Service health alerts are only for app services that do not run on an Isolated ASP? I do not think those that have functions on Isolated ASP get these service health alerts and could potentially miss out on this?

[harisubramaniam1]

When I tried to pin the version to ~2.0, I get a warning message as below. I am opting out of the upgrade for now and hence the setting of ~2.0. But the warning message seems to indicate that once the support stops for the current version, it will be auto-upgraded to a major version. This message seems to contradict the announcement made here that pinning it to ~2.0 will prevent the functionapp host upgrade.

Can this be clarified please

[pragnagopa]

@harisubramaniam1 - thanks for reporting warning on Portal. This appears to be a UI issue and will follow up on fixing it.

[harisubramaniam1]

@pragnagopa Thank you for the clarification.

[srininz77]

1. We are running on Azure Functions runtime version pinned to 2.0.13351.0 targeting .net core 2.1. From what I read:
   “If the FUNCTIONS_EXTENSION_VERSION is set to ~2.0, it will stay on Functions runtime v2. It will target .NET Core 2.2. It will not be patched.”
   Can you please confirm if this is true if the functions_extension_version is set to “2.0.13351.0”

2. I also read that “We will be sending notices to subscriptions for applications that are using .NET APIs that are at risk for breaking with this platform upgrade.”
   We are running azure functions in ASE and I believe the resource health status is not available for isolated app service plans. Does this mean we will not receive any notices?

[anthonychu]

@pragnagopa Do you know if ASE customers were / will be notified of potentially breaking apps?

[pragnagopa]

@srininz77 - I understand you pinned runtime version due to : Azure/azure-functions-host#6558. Version 2.0.13351.0 will be available until the issue is fixed. Once the issue is fixed, if you still want to run on Functions V2, please pin runtime version to ~2.0 . Our recommendation is to move functions V3 as soon as you can.

@anthonychu / @srininz77 - Our internal analysis of potential breaking apps does include ASEs.

[hctan]

Just to confirm, my azure function is set to ~2 (I don't see an ~2.0 option):

If I understand correctly, to test it out (I wish to stay at function V2) before the automatic upgrade, is it I can go to 'Application Settings', add:
FUNCTIONS_EXTENSION_VERSION = ~3
FUNCTIONS_V2_COMPATIBILITY_MODE = true

Doing above would be similar to staying at ~2 as in my runtime version setting after 31st Oct right?

[ishepherd]

as of now Functions V2 apps are running on .Net core 2.2

@pragnagopa @anthonychu suggest you update the announcement, as it says:

Azure Functions V2 apps target .NET Core 2.1 (by default)

Also please confirm @johndowns's comment above. It's nice and clear so I started people linking to it, want to have official confirmation :)

[anthonychu]

@ishepherd We’re planning to follow up with some updates in the next couple of week. We’ll try to use the opportunity to also clear up the confusion. @jeffhollan

While they sound contradictory, both of those statements are true. By default, V2 apps target and are compiled against netcoreapp2.1. But they run on the V2 functions host which runs on .NET Core 2.2. The upgrade involves changing the host that runs your app. ~2 apps will run on the V3 host (running on .NET Core 3.1) in V2 compatibility mode.

Apps with the setting updated to ~2.0 will continue to run on the V2 host.

[anthonychu]

FUNCTIONS_EXTENSION_VERSION = ~3
FUNCTIONS_V2_COMPATIBILITY_MODE = true

@hctan These settings will cause the app to run on the V3 runtime with compatibility mode enabled. This is equivalent to how your app will run after the upgrade if you leave it on ~2.

If you want to pin the app to V2, you have to manually update FUNCTIONS_EXTENSION_VERSION to ~2.0. I would, however, strongly advise you try to run it on the V3 runtime (with or without V2 compatibility mode) as that will continue to receive security updates.

[hctan]

@anthonychu thanks for clarifying this.

[Bimmerdude]

Hi! We have a function that was written 4-5 years ago. That function is currently targeting framework 4.6. It is not clear to me how much of this is related to the function SDK and how much is related to .NET Core 3.1. We initially intended to just upgrade all functions to .NET Core 3.1 and function SDK v3. However, since this function actually makes calls specific to the full framework, we are wondering if this will break once the upgrade is completed in Azure. For example, this function makes calls to the log4net method GetLogger which used to have an overload that allowed you to specify only a string but this overload no loaded exists in the .NET Core 3.1 compliant log4net package. We are trying to assess the need to upgrade this function. It will be deprecated soon, however we may need to continue running it in parallel with it's replacement until after the October 31th deadline.

[pragnagopa]

@Bimmerdude can you confirm if your existing function app is running on V1 or V2?
Please note that V1 apps are not affected the platform upgrade. If you are running Functions V2 and cannot update to functions V3 which runs on .Net core 3.1 , please pin to ~2.0

[Bimmerdude]

@pragnagopa We're using V1 for this function.

[esbeneickhardt]

How are functions using Docker containers affected by the update?

[pragnagopa]

@esbeneickhardt - If your app is using a custom container it will not be auto-upgraded. Please not that this means your app will be running on a runtime that will not be patched. We recommend re-building the container using Functions V3 base image

[YusukeTobo]

FYI: for Linux users.
https://github.com/Azure/azure-functions-host/wiki/Platform-upgrade-from-Azure-Functions-runtime-v2-to-v3#opting-out-for-linux-apps

[pragnagopa]

While rolling out auto-upgrade on windows, we found upgraded apps running into a blocking issue: Azure/Azure-Functions#1370 . Issue is scoped to dotnet apps using system.data.sqlclient and that are built using older versions of nuget package: Microsoft.NET.Sdk.Functions that did not copy “functions.deps.json” file to function app’s bin directory.

We have paused upgrade process on Windows.

If your app is running on Linux and is facing the issue Azure/Azure-Functions#1370, please pin your app using instructions : https://github.com/Azure/azure-functions-host/wiki/Platform-upgrade-from-Azure-Functions-runtime-v2-to-v3#opting-out-for-linux-apps

[pragnagopa]

We have resumed the auto-upgrade process. Regions completed: West Central US, West US2, West Europe

[ivan-sam]

For the feedback:
Right after the update

We have resumed the auto-upgrade process. Regions completed: West Central US, West US2, West Europe
our functions stopped working after restart.

I've checked that the host is upgraded. The error is related to the name resolution for the Service Bus trigger. It's similar to this issue: Azure/azure-webjobs-sdk#2063. We use the service bus trigger like this:

[FunctionName("Function1")]
        public static async Task Run(
            [ServiceBusTrigger(
                "topic-name",
                "%topic-subscription-name%",
                Connection = "secret-name-with-service-bus-connection")]
            Message msg,
            ILogger logger)

After upgrade we've got an error:
The 'Function1' function is in error: Microsoft.Azure.WebJobs.Host: Error indexing method 'Function1'. Microsoft.Azure.WebJobs.Host: '%topic-subscription-name%' does not resolve to a value.
And Function 'Function1' failed indexing and will be disabled.

We use "Microsoft.NET.Sdk.Functions" Version="1.0.29" and "Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="3.0.6"

The workaround with setting FUNCTIONS_EXTENSION_VERSION to ~2.0 solved the issue.

[pragnagopa]

@ivan-sam - Thank you for sharing the details. We will look through the logs to see if we detect these errors and proactively pin apps to Functions V2.

[soma-kurisu]

[Auto-Upgrade] broke by setting FUNCTIONS_EXTENSION_VERSION to ~2.0 our azure function apps. Without any meaningful alerting, monitoring and logging in place (ouch) our production site went down. We haven't touched them for quit some time. A look on the dependent function version in one of our broken functions reveals the whole nightmare <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.24" />

With the only hint available - the output of a logic app kicking a function app to do some stuff - i stumbled upon following error message "The type initializer for '{some}Function' threw an exception." The Platform upgrade for Azure Functions v2 announcement lead me to this discussion.

So do you suggest updating our function apps to the latest and greatest (at least i hope so) v3 or will there be some fix in place?

[pragnagopa]

@sca-ai - thank you reporting issue here. Yes, we do recommend moving to Functions V3 at your earliest convenience. If possible, please do share full stack trace for the error (without secrets) - to help us investigate further.

[soma-kurisu]

@pragnagopa - updating the Functions SDK version to the latest major 1.x release (1.0.38) and updating our MondoDB dependency too did the trick. With this mitigation in place next we aim for Functions V3. Thx.

[pragnagopa]

Thanks @sca-ai ! Copied mitigation steps to Azure/azure-functions-host#7059 as well

[pragnagopa]

Updated #175 (comment) above - clarifying this upgrade does not impact Functions V1 Apps.

[bumgardnera07]

@pragnagopa If we have observed that a Function App has had the App Setting 'FUNCTIONS_EXTENSION_VERSION' = '~2.0' applied, but we cannot account for how that App Setting update was made, can we assume that the platform made this change as an automatic mitigation related to logging as mentioned in the comment above? #175 (comment)

What is the best way to identify the source of this App Setting update?

Furthermore, is there a way to identify who in any given subscription would have received notifications regarding this upgrade? In this particular case, we are trying to determine what notification was made to alert users about a potential breaking change for a Function App on ASE.

[pragnagopa]

@bumgardnera07 - We have pinned apps that we identified as breaking apps i.e. apps that are using Types or Methods that changed between .Net Core 2.2 and .Net Core 3.1 before starting the upgrade.

Apart from that - as we were rolling upgrade we identified apps running into issues tracked here. We have been actively pininng these apps as well to avoid disruption.

For details on why your app was pinned

1. Run your apps locally with V3 Core Tools. This will give details on any unsupported Types/Methods your app is using
2. If we tried to upgrade your app and ran into issues, please look up logs for your app for the HostVersion: 3.0.15278.0

For specific details on contact Id listed for your subscription, please open a support ticket as we do not have direct access to that information.

[saibaskaran57]

Hey everyone,

Could someone share how could we perform the Platform upgrade from Azure Functions v2 -> v3 without any downtimes?

There are two issues I'm currently facing in terms of the platform upgrade:

1. ARM template changes on FUNCTIONS_EXTENSION_VERSION from ~2.0 to ~3 causes downtime with HTTP 503 Service Unavailable
   I did a simple ping check(every 100ms) on my Azure functions when ARM template is being deployed. I could reproduce this using both Powershell & Azure resource group deployment azure pipeline task as show below:
   

2. I've changed my strategy to upgrade using Functions deployment staging slots first before making a swap. However, the FUNCTIONS_EXTENSION_VERSION app settings did not bring forward to production slot during this swap.
   Just to test this out, in my staging slot I updated FUNCTIONS_EXTENSION_VERSION from ~2.0 to ~3 and created a new app settings called NewVar. From the screenshot below, only NewVar is brought forward to production slot.
   

How can we ensure the function app is upgraded with seamless approach without causing service downtime?

Thanks.

[jahnwa]

We have a Function App, that runs on a P2v2 hosting plan that stopped working as intended after the app got restarted. The Function app uses Vnet integration to connect to our FTP server, hosted on an Azure VM. This is so that we privately communicate with the FTP (FileZilla) server.

This all worked until recently, when the function app was restarted due to a change in an app setting.

At the time of deployment in November 2020, it was running using FUNCTIONS_EXTENSION_VERSION: ~2. From my knowledge, it was using a 2.2.x runtime during this time. However, after the restart, still having the app setting FUNCTIONS_EXTENSION_VERSION: ~2 it now started using a dotnet 2.2 compatible dotnet 3.1.x runtime. Now, the Function app is not able to connect to our FTP server using the private IP. Yet, if we change the Function App to connect to the public IP of our FTP server, it is able to connect. This is what we are using as a temporary solution, but we want it to operate using private communication.

We receive the error message "Unable to connect to the remote server" using the private IP and FUNCTIONS_EXTENSION_VERSION: ~2. It throws an exception at System.Net.FtpWebRequest.GetResponse().

Opting-out of the platform upgrade, setting the app setting to FUNCTIONS_EXTENSION_VERSION: ~2.0, makes the Function App able to connect to both public and private IP of the FTP server. We do, however, want to upgrade our app to 3.1.x, but as of now this is not possible as we are not able to figure out what is causing this error.

Using any of the settings above, we are able to tcpping to connect to our FTP server using its private IP.

We have checked the article Breaking changes in .NET Core 3.1, but none of the changes, as far as we see it, is applicable in this situation.

Thanks for reading! 👍

[pragnagopa]

@saibaskaran57 - opened Azure/Azure-Functions#1831 as your question is related to deployment.

[pragnagopa]

@jahnwa - Please open issue here with a repro. That will help us investigate further and provide guidance on how you can migrate to Functions V3. Until then please pin your app to Functions V2

[lintho]

After the automatic upgrade to 3.0 our function stopped working.

The error message was:
Unable to cast object of type 'generatedProxy_3' to type 'HK.Plugin.POS.Services.ItemInfo.IItemInfoService'

We are using a WCF service(IItemInfoService), and the second time the function ran the error occured, and it kept having errors every run, until i restarted, and it worked ONCE, then errors again.

The WCF service was registered as an interface in startup, and was used with DI in the function.

Temp fix:
After downgrading to 2.0, it once again works. Well spent 6 hours.

Questions:

1. If anyone knows how to make it work on 3.0, it would be great so we dont lack behind on versions.
2. When i publish the function app now, i get a notice\warning in visual studio that i have to upgrade the function version. If i press no, i cannot publish. If i press yes, the configuration on the web app gets changed to 3.0 and the app is not working anymore. Is there a way for me to publish without this happening?

[pragnagopa]

@lintho - Apologies for the inconvenience. The error your app ran into is not caught in our analysis. For further assistance to move to functions v3, please open issue here : https://github.com/Azure/azure-functions-host/issues and provide a simple repro.

[ivan-sam]

@lintho

After the automatic upgrade to 3.0 our function stopped working.

The error message was:
Unable to cast object of type 'generatedProxy_3' to type 'HK.Plugin.POS.Services.ItemInfo.IItemInfoService'

I reported this issue for v3 24 days ago: Azure/azure-functions-host#7033
It used to be working fine on v3, but was broken recently. We pinned our functions that are using WCF to the latest working version (you can find all details in the issue). There was no attention so far, unfortunately. As upgrade brings the extension to the latest v3 version, it breaks functions working on v2.

Apologies for the inconvenience. The error your app ran into is not caught in our analysis. For further assistance to move to functions v3, please open issue here : https://github.com/Azure/azure-functions-host/issues and provide a simple repro.

@pragnagopa, could you please have a look at the issue Azure/azure-functions-host#7033? It contains description, simple repro, results of testing.

[lintho]

@lintho

After the automatic upgrade to 3.0 our function stopped working.
The error message was:
Unable to cast object of type 'generatedProxy_3' to type 'HK.Plugin.POS.Services.ItemInfo.IItemInfoService'

I reported this issue for v3 24 days ago: Azure/azure-functions-host#7033
It used to be working fine on v3, but was broken recently. We pinned our functions that are using WCF to the latest working version (you can find all details in the issue). There was no attention so far, unfortunately. As upgrade brings the extension to the latest v3 version, it breaks functions working on v2.

Apologies for the inconvenience. The error your app ran into is not caught in our analysis. For further assistance to move to functions v3, please open issue here : https://github.com/Azure/azure-functions-host/issues and provide a simple repro.

@pragnagopa, could you please have a look at the issue Azure/azure-functions-host#7033? It contains description, simple repro, results of testing.

Exactly the same as my scenario. Works first run after restart, then errors every run.

[maximivanov]

Here's my observation of how FUNCTIONS_EXTENSION_VERSION affects the host runtime choice for Windows and Linux function apps (Node.js).

Tests were made on freshly deployed function app instances: deploy FA → call https://<func app name>.azurewebsites.net/admin/host/status?code=<master code> → check runtime version.

Windows (same results for consumption and premium):

• ~2: 3.0.15371.0
• ~2.0: 2.0.15229.0
• ~3: 3.0.15371.0

Linux (same results for consumption and premium):

• ~2: 3.0.14916.0
• ~2.0: 3.0.14916.0
• ~3: 3.0.15277.0

I know for Linux to change the host runtime, linuxFxVersion setting has to be used. Still it seems FUNCTIONS_EXTENSION_VERSION also has some effect (~2 vs ~3 above)? Not sure how to explain this.

[eirikb]

Thank you @maximivanov !

Our function have been running fine since late 2018, today it crashed after restart with extensions/bindings not being registered:

The 'Subscription' function is in error: The binding type(s) 'blob, orchestrationClient' are not registered. Please ensure the type is correct and the binding extension is installed.

The ... function is in error: The binding type(s) 'activityTrigger' are not registered. Please ensure the type is correct and the binding extension is installed.

We had ~2 in FUNCTIONS_EXTENSION_VERSION, changing this to ~2.0 solved the issue.

Notes:

• We deploy from DevOps, so no local changes were causing it.
• Function is running Windows.
• Previous successful deploy (from DevOps) was last week (2021-03-10).

[eirikb]

Thank you @maximivanov !

Our function have been running fine since late 2018, today it crashed after restart with extensions/bindings not being registered:

The 'Subscription' function is in error: The binding type(s) 'blob, orchestrationClient' are not registered. Please ensure the type is correct and the binding extension is installed.

The ... function is in error: The binding type(s) 'activityTrigger' are not registered. Please ensure the type is correct and the binding extension is installed.

We had ~2 in FUNCTIONS_EXTENSION_VERSION, changing this to ~2.0 solved the issue.

Notes:

• We deploy from DevOps, so no local changes were causing it.
• Function is running Windows.
• Previous successful deploy (from DevOps) was last week (2021-03-10).