Azure Portal can't list functions. Error: Azure Functions Runtime is unreachable. https://functions.azure.com/api/passthrough 500

[SaebAmini]

Investigative information

Please provide the following:

• Timestamp: 2019-08-28T03:15:31.304Z
• Function App version (1.0 or 2.0): 2.0
• Function App name: mhg-dev-dayforce-integrations-adp
• Function name(s) (as appropriate): All of them
• Invocation ID: this isn't for a specific invocation, as it's happening on the Azure Portal. I have "Session Id: 2c165a33f1934cf591994e1809771e7e"
• Region: Australia East

I've seen similar issues reported, but it seems all of them have been closed with no definitive outcome and below average investigations:

https://github.com/Azure/azure-functions-ux/issues/3166

#4081

Repro steps

1. Deploy any function app to our Isolated App Service Environment!
2. Open the app in the portal.

Expected behavior

Azure Portal should list functions and their status. This was working correctly for us about a month ago.

Actual behavior

Known workarounds

None

Related information

Provide any related information

• Programming language used: C#

[brettsam]

This happens consistently (i.e -- it's not temporary)? Looking at the logs, it looks like your functions are running correctly so it seems like something in the portal isn't working to communicate with the host.

@mathewc / @jeffhollan / @kulkarnisonia16 -- could this be related to https://github.com/Azure/azure-functions-ux/issues/3768?

[SaebAmini]

Thanks for replying Brett. I checked this morning and the issue is gone, so it was transient. However, it is a recurring issue and I've seen it many times, blocking us from managing things from the portal for hours. And I believe you're correct that it's because of the ILB ASE as that's the only place I see this.

[brettsam]

@SaebAmini -- do you continue to see this?

[jasseri]

I am seeing this issue right now.

[brettsam]

@jasseri -- are you also on an ILB ASE?

[jasseri]

@brettsam Not sure what you mean. I am running our function app in West Europe region, and the portal is now giving this error when I'm trying to list the proxies of the app.

[brettsam]

ILB ASE is a specific type of App Service -- https://docs.microsoft.com/en-us/azure/app-service/environment/create-ilb-ase. I'm assuming your answer is "no" :-) @SaebAmini was having a problem running as an ILB ASE.

Can you create a separate issue with your details? Specifically, if you could share your Function App name (either explicitly or privately), we can see if we find anything from the backend logs.

[jasseri]

@brettsam, I figured out what the issue with seeing my proxies was. I have limited the ip ranges from which my app can be accessed in Platform features -> Networking -> Access Restrictions. I wouldn't mind the portal being able to access the app and the confguration, but I guess that's just how it is, right?

[SaebAmini]

Thanks for checking in Brett. I see this continuously.

[brettsam]

@jasseri yes that may be an issue... someone with more knowledge about how the portal communicates may be able to weigh in (@btardif may know? -- see above comment: #4855 (comment))

@SaebAmini -- The issue that I linked above seems to indicate that this shouldn't be an issue anymore. @mathewc / @jeffhollan do we know if ILB ASE still have issues communicating with the functions host?

[jeffhollan]

Didn't read the full thread here but yes, until the portal UX refresh lands in ~Feb the portal will still rely on runtime calls for many things which will cause issues when using an ILB ASE and the users browser does not have permissions to call the function runtime within the ASE

[harishsune]

I had same issue this morning. By Scaling down the app service plan and again Scaling up to the previous plan it got resolved. I am using Python version of function app. Basically the thing I found was my every function app was affected due to this.

I tried to restart/stop function app but that did not resolved the issue. Then I created a new function app with same app service plan but that also got failed hence I found App service plan is the one which is causing other apps to fail so I tried to scale up and down it and it worked.

[xinyi-joffre]

Just ran into this with Azure Functions on Linux App Service Plan (S1) deployed from portal in East US. We've run into this multiple times with our function apps and az functionapp deployments are frequently failing. Function apps are not manageable from portal due to frequent inaccessible or unreachable errors.

We are not using ILB or any firewall rules.

Error:

OK - https://functions.azure.com/api/passthrough
Session Id: b8a2f32a843f4bc88b0c2d3b2bc40122

Timestamp: 2020-04-25T00:18:50.408Z

[brettsam]

@harishsune / @xinyi-joffre -- are you both using App Service Environments?

[harishsune]

Yes, the function kept showing same errors again and again. I had to delete the function app and start from scratch.

[xinyi-joffre]

No, we are not using ASE. We are on a linux dedicated app service plan for S1. We are frequently encountering issues with unreachable or inaccessible errors. Deployments also fail after getting 202s and then timeout.

[brettsam]

@xinyi-joffre -- if you're not on an ASE, can you open a separate issue with your symptoms? Then we can track that separately as they're likely separate issues.

[jeffhollan]

Unfortunately this error can be caused by any number of possible reasons. An invalid deployment package / format, an invalid container image (if you are publishing via container), networking hiccups, etc. I think the best bet to quick resolution would be opening a support ticket where we can dig into more of the specifics of this app and what could be causing it. Other places that can help is digging into the runtime logs during startup (or attempted startup). I generally look at App Insights and see if any host logs were registered, but also opening up the Log Streaming or pulling logs at the app level can give some clues (if something like a bad deployment package).

[sujitswaroop]

Unfortunately this error can be caused by any number of possible reasons. An invalid deployment package / format, an invalid container image (if you are publishing via container), networking hiccups, etc. I think the best bet to quick resolution would be opening a support ticket where we can dig into more of the specifics of this app and what could be causing it. Other places that can help is digging into the runtime logs during startup (or attempted startup). I generally look at App Insights and see if any host logs were registered, but also opening up the Log Streaming or pulling logs at the app level can give some clues (if something like a bad deployment package).

Unfortunately - the above reasoning does not apply to my situation. I am creating a basic azure function through the portal and it has been failing for the last two days. A default function app and it failed. There is no code or deployment package. I have checked the network settings of the storage account and it has "selected networks" and the "subnet" has been configured to allow the subnet the ASE is hosted in. Yet, I cannot get the function to work. I checked the storage account and the blob file contains the function name and few other details, so the function was able to access the storage account.

I tried scaling up and scaling down back again but it still did not work.

I have even tried deleting the function app and starting from scratch. Next, I am going to delete the App Service Plan and then create everything again from scratch.

[sujitswaroop]

What helped me resolve the issue was to add an "A" record to the on-prem DN Server and point it to the App Service Env IP.

[fabiocav]

Closing this as resolved/answered, but please let us know if you need any additional information.

[trivedihimanshu]

@fabiocav I am facing similar issue regarding this. my issue is that as soon as I navigate to function app it shows me Azure Functions runtime is unreachable.

Function APP is sitting behind ASE and ASE has more then 60 APPs and 5 Isolated APP Service plans

I've raised a SEV A case with Support and so far we have performed following steps.

1. Redeployed Function APP from Kudu using Zip.
2. Redeployed Function APP from Azure DevOps.
3. Verified TCPPING to storage account from kudu and Function app was able to reach to Storage account without any issue.
4. AzureWebJobsStorage has valid connection string to desired Storage account.
5. Storage Account doesn't have Network restrictions.
6. Verified all the mandatory/required appsettings/configuration on Function APP.
7. FUNCTIONS_EXTENSION_VERSION has valid value against it.
8. FUNCTIONS_WORKER_RUNTIME has valid value against it.
9. added https://functions.ext.azure.com to the Function app CORS setting and restarted FUNCTION app.

Request URL: https://functions.azure.com/api/passthrough
Request Method: OPTIONS
Status Code: 200
Remote Address: 13.107.246.10:443
Referrer Policy: no-referrer-when-downgrade

Request URL: https://functions.azure.com/api/passthrough
Request Method: POST
Status Code: 500
Remote Address: 13.107.246.10:443
Referrer Policy: no-referrer-when-downgrade

When browser is sending request method as OPTIONS its receive 200 OK but when it send POST it receives 500 on same Request URL.

I believe this issue can't be marked as closed.