App Settings not available in some instances

[davetse]

@brettsam

Description

Some of my azure function instances are missing environment variables

Details

• Timestamp: 2019-10-18T06:47:28.5473944Z
• Function App version (1.0 or 2.0): 2.0
• Function App name: -----
• Function name(s) (as appropriate): EventLogger
• Invocation ID: "3fbd6722-4605-41da-892c-0522026db996"
• Region: WestUS
• Language: NodeJs

Provide the steps required to reproduce the problem:

1. In portal detail app setting DOCUMENTDB_HOST = "https://cosmosplaceholderurl.com"
2. Write a service bus triggered function to log app setting value
   context.log.info(Creating cosmos client to host ${process.env["DOCUMENTDB_HOST"]});
3. Deploy function to consumption plan
4. Send a bunch of requests to function to cause function to scale

Expected behavior

All instances will log "Creating cosmos client to host https://cosmosplaceholderurl.com"

Actual behavior

Some instances will log "Creating cosmos client to host undefined" repeatedly for entire lifetime of instance.
Other instances will be fine and log "Creating cosmos client to host https://cosmosplaceholderurl.com"

Known workarounds

Related information

• Started appearing on our radar on Oct 16, 2019
• NodeJs
• Servicebus input trigger

[brettsam]

@pragnagopa / @mhoeger -- @davetse reached out to me internally about this, but I'm not sure what could be happening. Is it possible to be missing environment variables immediately after specialization of a language worker? Have either of you seen this in the past?

It may be related to #1577, but I had him create a new issue as we hadn't heard anything on that one in a while (and it may have been .NET-specific).

[tuomov-tieto]

We encountered this as well with our node.js functions using EventHub and Scheduled triggers in West Europe, starting Oct 16th. We were configuring our function from env when the application was loaded and apparently the host wasn't necessarily fully initialized yet.

Support was very helpful and their suggestion of pinning to an earlier runtime version 2.0.12742.0 remedied the situation. Hoping the hotfix in 2.0.12808 will resolve this.

[julius-tieto]

The same issue also occured with our Nodejs Azure functions which are using Storage Queue and Scheduled triggers (West Europe, Windows consumption plan). The problem started Oct 16th. Our function was missing some environment variables which are set from the app settings.

Using an older version 2.0.12742.0 also fixed the issue.

[pragnagopa]

Investigating this issue.

[mhoeger]

Thanks for the details all! The root cause appears to be: #4488. It is intermittent because the ordering of environment variables matters for this bug to appear, and the order of environment variables varies per instance.

We're working on a fix.

[mhoeger]

A fix has been applied, and if you bring your functions runtime version back to ~2, environment variables will be read normally again.