Az commands failing intermittently with StackOverflowException when CheckForUpgrade is enabled

[onetocny]

Description

The Az modules used in Azure Devops AzurePowerShell tasks are intermittently failing. We are using AzAccounts module to authenticate PowerShell scope against Azure resources. After importing Az.Accounts and running very first Az command (usually Get-AzConfig) PowerShell process intermittently exits with following error output (see the related issues below for more details about the symptoms of the issue):

Process is terminated due to StackOverflowException.
...
Exit code: 57005

The whole completed code that is responsible for Az module initialization could be found here. Here is the shortened code:

$module = Get-Module -Name "Az.Accounts" -ListAvailable | Sort-Object Version -Descending | Select-Object -First 1
Import-Module -Name $module.Path -Global -PassThru -Force
...
Get-AzConfig -AppliesTo Az -Scope CurrentUser

The process does not terminate immediately after Get-AzConfig but exists randomly in further code as it probably does not take constant time to overflow the stack. We have noticed that the issue appears across all versions starting 3.0.0.

Preliminary RCA

We were able to find these records in event log on build agent machines where the issue happens. The error messages are pointing to Azure Watson dump records. Here is the example. Looking at the callstack there I was able to localize that the issue happens in UpgradeNotificationHelper.RefreshVersionInfo. We were able to mitigate the issue on our side by leveraging this code and disabling the whole feature by following command:

Update-AzConfig -CheckForUpgrade $false -AppliesTo Az -Scope Process

However the Update-AzConfig has to be very first Az command we call otherwise the StackOverflowException occurs again. I believe that is caused by the fact that version check is called automatically after every Az command completes in AzurePSCmdlet.EndProcessing.

Related issues

• [BUG]: Exit 57005 StackOverflowException using AzurePowershell@5 microsoft/azure-pipelines-tasks#20156
• [BUG]: Exit 57005 StackOverflowException using AzureFileCopy@6 microsoft/azure-pipelines-tasks#20221

[isra-fel]

Thanks for the analysis. We indeed initiates a background thread to check for updates. Although it is still unclear why that would result in a stackoverflow.