[BUG]: Exit 57005 StackOverflowException using AzurePowershell@5 #20156

trevors20 · 2024-07-16T16:33:15Z

New issue checklist

Task name

AzurePowerShell

Task version

5.242.1

Issue Description

We are seeing a lot of Exit 57005 build breaks when using AzurePowerShell@5.
It looks like from the logs there is a StackOverflowException exception that is happening.
Was wondering if you might be able to help debug and fix this and/or help diagnose why this might be happening and provide a workaround or anything like that. Much appreciated.

Environment type (Please select at least one enviroment where you face this issue)

Self-Hosted
Microsoft Hosted
VMSS Pool
Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 2022

Relevant log output

2024-07-16T14:22:16.2568594Z VERBOSE: Testing whether type can be loaded: 'Microsoft.VisualStudio.Services.WebApi.VssClientHttpRequestSettings'
2024-07-16T14:22:16.2573359Z VERBOSE: The type was loaded successfully.
2024-07-16T14:22:16.2578674Z VERBOSE: Leaving Get-OMType.
2024-07-16T14:22:16.2793622Z 
2024-07-16T14:22:16.2793956Z Process is terminated due to StackOverflowException.
2024-07-16T14:22:16.2852511Z VERBOSE: AGENT_SKIPCERTVALIDATION (empty)
2024-07-16T14:22:16.2857885Z VERBOSE:  Converted to bool: False
2024-07-16T14:22:16.2863891Z VERBOSE: Constructing HTTP client.
2024-07-16T14:22:16.3287111Z VERBOSE: Leaving Get-VstsVssHttpClient.
2024-07-16T14:22:16.3296507Z VERBOSE: Removing assemlby resolver.
2024-07-16T14:22:16.3347027Z VERBOSE: SYSTEM_PLANID: '[REDACTED]'
2024-07-16T14:22:16.3377086Z VERBOSE: SYSTEM_JOBID: '[REDACTED]'
2024-07-16T14:22:16.3406912Z VERBOSE: SYSTEM_HOSTTYPE: 'build'
2024-07-16T14:22:16.3436924Z VERBOSE: SYSTEM_TEAMPROJECTID: '[REDACTED]'
2024-07-16T14:22:16.7096512Z ##[command]Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
2024-07-16T14:22:16.8158020Z ##[command]Clear-AzContext -Scope Process
2024-07-16T14:22:16.8497381Z ##[command]Connect-AzAccount 
2024-07-16T14:22:16.8497676Z Name                           Value                                                                                   
2024-07-16T14:22:16.8497908Z ----                           -----                                                                                   
2024-07-16T14:22:16.8498114Z Tenant                         [REDACTED]                                                   
2024-07-16T14:22:16.8498310Z Scope                          Process                                                                                 
2024-07-16T14:22:16.8498491Z Environment                    AzureCloud                                                                              
2024-07-16T14:22:16.8498754Z ApplicationId                  ***                                                    
2024-07-16T14:22:16.8498944Z ServicePrincipal               True                                                                                    
2024-07-16T14:22:16.8499030Z 
2024-07-16T14:22:16.8499074Z 
2024-07-16T14:22:16.8499122Z 
2024-07-16T14:22:16.8635574Z WARNING: The feature related to parameter name 'FederatedToken' is under preview.
2024-07-16T14:22:18.4771754Z ##[debug]Agent environment resources - Disk: D:\ Available 297357.35 MB out of 307198.00 MB, Memory: Used 5979.00 MB out of 32767.00 MB, CPU: Usage 7.00%
2024-07-16T14:22:23.7474754Z ##[debug]Agent environment resources - Disk: D:\ Available 297357.35 MB out of 307198.00 MB, Memory: Used 6086.00 MB out of 32767.00 MB, CPU: Usage 4.00%
2024-07-16T14:22:29.0309961Z ##[debug]Agent environment resources - Disk: D:\ Available 297357.35 MB out of 307198.00 MB, Memory: Used 6055.00 MB out of 32767.00 MB, CPU: Usage 9.00%
2024-07-16T14:22:34.2999428Z ##[debug]Agent environment resources - Disk: D:\ Available 297357.35 MB out of 307198.00 MB, Memory: Used 6075.00 MB out of 32767.00 MB, CPU: Usage 5.00%
2024-07-16T14:22:39.5848325Z ##[debug]Agent environment resources - Disk: D:\ Available 297357.35 MB out of 307198.00 MB, Memory: Used 6110.00 MB out of 32767.00 MB, CPU: Usage 8.00%
2024-07-16T14:22:44.8577747Z ##[debug]Agent environment resources - Disk: D:\ Available 297357.35 MB out of 307198.00 MB, Memory: Used 6151.00 MB out of 32767.00 MB, CPU: Usage 7.00%
2024-07-16T14:22:47.1743146Z ##[debug]Exit code: 57005

Full task logs with system.debug enabled

 LOGGING GOES OVER 65536 characters!  I will respond to this thread and upload it as a file.

Repro steps

- task: AzurePowerShell@5
          displayName: DisplayName
          inputs:
            azureSubscription: '[REDACTED]'
            workingDirectory: '$(Build.SourcesDirectory)'
            ScriptPath: 'src\Scripts\script.ps1'
            ScriptArguments: '-AccessToken $(System.AccessToken) -Param2 $(param2) -Param3 $(param3)'
            azurePowerShellVersion: LatestVersion
          condition: succeeded()

trevors20 · 2024-07-16T16:34:04Z

I'm attaching the
azurepowershell_error.txt
logs as a file to this comment.

trevors20 · 2024-07-16T16:39:28Z

This is happening 80% of the time. For the 20% of the time, it does work.
I'm uploading an example of logging when the task does actually work to this comment.
azurepowershell_working.txt

Please help. Thanks muchly!

trevors20 · 2024-07-16T22:18:03Z

We have decided to pin to an old version of AzurePowerShell@5 until this is fixed or figured out. That works great.

v-schhabra · 2024-07-17T17:13:17Z

Hi @trevors20
Thanks for reporting this issue. We are investigating on it.

huizhonglong · 2024-07-17T19:06:32Z

It'd heavily impacted our VS builds across pipelines before we pinned to the old version as a workaround to unblock folks.

You can get all the failed build IDs from below query if comment out the last line (| summarize) if that helps:

Go to Log Analytics and run query

let _start = datetime('2024-07-12T00:00:00Z');
let _end = datetime('2024-07-17T00:00:00Z');
customEvents
| extend BuildDefinitionId = tostring(customDimensions.BuildDefinitionId)
| extend BuildDefinitionName = tostring(customDimensions.BuildDefinitionName)
| extend BuildId = tostring(customDimensions.BuildId)
|where timestamp between(_start .. _end)
| where customDimensions.RecordName == "Layout BuildArtifacts" and customDimensions.Message contains "PowerShell exited with code '57005'"
| distinct BuildDefinitionId, BuildDefinitionName, BuildId
| summarize FailedCount = count() by BuildDefinitionId, BuildDefinitionName

BuildDefinitionId	BuildDefinitionName	FailedCount
10310	DD-CB-PR	41
10289	DD-CB-TestSignVS	13
11051	DD-CB-EUCU-TestSignVS	1

fernando-lopezblanco-lrn · 2024-07-18T16:13:35Z

Hi. Same. We've started experiencing these stack overflows today, with that AzurePowershell@5 task. Testing now version 4...

trevors20 · 2024-07-18T16:18:39Z

Thanks @fernando-lopezblanco-lrn . If it helps, we pinned to version 5.241.5 and that has stabilized our builds.

-task: AzurePowerShell@5.241.5

fernando-lopezblanco-lrn · 2024-07-18T16:27:01Z

Thanks @trevors20 for very quick response. Will try that version later on.

Same issue with AzurePowershell@4 (shown as Version : 4.242.0 in the pipeline logs). Testing now @5.241.5

fernando-lopezblanco-lrn · 2024-07-18T19:28:59Z

@5.241.5 is looking good, but other pipelines with @4 are starting to fail as well

trevors20 · 2024-07-18T19:46:05Z

@fernando-lopezblanco-lrn , I'm not sure which version to pin to for the AzurePowershell@4. But, you should be able to find that by looking at your older builds that were working before you started using the 4.242.0 version of the task.

At the top of the AzurePowershell@4 step in the build, you should see a header at the top of the logs. Something like this:

Task : Azure PowerShell
Description : Run a PowerShell script within an Azure environment
Version : 5.241.5
Author : Microsoft Corporation
Help : https://aka.ms/azurepowershelltroubleshooting

If you work backwards and find one of your older builds with a previous version than the 4.242.0 version then try pinning the task to the previous version that you find. To give a rough timeframe, I know that the AzurePowershell@5 task was updated a couple of weeks ago which started the instability here. So, you may need to look back a couple of weeks or so. :(

trevors20 · 2024-07-18T19:50:38Z

@fernando-lopezblanco-lrn , actually, we should be able to see the versions in the source code history. The version is specified in the task.json for the task. In this case for AzurePowershell@4, that would be here: https://github.com/microsoft/azure-pipelines-tasks/commits/master/Tasks/AzurePowerShellV4/task.json

Looking at the history, the change that was checked in a couple of weeks ago, if you look at the previous change it looks like you may want to pin to 4.241.2. Try it and see?

fernando-lopezblanco-lrn · 2024-07-19T07:18:36Z

@trevors20 yeah, the version 4.242.0 I gave you above, I took it from the header of the logs as you later mentioned.
Testing 4.241.0 now, for 4.241.2 does not look to exist as it's prompting me to install it from the Marketplace

admachina · 2024-08-26T16:15:03Z

Curious if anyone has seen this issue recently. We took the pinned version workaround, and I'm looking to unwind it if it's fixed.

admachina · 2024-08-27T16:12:43Z

Was able to get this to reproduce on 5.243.7

admachina · 2024-08-29T17:29:01Z

This bug seems it may be mitigated also by switching to pwsh mode instead of legacy powershell.
Adding pwsh: true to the pipeline

Having good success with this if you wanted to unpin the version

daxianji007 · 2024-09-02T08:23:56Z

Also have this issue. Waiting for advices.

onetocny · 2024-09-13T12:33:30Z

Hi @trevors20,

In order to proceed with investigation, could please share following details?

What kind of ARM service endpoint are you using in AzurePowerShell tasks?
Where is the build agent running?
What is content of script.ps1?
How does the whole pipeline definition looks like?

v-schhabra · 2024-09-25T04:46:47Z

Hi @trevors20
Could you please share the above details if the issue is occurring?

rcpd-alt · 2024-09-27T00:23:23Z

We're seeing this too, earliest seen build affected is 5.245.1 and still seeing it in 5.245.3

trevors20 · 2024-10-02T19:18:15Z

@onetocny , @v-schhabra
My apologies. Was on vacay when the questions were sent. Just now looking to see the status and just saw the questions.

This is an ARM using workload identity federation with openid connect
Not quite sure what you are looking for here. Do you mean the azure subscription? Or, the agent pool?
I'll send directly
I'll send directly

trevors20 added the bug label Jul 16, 2024

github-actions bot added Area: Release triage labels Jul 16, 2024

PhilipsonJoseph added Area:RM RM task team and removed Area:RM RM task team labels Jul 24, 2024

directhex mentioned this issue Jul 29, 2024

[BUG]: Exit 57005 StackOverflowException using AzureFileCopy@6 #20221

Open

7 tasks

onetocny mentioned this issue Oct 9, 2024

Log exceptions that occurs in Initialize-AzModule #20528

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG]: Exit 57005 StackOverflowException using AzurePowershell@5 #20156

[BUG]: Exit 57005 StackOverflowException using AzurePowershell@5 #20156

trevors20 commented Jul 16, 2024

trevors20 commented Jul 16, 2024

trevors20 commented Jul 16, 2024

trevors20 commented Jul 16, 2024 •

edited

Loading

v-schhabra commented Jul 17, 2024

huizhonglong commented Jul 17, 2024 •

edited

Loading

fernando-lopezblanco-lrn commented Jul 18, 2024

trevors20 commented Jul 18, 2024

fernando-lopezblanco-lrn commented Jul 18, 2024 •

edited

Loading

fernando-lopezblanco-lrn commented Jul 18, 2024

trevors20 commented Jul 18, 2024

trevors20 commented Jul 18, 2024

fernando-lopezblanco-lrn commented Jul 19, 2024 •

edited

Loading

admachina commented Aug 26, 2024

admachina commented Aug 27, 2024

admachina commented Aug 29, 2024

daxianji007 commented Sep 2, 2024

onetocny commented Sep 13, 2024

v-schhabra commented Sep 25, 2024

rcpd-alt commented Sep 27, 2024 •

edited

Loading

trevors20 commented Oct 2, 2024

[BUG]: Exit 57005 StackOverflowException using AzurePowershell@5 #20156

[BUG]: Exit 57005 StackOverflowException using AzurePowershell@5 #20156

Comments

trevors20 commented Jul 16, 2024

New issue checklist

Task name

Task version

Issue Description

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

Azure DevOps Server Version (if applicable)

Operation system

Relevant log output

Full task logs with system.debug enabled

Repro steps

trevors20 commented Jul 16, 2024

trevors20 commented Jul 16, 2024

trevors20 commented Jul 16, 2024 • edited Loading

v-schhabra commented Jul 17, 2024

huizhonglong commented Jul 17, 2024 • edited Loading

fernando-lopezblanco-lrn commented Jul 18, 2024

trevors20 commented Jul 18, 2024

fernando-lopezblanco-lrn commented Jul 18, 2024 • edited Loading

fernando-lopezblanco-lrn commented Jul 18, 2024

trevors20 commented Jul 18, 2024

trevors20 commented Jul 18, 2024

fernando-lopezblanco-lrn commented Jul 19, 2024 • edited Loading

admachina commented Aug 26, 2024

admachina commented Aug 27, 2024

admachina commented Aug 29, 2024

daxianji007 commented Sep 2, 2024

onetocny commented Sep 13, 2024

v-schhabra commented Sep 25, 2024

rcpd-alt commented Sep 27, 2024 • edited Loading

trevors20 commented Oct 2, 2024

trevors20 commented Jul 16, 2024 •

edited

Loading

huizhonglong commented Jul 17, 2024 •

edited

Loading

fernando-lopezblanco-lrn commented Jul 18, 2024 •

edited

Loading

fernando-lopezblanco-lrn commented Jul 19, 2024 •

edited

Loading

rcpd-alt commented Sep 27, 2024 •

edited

Loading