Azure Service Connection using certificate authentication fails to get cert file password

[AngleOSaxon]

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name:

Any that take an AzureServiceConnection argument

Environment

• Server - Azure Pipelines or TFS on-premises?
  • Azure Pipelines

Account Name: transcardscm
Team Project Name: AccessControl
Build Definition Name/Build number: AccessControl Site - UAT - Release, build number 20221216.15

• Agent - Private:
  - If using private agent, provide the OS of the machine running the agent and the agent version:

OS: Windows Server 2016 Datacenter, version 1607, build 14393.5582
Agent Version: 2.213.2

Issue Description

I am using a privately-hosted agent and an Azure Service Connection configured with certificate authentication. When it attempts to execute a pipeline that authenticates to Azure using this service connection, it fails with this error:

"D:\AzureDevOps\agent\_work\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.209.0\ps_modules\VstsAzureHelpers_\openssl\openssl.exe" pkcs12 -export -in D:\AzureDevOps\agent\_work\_temp\clientcertificate.pem -out D:\AzureDevOps\agent\_work\_temp\clientcertificate.pfx -password file:"D:\AzureDevOps\agent\_work\_temp\clientcertificatepassword.txt"
Can't open file "D:\AzureDevOps\agent\_work\_temp\clientcertificatepassword.txt"
Error getting passwords

These pipelines and Service Connections work flawlessly on Microsoft-hosted agents.

Examining the private build agent with Procmon shows that openssl is handling the supplied password file path incorrectly, treating it as a relative file name rather than an absolute path.

In the task scripting, the password file path appears to be surrounded in quotes by line 257 in the Utility.ps1 file. Removing those quote marks allows the task to successfully read the password file and certificate, and to successfully authenticate.

Although removing the quotes does potentially allow spaces in the file path to break the command, the other paths supplied to the same command are in the same directory and are not quoted either, so this seems unlikely to cause any new issues.

Is there a known configuration change I can make to my private agent host so that openssl will process the $pfxPasswordFilePath properly, the way it does on the Microsoft-hosted agents? Or should I open a PR to remove the quotes from that argument in Utility.ps1?

Task logs

PrivateHost_FailedAuth_Cert.zip

Error logs

VERBOSE: Entering Invoke-VstsTool.
VERBOSE:  FileName: 'D:\AzureDevOps\agent\_work\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.209.0\ps_modules\VstsAzureHelpers_\openssl\openssl.exe'
VERBOSE:  Arguments: 'pkcs12 -export -in D:\AzureDevOps\agent\_work\_temp\clientcertificate.pem -out D:\AzureDevOps\agent\_work\_temp\clientcertificate.pfx -password file:"D:\AzureDevOps\agent\_work\_temp\clientcertificatepassword.txt"'
VERBOSE:  RequireExitCodeZero: 'True'
"D:\AzureDevOps\agent\_work\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.209.0\ps_modules\VstsAzureHelpers_\openssl\openssl.exe" pkcs12 -export -in D:\AzureDevOps\agent\_work\_temp\clientcertificate.pem -out D:\AzureDevOps\agent\_work\_temp\clientcertificate.pfx -password file:"D:\AzureDevOps\agent\_work\_temp\clientcertificatepassword.txt"
Can't open file "D:\AzureDevOps\agent\_work\_temp\clientcertificatepassword.txt"
Error getting passwords
VERBOSE: Exit code: 1
VERBOSE: Leaving Invoke-VstsTool.
##[error]Process 'openssl.exe' exited with code '1'.
##[debug]Processed: ##vso[task.logissue type=error]Process 'openssl.exe' exited with code '1'.
VERBOSE: Leaving Initialize-AzModule.

[peterhut]

I had the same issue on 1 server, but not on another. The key difference ended up being the PowerShell version: using 7.3.2 gives the error, but PowerShell version 7.2.9 does not. Therefore my workaround was installing 7.2.9 on both servers.

[jberezanski-mdg]

I have the same problem. In my case: AzurePowerShell task 5.225.1, PowerShell version 7.3.5. Downgrading PowerShell is not an option for my team.

==============================================================================
Task         : Azure PowerShell
Description  : Run a PowerShell script within an Azure environment
Version      : 5.225.1
Author       : Microsoft Corporation
Help         : https://aka.ms/azurepowershelltroubleshooting
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\B\2\_temp\8b3411ed-3826-4e75-98d1-4958ce54e739.ps1'"
Added TLS 1.2 in session.
Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Accounts\2.12.1\Az.Accounts.psd1 -Global
Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
Clear-AzContext -Scope Process
Clear-AzConfig -DefaultSubscriptionForLogin
"D:\B\2\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.225.1\ps_modules\VstsAzureHelpers_\openssl\openssl.exe" pkcs12 -export -in D:\B\2\_temp\clientcertificate.pem -out D:\B\2\_temp\clientcertificate.pfx -password file:"D:\B\2\_temp\clientcertificatepassword.txt"
Can't open file "D:\B\2\_temp\clientcertificatepassword.txt"
Error getting passwords
##[error]Process 'openssl.exe' exited with code '1'.
##[error]There was an error with the service principal used for the deployment.
##[error]PowerShell exited with code '1'.
Disconnect-AzAccount -Scope Process -ErrorAction Stop
Clear-AzContext -Scope Process -ErrorAction Stop

[KrazeyKami]

Same issue. Can't downgrade.

[jberezanski-mdg]

The immediate cause of the problem seems to be a change in the way Invoke-Expression quotes command lines passed to external executables, which apparently happened between PS 7.2 and 7.3.

The AzurePowerShell task calls Initialize-AzModule from VstsAzureHelpers_ and execution flow eventually enters the ConvertTo-Pfx function. There a command line for openssl is constructed and passed to Invoke-VstsTool from VstsTaskSdk:

$openSSLArgs = "pkcs12 -export -in $pemFilePath -out $pfxFilePath -password file:`"$pfxPasswordFilePath`""

Invoke-VstsTool -FileName $openSSLExePath -Arguments $openSSLArgs -RequireExitCodeZero

Invoke-VstsTool passes its input directly to Invoke-Expression:

Invoke-Expression "& '$FileName' --% $Arguments"

Here are the actual command lines, as observed via Sysinternals Process Monitor, on various PowerShell versions, with System_DefaultWorkingDirectory set to a path with and without spaces:

PS 5.1

"F:\Temp\openssl\openssl.exe"  pkcs12 -export -in F:\Temp\nospaces\clientcertificate.pem -out F:\Temp\nospaces\clientcertificate.pfx -password file:"F:\Temp\nospaces\clientcertificatepassword.txt"

"F:\Temp\openssl\openssl.exe"  pkcs12 -export -in F:\Temp\with spaces\clientcertificate.pem -out F:\Temp\with spaces\clientcertificate.pfx -password file:"F:\Temp\with spaces\clientcertificatepassword.txt"

PS 7.2.12

"F:\Temp\openssl\openssl.exe"  pkcs12 -export -in F:\Temp\nospaces\clientcertificate.pem -out F:\Temp\nospaces\clientcertificate.pfx -password file:"F:\Temp\nospaces\clientcertificatepassword.txt"

"F:\Temp\openssl\openssl.exe"  pkcs12 -export -in F:\Temp\with spaces\clientcertificate.pem -out F:\Temp\with spaces\clientcertificate.pfx -password file:"F:\Temp\with spaces\clientcertificatepassword.txt"

PS 7.3.6

"F:\Temp\openssl\openssl.exe" pkcs12 -export -in F:\Temp\nospaces\clientcertificate.pem -out F:\Temp\nospaces\clientcertificate.pfx -password "file:\"F:\Temp\nospaces\clientcertificatepassword.txt\""

"F:\Temp\openssl\openssl.exe" pkcs12 -export -in F:\Temp\with spaces\clientcertificate.pem -out F:\Temp\with spaces\clientcertificate.pfx -password "file:\"F:\Temp\with" "spaces\clientcertificatepassword.txt\""

Two things are apparent here:

1. Quotes embedded in the command line are doubly-quoted on PS 7.3. The command line is even more mangled when the path contains a space.
2. The code does not attempt to handle spaces in the values of the -in and -out arguments to openssl. On all PowerShell versions, the code does not work correctly if $Env:System_DefaultWorkingDirectory (or $Env:Agent_TempDirectory) contains spaces.

Fully fixing the problem with spaces in paths seems difficult to do, due to the design of Invoke-VstsTool and its reliance on Invoke-Expression and a single parameters string (instead of an array of parameters). However, the regression on PS 7.3 can be fixed simply by removing the quotes from the -password argument. This will make the behavior on PS 7.3 not worse than on earlier PowerShell versions - paths with spaces will still not work, but paths without them will (and I suspect this is the typical case for agent working directories).

Specifically, I propose the following change:

diff --git a/Tasks/Common/VstsAzureHelpers_/Utility.ps1 b/Tasks/Common/VstsAzureHelpers_/Utility.ps1
index 3b49b5c2c9..ef482fedee 100644
--- a/Tasks/Common/VstsAzureHelpers_/Utility.ps1
+++ b/Tasks/Common/VstsAzureHelpers_/Utility.ps1
@@ -355,7 +355,7 @@ function ConvertTo-Pfx {
     $env:OPENSSL_CONF = "$PSScriptRoot\openssl\openssl.cnf"
     $env:RANDFILE=".rnd"

-    $openSSLArgs = "pkcs12 -export -in $pemFilePath -out $pfxFilePath -password file:`"$pfxPasswordFilePath`""
+    $openSSLArgs = "pkcs12 -export -in $pemFilePath -out $pfxFilePath -password file:$pfxPasswordFilePath"

which will result in the same command line being generated in all three PS versions, fixing this issue. Would this be acceptable?

[andreascaesar]

Any progress on this?
Still the same issue in 7.4...

[btull89]

This has become a problem in several of my azure pipelines over the weekend. Can Microsoft provide an update to this issue?
We are using Azure DevOps Hosted Agents.

[AngleOSaxon]

As a temporary workaround, they did provide scripts to downgrade Powershell back to 7.2.17 when they announced the update. I added them as steps at the start of my pipelines and they successfully downgraded the Powershell version used by all subsequent steps and got my pipelines running again.

It does need to get fixed, though; staying on 7.2 forever isn't a great plan.

[Jetelaczek]

downgrade

Morning, could you please give me some example? Facing same issue, but don't understand how you 'added them as steps at the start of my pipelines'.

Thanks!

[AngleOSaxon]

Sure. I created a script file called downgrade-powershell.cmd and placed it in a scripts directory, and then added this step early in each pipeline:

  - task: CmdLine@2
    displayName: Downgrade to Powershell 7.2
    inputs:
      failOnStderr: true
      workingDirectory: $(Agent.BuildDirectory)/scripts
      script: downgrade-powershell.cmd

the contents of downgrade-powershell.cmd are

set "extractPath=C:\Program Files\PowerShell\7"
curl -sLO https://github.com/PowerShell/PowerShell/releases/download/v7.2.17/PowerShell-7.2.17-win-x64.zip
RMDIR "%extractPath%" /S /Q
7z x PowerShell-7.2.17-win-x64.zip -o"%extractPath%"
pwsh --version

My pipelines almost all run on Windows, so I'm able to get away with just using a CmdTask and a batch file. If you've got a more heterogeneous environment, you might need to also add tasks with the macOS and Linux downgrade scripts they provided. You should be able to pick which one to run with something like this:

  - ${{ if eq(Agent.OS, 'Windows_NT') }}:
    - script: windows-downgrade-powershell.cmd
  - ${{ elseif eq(Agent.OS, 'Darwin') }}:
    - script: macos-downgrade-powershell.sh
  - ${{ elseif eq(Agent.OS, 'Linux') }}:
    - script: linux-downgrade-powershell.sh

(disclaimer: not tested)

[quality-leftovers]

Our pipelines are broken now, too. It would be nice if either

• azure devops pipeline tasks didn't randomly break so often
• microsoft would actually try to fix stuff asap when stuff breaks instead of being so passive

[Jetelaczek]

I had a chat with Microsoft support and what works for our pipelines is to use previous version of MS hosted agent (use 'windows-2019' instead of 'windows-latest') and use 'AzurePowerShell@4'...

[Jetelaczek]

Thanks much! Tomas J. From: Andrew Bikle ***@***.***> Sent: Thursday, February 1, 2024 3:06 PM To: microsoft/azure-pipelines-tasks ***@***.***> Cc: Tomáš Jetelina ***@***.***>; Comment ***@***.***> Subject: Re: [microsoft/azure-pipelines-tasks] Azure Service Connection using certificate authentication fails to get cert file password on private agent (Issue #17581) Sure. I created a script file called downgrade-powershell.cmd and placed it in a scripts directory, and then added this step early in each pipeline: - task: ***@***.*** displayName: Downgrade to Powershell 7.2 inputs: failOnStderr: true workingDirectory: $(Agent.BuildDirectory)/scripts script: downgrade-powershell.cmd the contents of downgrade-powershell.cmd are set "extractPath=C:\Program Files\PowerShell\7" curl -sLO https://github.com/PowerShell/PowerShell/releases/download/v7.2.17/PowerShell-7.2.17-win-x64.zip RMDIR "%extractPath%" /S /Q 7z x PowerShell-7.2.17-win-x64.zip -o"%extractPath%" pwsh --version My pipelines almost all run on Windows, so I'm able to get away with just using a CmdTask and a batch file. If you've got a more heterogeneous environment, you might need to also add tasks with the macOS and Linux downgrade scripts they provided. You should be able to pick which one to run with something like this: - ${{ if eq(Agent.OS, 'Windows_NT') }}: - script: windows-downgrade-powershell.cmd - ${{ elseif eq(Agent.OS, 'Darwin') }}: - script: macos-downgrade-powershell.sh - ${{ elseif eq(Agent.OS, 'Linux') }}: - script: linux-downgrade-powershell.sh (disclaimer: not tested) — Reply to this email directly, view it on GitHub<#17581 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHI3LVGBRVGGCEZI6H6PQZDYROONHAVCNFSM6AAAAAATVRCPI2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRRGQYDMNRQGE>. You are receiving this because you commented.Message ID: ***@***.******@***.***>>

[lenapera]

We have the same issue in our pipelines with the Azure DevOps hosted agents. This affects productive deployments and downgrading isn't a permanent solution.

[quality-leftovers]

Can you please provide an update on this?

[AngleOSaxon]

I have no updates to provide. I've created GH-19558, a pull request with a change that I believe solves the problem. It needs to be reviewed by someone at Microsoft, but I do not have a way to cause that to happen.

[AngleOSaxon]

It looks as though a change just went in that will resolve the issue. Hopefully a new version incorporating it will be released in the next month or so.

[AngleOSaxon]

My pipeline agents just updated to version 3.238.0, which incorporates the fix for this bug. I was able to successfully run pipelines using certificate authentication with Powershell 7.4, so I believe this issue is resolved.

If your pipelines are still encountering this issue, make sure they're updated to the latest version.

[andreascaesar]

Is it only me or is this still a problem even in 7.5.0?
I'm getting the exact same error on two different build agents, had to go back to powershell 7.2.24. Don't know how long this old version will work with everything else in our pipeline...

Using the latest agent 4.248.1
Using powershell 7.5.0

==============================================================================
Task         : Azure PowerShell
Description  : Run a PowerShell script within an Azure environment
Version      : 5.234.0
Author       : Microsoft Corporation
Help         : https://aka.ms/azurepowershelltroubleshooting
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\agent\_work\_temp\38e24e74-3e66-4d72-aba3-32beff9583c8.ps1'"
Added TLS 1.2 in session.
Import-Module -Name C:\Users\*****\Documents\PowerShell\Modules\Az.Accounts\4.0.1\Az.Accounts.psd1 -Global
Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
Clear-AzContext -Scope Process
Clear-AzConfig -DefaultSubscriptionForLogin
"D:\agent\_work\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.234.0\ps_modules\VstsAzureHelpers_\openssl\openssl.exe" pkcs12 -export -in D:\agent\_work\_temp\clientcertificate.pem -out D:\agent\_work\_temp\clientcertificate.pfx -password file:"D:\agent\_work\_temp\clientcertificatepassword.txt"
Can't open file "D:\agent\_work\_temp\clientcertificatepassword.txt"
Error getting passwords
##[error]Process 'openssl.exe' exited with code '1'.
##[error]There was an error with the service principal used for the deployment.
##[error]PowerShell exited with code '1'.
Added TLS 1.2 in session.

[AngleOSaxon]

I haven't had this issue reoccur so far, although I'm still on the 3.x series of agent versions instead of 4.x.

The one thing that sticks out to me is the task version listed in that output--it's version 5.234.0. That seems somewhat behind--my AzurePowershellV5 tasks are reporting a version of 5.248.3. I believe the fix for this issue was applied in version 5.237.0.

Unfortunately I'm not sure how the Agent fetches its copies of the tasks, so I don't know why you would still have such an old task version even with one of the latest agent version.

[andreascaesar]

I'm pretty sure that tasks like this only are upgraded from the DevOps server, and we are using On Premise DevOps 2022 Update 2 (and that is unfortunately the latest version).
The tasks are almost never updated in On Premise version and I cannot find information on how to upgrade task manually...

So I believe we just have to wait until the next big DevOps release, since it is called 2022 it should be anytime I guess:)...

But if anyone finds information on how to upgrade tasks manually in DevOps On Premise I would be very happy!

[AngleOSaxon]

I haven't tried it personally, but they're largely just Powershell/node scripts sitting in <agent directory>/_work/_tasks/<task name/guid>/<version>. You could theoretically download the latest version of the tasks and their support files and put them in place in the relevant directories. You may have to manually sort out dependency issues, since they have some shared components like the openssl binary.

The lightest-touch manual fix I can think of would be to locate the AzurePowerShell task directory for your task version and update the file Utility.ps1 inside ps_modules/VstsAzureHelpers_. In my notes from two years ago, that was at <agent directory>\_work\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.209.0\ps_modules\VstsAzureHelpers_\Utility.ps1. In your case the task number would be 5.234.0 instead of 5.209.0, and I have no idea if that GUID would be consistent.

If you can find that file, there's a function ConvertTo-Pfx that sets a variable called $openSSLArgs. Change that line from $openSSLArgs = "pkcs12 -export -in $pemFilePath -out $pfxFilePath -password file:`"$pfxPasswordFilePath`"" to $openSSLArgs = "pkcs12 -export -in $pemFilePath -out $pfxFilePath -password file:$pfxPasswordFilePath". Save the file and restart the agent service.

It's not my first choice for a lot of reasons, but it should work until the tasks get updated. You'll lose the change when that happens, but any update should contain the fix anyway.

[andreascaesar]

I actually tried adding the latest version within the same structure (with new version folder), but it did not pick up the new version...
I didn't dare replacing all files in the current version...

Anyway, I did the replace in the file instead and it works like a charm!

Thanks!!!