Service Principal Authentication #747
Conversation
Adding the JFrog V2 service connections. I just copy+pasted the existing Artifactory resource, tests & docs then adjusted accordingly. They all use the same parameters. One unit test needed fixing, the argocd connection seemed to be based on these so I adopted the same fix that was in that resource. Unit & acceptance tests for the service connections have passed. Docs have been updated. The formatting scripts in the repo want to reformat all sorts of stuff. I'll leave that to Microsoft's discretion when I upstream this.
This provider stores secret values in the state file by hashing them with bcrypt. bcrypt only uses the first 72 characters for the hash which results in issues with Artifactory JWTs because the first 72 characters of those tokens contain data about the algorithms and subjects which don't change when they're regenerated. The recommendation from OWASP is to use Argon2 as the current standard for password hashing. I've adopted that for our fork of the provider in this PR. I have no inclination of what Microsoft might ultimately do to resolve the issue upstream but worst case our secrets will get re-pushed to Azure DevOps when upstream resolves this issue and we either merge it in or use the upstream provider again.
* Created service end point for NuGet * Create documentation * Minor fixes * Modelled along the lines of jfrog_artifactory_v2 * Fix doc as per JFrog V2 * Added the third variant * Fix implementation * Fix nugetkey * Cleanup * Update azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_nuget.go Co-authored-by: David Corrigan <47994662+davidcorrigan714@users.noreply.github.com> * Update azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_nuget.go Co-authored-by: David Corrigan <47994662+davidcorrigan714@users.noreply.github.com> * Update azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_nuget.go Co-authored-by: David Corrigan <47994662+davidcorrigan714@users.noreply.github.com> * Update azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_nuget.go Co-authored-by: David Corrigan <47994662+davidcorrigan714@users.noreply.github.com> * Update azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_nuget.go Co-authored-by: David Corrigan <47994662+davidcorrigan714@users.noreply.github.com> * Update azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_nuget.go Co-authored-by: David Corrigan <47994662+davidcorrigan714@users.noreply.github.com> * Update azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_nuget.go Co-authored-by: David Corrigan <47994662+davidcorrigan714@users.noreply.github.com> * Update azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_nuget.go Co-authored-by: David Corrigan <47994662+davidcorrigan714@users.noreply.github.com> * Update azuredevops/internal/service/serviceendpoint/resource_serviceendpoint_nuget.go Co-authored-by: David Corrigan <47994662+davidcorrigan714@users.noreply.github.com> * Put fixes after addressing review comments --------- Co-authored-by: David Corrigan <47994662+davidcorrigan714@users.noreply.github.com>
Minor data handling fixes, affects importing and some picky unit tests.
davidcorrigan714
force-pushed
the
users/dcorriga/upstream-sp-auth
branch
from
3ceefae to
f45caa9
Compare
* Service Principal Support
* remove hashing shenanigans
Handle padded & unpadded base64 from Terraform tokens
|
@davidcorrigan714 can you help rebase to the latest code, we upgrade the SDK from v6 to v7,. |
azuredevops/provider.go
Outdated
| return nil, diag.FromErr(err) | ||
| } | ||
|
|
||
| client, err := client.GetAzdoClient(token, d.Get("org_service_url").(string), terraformVersion) |
There was a problem hiding this comment.
The call to GetAzdoClient will call NewPatConnection, what in turn will create a Basic Auth HTTP header using an OAuth token, what eventually will fail the authentication, because OAuth tokens must be passed as a Bearer Auth Header.
An even bigger issue is here that the Azure DevOps GO API uses a struct instead of an interface to pass a Connection around to the various REST API endpoint implementations and the Connection struct only supports a static auth header value. This is sufficient for a long living PAT but far from ideal for short living OAuth tokens. As long as this issue isn't fixed in the GO API is don't see a reliable way to support Service Principal Authentication for Azure DevOps.
@xuzhang3 any infos who's actually in charge for the GO API? Or is it abandoned again?
There was a problem hiding this comment.
That's a good point, I thought the OAuth tokens were valid long enough that it wouldn't practically be a concern. Looking at your other comment on the other thread I can test that and put it in a PR up to the Go SDK if someone is available to review it.
There was a problem hiding this comment.
@davidcorrigan714 The issue with the Azure DevOps GO SDK ist, that it's generated using a framework not created manually. So that's why most PRs got rejected there in past, because issues have to be fixed in the framework (which is not available here on GitHub) and not in the generated code in the repo you see. Of course, the fact that the AzDO team maintains the GO SDK so carelessly makes things even worse. This is one of the main reasons why I personally stopped contributing to the project and in addition to me I think Microsoft focuses on GitHub and slowly let AzDO die. That AzDO has much more stability issues compared to GitHub is a another good indicator that the manpower behind AzDO is very limited.
There was a problem hiding this comment.
Oh I'm well aware of the code-gen happening in all the SDKs. Pretty sure they completely lost or broke the Python SDK generator tooling a few years back, or lost the expertise that was running it. connection.go just doesn't show any history of ever being a generated file like the rest of the SDK is.
Don't want to derail this into GH vs AzDO too much. The reality is they don't have feature parity on the roadmap for the two platforms in certain areas and the rate they've been improving since trying to pitch GH as an AzDO alternative still puts it at least 5+ years out as a viable alternative for most of our use cases. We also do some "interesting" things with Pipelines that GH may never support in Actions.
There was a problem hiding this comment.
Any update? I would say ship a first version without refresh token support 😊
There was a problem hiding this comment.
The Azure DevOps team doesn't really have go developers, Azure DevOps is built in .NET, PowerShell and a bit of NodeJS and some web front-end stuff. The fact they even have a Go SDK amazes me, the team generates the Go and the Node SDK from the .NET API metadata as far as I know.
I don't think they maintain these libraries "carelessly", it's just that the person who wrote it, probably isn't available at the moment.
It's one of the big pains of building SDKs at all, you tend to need to build them in a plethora of languages, Python, .NET, JS, Go, Ruby etc, but the team that builds the SDK doesn't normally work with any of thee languages other than .NET. I see the same issue with many other frameworks and SDKs, the best SDK is written in for the language of the tool itself.
There was a problem hiding this comment.
The team maintaining TF az* providers is a central team.
While you wait, consider this workaround that uses the external datasource to acquire a token with the Azure CLI:
data external azdo_token {
program = [
"az", "account", "get-access-token",
"--resource", "499b84ac-1321-427f-aa17-267ca6975798", # Azure DevOps
"--query","{accessToken:accessToken}",
"-o","json"
]
}
provider azuredevops {
org_service_url = local.azdo_organization_url
personal_access_token = data.external.azdo_token.result.accessToken
}There was a problem hiding this comment.
@jessehouwing yeah the choice of Go for Terraform always makes me a bit sad, it's the only reason I've picked up Go and I'm certainly not amazing at it though the Terraform stuff usually has plenty of examples to draw from.
Catching up on this thread again now and seeing where things are at with the code.
|
@davidcorrigan714 can you help resolve the conflict? |
|
I am very interested in the final implementation of this functionality, as it would enable the use of this functionality for authentication through Managed Identities, on the road to password-less processes. |
|
Got most of the feedback incorporated with refreshing the access token as needed and aligning with the other Azure terraform provider variables where possible. Going to do a bit of manual testing tomorrow for the various cloud identity scenarios - GH, Managed Identities & HCP Workload Identities. The changes to the Azure DevOps Go API are currently hacked into it in the vendor folder so I need to do a PR there before this goes in. |
|
Maybe @xuzhang3 can provide some insights on what's failing in the automated checks. Wouldn't surprise me if I missed a format or package checks but can't see the logs ☹ |
@davidcorrigan714 The pipe fails due to the line break in the vendor files. I usually add the git config |
|
Can anyone check and fix the build so we can get this merged? Thanks! |
|
@davidcorrigan714 Looks to me like you have a lot of great work in here! Do you think you have any time to take this through the home stretch? I think this would be very useful for a lot of people and orgs! 🙌 |
| ```hcl | ||
| terraform { | ||
| required_providers { | ||
| azapi = { |
There was a problem hiding this comment.
should be azuredevops not azapi
| ```hcl | ||
| terraform { | ||
| required_providers { | ||
| azapi = { |
There was a problem hiding this comment.
should be azuredevops not azapi
|
|
||
| client_id = "00000000-0000-0000-0000-000000000001" | ||
| tenant_id = "00000000-0000-0000-0000-000000000001" | ||
| oidc_hcp = true |
There was a problem hiding this comment.
oidc_hcp is not supported
| client_id_apply = "00000000-0000-0000-0000-000000000001" | ||
| tenant_id_plan = "00000000-0000-0000-0000-000000000001" | ||
| tenant_id_apply = "00000000-0000-0000-0000-000000000001" | ||
| oidc_hcp = true |
There was a problem hiding this comment.
oidc_hcp is not supported
|
|
||
| The OIDC service principal authentication methods allow for secure passwordless authentication from [Terraform Cloud](https://developer.hashicorp.com/terraform/cloud-docs/workspaces/dynamic-provider-credentials) & [GitHub Actions](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect). | ||
|
|
||
| * [Authenticating to a Service Principal with a Terraform Cloud Workload Identity Token](guides/authenticating_service_principal_using_hcp_token.html) |
There was a problem hiding this comment.
Documention authenticating_service_principal_using_hcp_token.html does not exist，should be the manage identity document
| - `oidc_github_actions_audience` - Custom audience for the GitHub Actions OIDC token. | ||
| It can also be sourced from the `ARM_OIDC_GITHUB_ACTIONS_AUDIENCE` environment variable. | ||
|
|
||
| - `oidc_hcp` - Boolean, set to true to use the Terraform Cloud OIDC workload identity token to authenticate to a service principal. |
|
@davidcorrigan714 if you do not have time to update this PR, can we create another PR based on your changes? |
|
@davidcorrigan714 close this PR via #970 |
|
@xuzhang3 appreciate you all finishing this up and getting it in. Life has been too darn busy and still is. |
All Submissions:
Allows the terraform provider to authenticate to azure devops through a service principal using a variety of authentication methods: client secret, client certificate, GitHub Actions OIDC tokens, Terraform Cloud OIDC tokens, & any other OIDC token provided directly as an attribute
Issue Number: #745
Does this introduce a change to
go.mod,go.sumorvendor/?Adds the Azure Go SDK & some crypto functions.
Does this introduce a breaking change?