Azure DevOps: SSH-RSA is deprecated 💥

[uidmehdi]

Describe the bug

I'm seeing errors stating that source reconciliation is failing with the following error:

failed to checkout and determine revision: unable to list remote for 'ssh://git@ssh.dev.azure.com/v3/<organization_name>/<project_name>/<repo_name>': unknown error: remote: Command git-upload-pack: You’re using ssh-rsa that is about to be deprecated and your request has been blocked intentionally. Any SSH session using SSH-RSA is subject to brown out (failure during random time periods). Please use rsa-sha2-256 or rsa-sha2-512 instead. For more details see https://aka.ms/ado-ssh-rsa-deprecation.

Steps to reproduce

1. Install Flux
2. Create source flux create source git flux-system --url=ssh://git@ssh.dev.azure.com/v3/<organization_name>/<project_name>/<repo_name> --branch=main --ssh-key-algorithm=rsa --ssh-rsa-bits=4096 --interval=1m
3. Add the provided deploy key in Azure DevOps SSH Public Keys
4. Wait for the brown out periods of Azure DevOps repos

Expected behavior

Source reconciliation should succeed

Screenshots and recordings

No response

OS / Distro

Kubernetes

Flux version

v2.2.3

Flux check

► checking prerequisites
✔ Kubernetes 1.27.9 >=1.26.0-0
► checking version in cluster
✔ distribution: flux-v2.2.3
✔ bootstrapped: true
► checking controllers
✔ helm-controller: deployment ready
► digprvtstgwecr.azurecr.io/fluxcd/helm-controller:v0.37.4
✔ image-automation-controller: deployment ready
► digprvtstgwecr.azurecr.io/fluxcd/image-automation-controller:v0.37.1
✔ image-reflector-controller: deployment ready
► digprvtstgwecr.azurecr.io/fluxcd/image-reflector-controller:v0.31.2
✔ kustomize-controller: deployment ready
► digprvtstgwecr.azurecr.io/fluxcd/kustomize-controller:v1.2.2
✔ notification-controller: deployment ready
► digprvtstgwecr.azurecr.io/fluxcd/notification-controller:v1.2.4
✔ source-controller: deployment ready
► digprvtstgwecr.azurecr.io/fluxcd/source-controller:v1.2.4
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta2
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider

Azure DevOps

Container Registry provider

No response

Additional context

I have also tried to create a new ssh key ssh-keygen -t rsa-sha2-256 and a secret using flux create secret git flux-system -n flux-system --private-key-file=<path-to-generated-private-key> --url=ssh://git@ssh.dev.azure.com/v3/<organization_name>/<project_name>/<repo_name> and referencing it in the source manifest by I get similar error

Code of Conduct

• I agree to follow this project's Code of Conduct

[stefanprodan]

Not sure how is this a Flux issue if SHA2 keys get the same error, please contact Azure support.

[psdonas]

@uidmehdi @stefanprodan. We had the same issue after the changes Microsoft announced for ssh-rsa deprecation. After adding a rsa-sha2-512 key last week, we thought the issue was resolved. However, this morning started happening again.
Based on the comment here https://devblogs.microsoft.com/devops/ssh-rsa-deprecation/#comment-4327, we added this to the source controller. However, debugging this is extremely difficult since there is no way to check if this patch really fixes the issue. We won't know if it worked due to the brow out procedure implemented by Microsoft. We are just waiting to start crashing again.

- patch: | - op: add path: /spec/template/spec/containers/0/args/- value: --ssh-hostkey-algos=rsa-sha2-512 target: kind: Deployment name: "(source-controller)"

[stefanprodan]

To migrate off RSA-SHA a Flux bootstrapped cluster with Azure DevOps there are two options:

• Migrate to RSA SHA-2
• Migrate from SSH to HTTP/S

If someone with access to Azure DevOps can test the following procedures it would be great.

Migrate to RSA SHA-2

Assuming you have bootstrap Flux with a SSH key as described in https://fluxcd.io/flux/installation/bootstrap/azure-devops/

Configure RSA SHA-2 SSH Keys

Generate a RSA SHA-2 public private key pair with:

ssh-keygen -t rsa-sha2-512

Upload the public key to Azure DevOps.

Rotate the private key in the flux-system secret with:

kubectl -n flux-system delete secret flux-system

flux -n flux-system create secret git flux-system \
--url=sssh://git@ssh.dev.azure.com/v3/<org-name>/<project-name>/<repo-name> \
--private-key-file=<path-to-private-key> \
--password=<key-passphrase>

Configure RSA SHA-2 Host Keys

Clone the Azure DevOps repository locally and add the following patch to the clusters/<your-cluster>/flux-system/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
patches:
  - patch: |
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --ssh-hostkey-algos=rsa-sha2-512,rsa-sha2-256      
    target:
      kind: Deployment
      name: (source-controller|image-automation-controller)

Note

Even if Azure DevOps advertises the host key as ssh-rsa the server offeres: ssh-rsa, rsa-sha2-256 and rsa-sha2-512. See #4726 (comment)

Commit and push change to the Azure DevOps repository.

Force apply changes

If Flux is currently failing to pull the changes from Azure DevOps, you can apply the manifests directly on the cluster with:

kubectl apply -k clusters/<your-cluster>/flux-system

Wait for source-controller to restart, then test the new keys with:

flux reconcile source git flux-system

Migrate from SSH to HTTP/S

Generate an personal access token in the Azure portal.

Delete the flux-system secret that contains the SSH key:

kubectl -n flux-system delete secret flux-system

Finally run bootstrap using the PAT and the HTTP/S address of the repo:

echo <DEVOPS-PAT> | flux bootstrap git \
  --token-auth=true \
  --url=https://dev.azure.com/<org>/<project>/_git/<repository> \
  --branch=main \
  --path=clusters/<your-cluster-name>

[psdonas]

Thanks @stefanprodan . That's exactly how I deployed the key last week and earlier today I configured the kustomization.yaml file same as above. Is there a way to see what algorithm the source controller pod is using? Otherwise, we will have to wait to start erroring out again.

[stefanprodan]

Well looks like Azure DevOps advertises a single host key:

$ ssh-keyscan ssh.dev.azure.com
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
ssh.dev.azure.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

Testing the key for RSA-SHA2 in fluxcd/pkg/ssh works:

func TestAzureDevOps(t *testing.T) {
	g := NewWithT(t)

	knownHosts, err := ScanHostKey("ssh.dev.azure.com:22", time.Second, []string{"rsa-sha2-256", "rsa-sha2-512"}, false)
	t.Log(string(knownHosts))
	g.Expect(err).NotTo(HaveOccurred())
}

@psdonas after you've set the SSH private key to rsa-sha2-512 in cluster, did you got the same error?

[psdonas]

@stefanprodan yes, I got the same error after deploying the rsa-sha2-512 key in cluster.

[stefanprodan]

If with a rsa-sha2-512 key still errors out and with Azure DevOps advertising a single rsa-sha host key, this feels like a major issue on their side, there is nothing we can do in Flux to fix this.

In any case, there is always the option to switch from SSH to HTTP/S and Azure PAT like so:

kubectl -n flux-system delete secret flux-system

echo <DEVOPS-PAT> | flux bootstrap git \
  --token-auth=true \
  --url=https://dev.azure.com/<org>/<project>/_git/<repository> \
  --branch=main \
  --path=clusters/my-cluster

[uidmehdi]

@stefanprodan @psdonas, Thanks for providing more details and testing. I've got errors as well during the brown out session yesterday although rsa-sha2-512 keys were deployed. Just applied the patch and waiting for the next brown out session.

[bondido]

Well looks like Azure DevOps advertises a single host key:

$ ssh-keyscan ssh.dev.azure.com
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
ssh.dev.azure.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H

@stefanprodan , isn't it that ssh-rsa in host key doesn't imply specific signature algorithm?
ssh-rsa only means "it's RSA key" but algorithm can be as well rsa-sha2-256, rsa-sha2-512 ..or ssh-rsa. Correct me please if I'm wrong.

Doesn't flux assume it should use ssh-rsa algorithm based on the host key containing ssh-rsa field ?

[stefanprodan]

@bondido as posted here #4726 (comment) the Flux SSH host key scanner finds all 3 algos. The patch posted here #4726 (comment) configures Flux source-controller to prioritise SHA-2 over SHA-1 and hopefully this will solve the issue with Azure blocking Flux. Please test and confirm if it works for you when your cluster enters the brown out session.

[bondido]

Thanks @stefanprodan
As it's painful to test it because you never know if the fix really worked (ofc MS is the only one to blame here), we decided to migrate to HTTPS with PAT.

[dipti-pai]

Validated that ssh works with Azure dev ops repos if ssh-keygen is used to pre-create ssh keys and use with flux.

[stefanprodan]

@dipti-pai can you confirm that there is no need for --ssh-hostkey-algos=rsa-sha2-512,rsa-sha2-256, using a rsa-sha2-512 is enough?

[uidmehdi]

A brownout session happened again and I believe the patch is needed. Make sure to apply it for image-automation-controller as well.

[psdonas]

@uidmehdi is right. I received an error during the brown out period last night too, but only for image-automation-controller. Source controller was configured with the patch and did not complain. Error from image-automation-controller below:
imageupdateautomation/flux-system.flux-system unable to clone 'ssh://git@ssh.dev.azure.com/some/repo/am.devops.deploy': unknown error: remote: Public key authentication failed.
If you add it to both places you should be fine.
Thank you @stefanprodan

[dipti-pai]

@dipti-pai can you confirm that there is no need for --ssh-hostkey-algos=rsa-sha2-512,rsa-sha2-256, using a rsa-sha2-512 is enough?

@stefanprodan "--ssh-hostkey-algos=rsa-sha2-512,rsa-sha2-256" is needed.

[stefanprodan]

Thanks everyone for testing the fix. We've updated the procedure on how to bootstrap Azure DevOps via SSH here: https://fluxcd.io/flux/installation/bootstrap/azure-devops/#bootstrap-using-ssh-keys

[cveld]

I believe an rsa key can be used for any of the three signatures. It is really the client that needs to select one of those. Can anybody confirm that it is not required to regenerate an rsa key pair?

[rouke-broersma]

I believe an rsa key can be used for any of the three signatures. It is really the client that needs to select one of those. Can anybody confirm that it is not required to regenerate an rsa key pair?

It's in line with the ssh-keygen -t docs I think. They specify that the sha2 signature specification applies to generating certificates signed by a ca (-s flag). Normal openssh ppk pairs are raw keys and not certificates so they don't contain the signature info. This is added in-flight during key negotiations and so the signature algo is chosen from the host key algorithms.

This is how I understand how it works after working on this since the brownouts started however I'm not an openssh expert so if anyone is feel free to chime in and correct me.

[adhoc-am]

Hi,
The same issue with ADO git
flux version 2.2.2

    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    resources:
      - gotk-components.yaml
      - gotk-sync.yaml
    patches:
      - target:
          kind: Deployment
          name: (source-controller|image-automation-controller)
        patch: |-
          - op: add
            path: /spec/template/spec/containers/0/args/-
            value: --ssh-hostkey-algos=rsa-sha2-512,rsa-sha2-256

flux managed by Terraform. Several days ago if run terraform for flux there is the error during terraform plan:

ssh key generated by cmd:

ssh-keygen -t rsa-sha2-512 -b 4096 -N '' -P ''

[stefanprodan]

@adhoc-am see fluxcd/terraform-provider-flux#679

[adhoc-am]

@stefanprodan thank you for the reference, but I haven't caught what should be added in a terraform configuration of flux bootstrap resource? e.g. below what is configured now. As you can see the patch is added:

resource "flux_bootstrap_git" "this" {

  version                 = "v2.2.2"
  disable_secret_creation = true
  ...
  kustomization_override  = <<EOT
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    resources:
      - gotk-components.yaml
      - gotk-sync.yaml
    patches:
      - target:
          kind: Deployment
          name: (source-controller|image-automation-controller)
        patch: |-
          - op: add
            path: /spec/template/spec/containers/0/args/-
            value: --ssh-hostkey-algos=rsa-sha2-512,rsa-sha2-256
EOT
}

[stefanprodan]

@adhoc-am after we release Flux v2.3, you'll be able to set the host key algos in the provider like so:

provider "flux" {
        ssh = {
          hostkey_algos = ["rsa-sha2-512", "rsa-sha2-256"]
        }
}

[adhoc-am]

@stefanprodan
i'm wondering what is the expected date of release v2.3 ? do we have to wait the terraform flux provider new version as well or can use the latest one 1.2.3 ?
we've recently migrated several clusters to flux v2 and have more clusters in our schedule, so, this issue is really headache for us now =((
Therefore, it will be very appreciated if we can have the ETA of final solution for fluxv2 - terraform - ADO git case.

Thank you and BR!

[stefanprodan]

@adhoc-am see #4712