[Bug]: flux_bootstrap_git fails during Azure DevOps SSH SHA1 brownouts

[craighurt]

Describe the bug

During Azure Devops SHA1 brownouts, Terraform fails during the plan phase when reading the flux_bootstrap_git resource. This is despite the SSH key being in SHA2-512 format and being accepted by Terraform to git source code during terraform init.

Outside of brownout periods, the same code executes successfully.

We have implemented the controller argument updates (ie. --ssh-hostkey-algos=rsa-sha2-512,rsa-sha2-256) which resolves problems once Flux has been deployed, however, there do not appear to be any options within the provider to enable this configuration at bootstrap time.

Steps to reproduce

Note: It is only possible to reproduce the error during SSH brownout periods.

• Any SSH-based plan/refresh using the flux_bootstrap_git resource will produce the errors shown below.
• Keys are confirmed to be SHA2-512 format
• The same key works with the controllers during brownout periods when using --ssh-hostkey-algos arguments with the source and image-automation controllers

Expected behavior

During non-brownout times, plan will succeed when refreshing state.

module.flux_bootstrap[0].flux_bootstrap_git.main: Refreshing state... [id=flux-system]

Screenshots and recordings

No response

Terraform and provider versions

Terraform v1.7.5
on linux_amd64

fluxcd/flux v1.2.3

Terraform provider configurations

Note the locals used are populated from data lookups (kubernetes) and local config file (git/ssh).

provider "flux" {
  kubernetes = {
    host                   = local.kube_provider.host
    client_certificate     = local.kube_provider.client_certificate
    client_key             = local.kube_provider.client_key
    cluster_ca_certificate = local.kube_provider.cluster_ca_certificate
  }
  git = {
    url                     = local.config.flux.repos["system"].url
    author_email            = local.flux_bootstrap_credentials.author_email
    author_name             = local.flux_bootstrap_credentials.author_name
    branch                  = local.config.flux.repos["system"].branch
    commit_message_appendix = local.flux_bootstrap_credentials.commit_message
    ssh = {
      private_key = local.flux_bootstrap_private_key
      username    = local.flux_bootstrap_credentials.ssh_username
      password    = local.flux_bootstrap_credentials.ssh_passphrase
    }
  }
}

flux_bootstrap_git resource

resource "flux_bootstrap_git" "main" {
  cluster_domain          = var.flux_properties.cluster_domain
  components              = var.flux_properties.components
  components_extra        = var.flux_properties.components_extra
  image_pull_secret       = var.flux_properties.image_pull_secret
  kustomization_override  = yamlencode(local.kustomization_override)
  log_level               = var.flux_properties.log_level
  namespace               = var.system_repo.namespace
  network_policy          = var.flux_properties.network_policy
  path                    = var.system_repo.path == null ? format("clusters/%s", var.cluster_name) : var.system_repo.path
  registry                = var.flux_properties.registry
  secret_name             = var.system_repo.secret
  interval                = var.system_repo.interval
  toleration_keys         = var.flux_properties.toleration_keys
  version                 = var.flux_properties.version
  disable_secret_creation = false
}

Flux version

v2.2.3

Additional context

The following error is produced during brownout:

Error: Git Client

with module.flux_bootstrap[0].flux_bootstrap_git.main,
on .terraform/modules/flux_bootstrap/terraform/main.tf line 2, in resource "flux_bootstrap_git" "main":
2: resource "flux_bootstrap_git" "main" {

could not clone git repository: unable to clone
'ssh://git@ssh.dev.azure.com/v3///***':
unknown error: remote: Command git-upload-pack: You’re using ssh-rsa that is
about to be deprecated and your request has been blocked intentionally. Any
SSH session using SSH-RSA is subject to brown out (failure during random time
periods). Please use rsa-sha2-256 or rsa-sha2-512 instead. For more details
see https://aka.ms/ado-ssh-rsa-deprecation.

Code of Conduct

• I agree to follow this project's Code of Conduct

Would you like to implement a fix?

None

[stefanprodan]

We have implemented the controller argument updates (ie. --ssh-hostkey-algos=rsa-sha2-512,rsa-sha2-256) which resolves problems once Flux has been deployed, however, there do not appear to be any options within the provider to enable this configuration at bootstrap time.

See the kustomization_override from this provider that allows you to make any changes to Flux manifests. Docs: https://registry.terraform.io/providers/fluxcd/flux/latest/docs/resources/bootstrap_git#customizing-flux

[craighurt]

Thanks @stefanprodan - I probably didn't explain the issue there. We have applied the kustomization_override configuration via the provider which is working as expected.

The problem we have is with the bootstrap, not the configuration of the controllers. It looks like the initial bootstrap is falling back to SHA1 instead of SHA2, this causes bootstrapping to fail. We may need a way to force the bootstrap in the same way as the source controller to ensure SHA2 is used first?