Unable to clone gitlab private GitRepository with libgit2

[jjlakis]

Hello.
I have a GitRepository poitning to a private repo with private keys (with write permissions) included:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: cnm-dep
spec:
  interval: 24h
  url: ssh://git@my.gitlab/jj/cnm-dep.git
  secretRef:
    name: gitlab-key
  ref:
    branch: master

This works as expected, reconciles itself, no problem with updating corresponding Kustomization resource. However, image automation controller is unable to clone this repository when ImagePolicy is met. ImageUpdateAutomation resource is the following:

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: cnm-dep
spec:
  interval: 30s
  sourceRef:
    kind: GitRepository
    name: cnm-dep
  git:
    checkout:
      ref:
        branch: master
    commit:
      author:
        email: ml3k@interia.pl
        name: fluxcd
      messageTemplate: |
        (FluxCD) Update images
        {{range .Updated.Images}}
        - {{ . }}
        {{end}}
    push:
      branch: master
  update:
    path: ./
    strategy: Setters

Logs from image-automation-controller:

{"level":"error","ts":"2021-08-13T12:49:40.933Z","logger":"controller-runtime.manager.controller.imageupdateautomation","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"cnm-dep","namespace":"cnm","error":"unable to clone 'ssh://git@my.gitlab/jj/cnm-dep.git', error: Certificate"}

Identical configuration works as expected for github private repositories in the same cluster. Version i run is ghcr.io/fluxcd/image-automation-controller:v0.14.0.

Thank you in advance.

[squaremo]

This is pretty baffling -- image-automation-controller uses substantially the same code as source-controller for cloning the repo. The only difference I can see is that source-controller assigns auth to a local:

	// determine auth method
	auth := &git.Auth{}
	if repository.Spec.SecretRef != nil {
		authStrategy, err := strategy.AuthSecretStrategyForURL(
			repository.Spec.URL,
			git.CheckoutOptions{
				GitImplementation: repository.Spec.GitImplementation,
				RecurseSubmodules: repository.Spec.RecurseSubmodules,
			})
		if err != nil {
			return sourcev1.GitRepositoryNotReady(repository, sourcev1.AuthenticationFailedReason, err.Error()), err
		}

		name := types.NamespacedName{
			Namespace: repository.GetNamespace(),
			Name:      repository.Spec.SecretRef.Name,
		}

		var secret corev1.Secret
		err = r.Client.Get(ctx, name, &secret)
		if err != nil {
			err = fmt.Errorf("auth secret error: %w", err)
			return sourcev1.GitRepositoryNotReady(repository, sourcev1.AuthenticationFailedReason, err.Error()), err
		}

		auth, err = authStrategy.Method(secret)
		if err != nil {
			err = fmt.Errorf("auth error: %w", err)
			return sourcev1.GitRepositoryNotReady(repository, sourcev1.AuthenticationFailedReason, err.Error()), err
		}
	}

	checkoutStrategy, err := strategy.CheckoutStrategyForRef(
		repository.Spec.Reference,
		git.CheckoutOptions{
			GitImplementation: repository.Spec.GitImplementation,
			RecurseSubmodules: repository.Spec.RecurseSubmodules,
		},
	)
	if err != nil {
		return sourcev1.GitRepositoryNotReady(repository, sourcev1.GitOperationFailedReason, err.Error()), err
	}

	gitCtx, cancel := context.WithTimeout(ctx, repository.Spec.Timeout.Duration)
	defer cancel()

	commit, revision, err := checkoutStrategy.Checkout(gitCtx, tmpGit, repository.Spec.URL, auth)
	if err != nil {
		return sourcev1.GitRepositoryNotReady(repository, sourcev1.GitOperationFailedReason, err.Error()), err
	}

while image-automation-controller assigns into a struct, then uses the value from the struct:

type repoAccess struct {
	auth *git.Auth
	url  string
}

func (r *ImageUpdateAutomationReconciler) getRepoAccess(ctx context.Context, repository *sourcev1.GitRepository) (repoAccess, error) {
	var access repoAccess
	access.auth = &git.Auth{}
	access.url = repository.Spec.URL

	authStrat, err := gitstrat.AuthSecretStrategyForURL(access.url, git.CheckoutOptions{GitImplementation: sourcev1.LibGit2Implementation})
	if err != nil {
		return access, err
	}

	if repository.Spec.SecretRef != nil && authStrat != nil {

		name := types.NamespacedName{
			Namespace: repository.GetNamespace(),
			Name:      repository.Spec.SecretRef.Name,
		}

		var secret corev1.Secret
		err = r.Client.Get(ctx, name, &secret)
		if err != nil {
			err = fmt.Errorf("auth secret error: %w", err)
			return access, err
		}

		access.auth, err = authStrat.Method(secret)
		if err != nil {
			err = fmt.Errorf("auth error: %w", err)
			return access, err
		}
	}
	return access, nil
}

func (r repoAccess) remoteCallbacks() libgit2.RemoteCallbacks {
	return libgit2.RemoteCallbacks{
		CertificateCheckCallback: r.auth.CertCallback,
		CredentialsCallback:      r.auth.CredCallback,
	}
}

// cloneInto clones the upstream repository at the `ref` given (which
// can be `nil`). It returns a `*gogit.Repository` since that is used
// for committing changes.
func cloneInto(ctx context.Context, access repoAccess, ref *sourcev1.GitRepositoryRef, path string) (*gogit.Repository, error) {
	checkoutStrat, err := gitstrat.CheckoutStrategyForRef(ref, git.CheckoutOptions{GitImplementation: sourcev1.LibGit2Implementation})
	if err == nil {
		_, _, err = checkoutStrat.Checkout(ctx, path, access.url, access.auth)
	}
	if err != nil {
		return nil, err
	}

	return gogit.PlainOpen(path)
}

[squaremo]

The only difference [between image-automation-controller and source-controller, when cloning]

Well that's barring the difference that image-automation-controller always uses libgit2, rather than the default go-git. @jjlakis If you set the GitRepository object's .spec.gitImplementation field to libgit2, does source-controller still succeed?

[jjlakis]

@squaremo Good point, source-controller indeed fails with gitImplementation set to libgit2, same error.

{"level":"error","ts":"2021-08-21T16:27:37.925Z","logger":"controller.gitrepository","msg":"Reconciler error","reconciler group":"source.toolkit.fluxcd.io","reconciler kind":"GitRepository","name":"cnm-dep","namespace":"cnm","error":"unable to clone 'ssh://git@gl.lakis.io/jj/cnm-dep.git', error: Certificate"}

Should I consider the Git server to be a problem then maybe? Ideally image automation controller uses the git implementation from GitRepository resource. Thanks.

[squaremo]

Ideally image automation controller uses the git implementation from GitRepository resource.

I switched to using libgit2 regardless of the GitRepository object .spec.gitImplementation in fluxcd/image-automation-controller#177. This was because the go-git code in source-controller/pkg unconditionally does a shallow clone, and this caused problems for the branching (see fluxcd/image-automation-controller#164 (comment) and the comments following). Another way to fix it would be to alter the code in source-controller/pkg so that it can optionally not do a shallow clone when using go-git. But there would still be reasons to specify libgit2 as the implementation, so it's still useful to get to the bottom of this issue.

[squaremo]

I'm going to move this to source-controller, since it fails there too (and image-automation-controller uses that code).

[squaremo]

A bit of info that might help: @jjlakis Can you tell us what data fields are in the secret you use for the GitRepository? (not the contents obviously! just which fields have values, in other words, the keys)

[jjlakis]

@squaremo Here's the gitlab-key secret I use with gitRepository:

data:           
  identity: private_key
  identity.pub: public_key
  known_hosts: known_hosts

I just realized that I didn't try http user/token access to GitLab. Not sure if this would make any difference though

[squaremo]

In https://github.com/fluxcd/source-controller/blob/main/pkg/git/libgit2/transport.go#L173, error: Certificate is returned when it can't find a match for the host in known_hosts. Is it possible that there's no entry for the private gitlab host in known_hosts, @jjlakis? (Or if there is, maybe it isn't being recognised by the matching code; if you post what you think is a match, I can confirm that ..)

[hiddeco]

@jjlakis if your known_hosts has an ECDSA* entry, the solution is to replace this with an entry of another type, as the current version of libgit2 we make use of does not have support for ECDSA* types. Support for this will be added once we have a working libgit2 version >=1.2.0.

For other folks looking for authentication related issues: the latest release of the image-automation-controller (v0.15.0) contains libgit2 linked against OpenSSL and LibSSH2, which based on my research and extensive testing, should solve most issues around private key formats (not host keys).

[ilya-git]

I have installed latest flux with v0.15.0 image-automation-controller and get this error: unable to clone 'ssh://bitbucket.org/<myrepo>', error: callback returned unsupported credentials type. This is my known_hosts entry:

bitbucket.org ssh-rsa AAAA...

Is there any workaround? I have upgraded to the latest flux and image automation stopped working after this...

Switching GitRepository to libgit2 makes it throw the same error