GitLab self-hosted: git clone unknown error #120

dbluxo · 2020-08-28T16:50:40Z

Hi,

I tried to bootstrap the toolkit components in a GitLab (self-hosted) repository and got the following error in the last step:

$ tk bootstrap gitlab --verbose --namespace=gitops-toolkit --owner=devops --repository=gitops-toolkit --hostname=gitlab.mydomain --ssh-hostname=ssh.gitlab.mydomain
► connecting to gitlab.mydomain
✔ repository cloned
✚ generating manifests
✔ components manifests pushed
► installing components in gitops-toolkit namespace
namespace/gitops-toolkit created
customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/providers.notification.toolkit.fluxcd.io created
customresourcedefinition.apiextensions.k8s.io/receivers.notification.toolkit.fluxcd.io created
role.rbac.authorization.k8s.io/crd-controller-gitops-toolkit created
rolebinding.rbac.authorization.k8s.io/crd-controller-gitops-toolkit created
clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-gitops-toolkit created
service/notification-controller created
service/source-controller created
service/webhook-receiver created
deployment.apps/helm-controller created
deployment.apps/kustomize-controller created
deployment.apps/notification-controller created
deployment.apps/source-controller created
networkpolicy.networking.k8s.io/deny-ingress created
Waiting for deployment "source-controller" rollout to finish: 0 of 1 updated replicas are available...
deployment "source-controller" successfully rolled out
deployment "kustomize-controller" successfully rolled out
deployment "helm-controller" successfully rolled out
deployment "notification-controller" successfully rolled out
✔ install completed
► configuring deploy key
✔ deploy key configured
► generating sync manifests
✔ sync manifests pushed
► applying sync manifests
◎ waiting for cluster sync
✗ git clone error: unknown error: remote:

The status of the GitRepository CR:

$ kubectl get gitrepository gitops-toolkit -o yaml
apiVersion: source.toolkit.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
  creationTimestamp: "2020-08-24T09:32:20Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 1
  name: gitops-toolkit
  namespace: gitops-toolkit
  resourceVersion: "61696"
  selfLink: /apis/source.toolkit.fluxcd.io/v1alpha1/namespaces/gitops-toolkit/gitrepositories/gitops-toolkit
  uid: f5029ed7-a07b-46d7-9b64-8a681df30521
spec:
  interval: 1m0s
  ref:
    branch: master
  secretRef:
    name: gitops-toolkit
  url: ssh://git@ssh.gitlab.mydomain/devops/gitops-toolkit
status:
  conditions:
  - lastTransitionTime: "2020-08-24T10:44:33Z"
    message: 'git clone error: unknown error: remote: '
    reason: GitOperationFailed
    status: "False"
    type: Ready

As of the source-controller logs:

{"level":"error","ts":"2020-08-24T10:27:52.897Z","logger":"controller-runtime.controller","msg":"Reconciler error","controller":"gitrepository","name":"gitops-toolkit","namespace":"gitops-toolkit","error":"git clone error: unknown error: remote: ","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.1/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90"}

$ tk --version
tk version 0.0.18

The text was updated successfully, but these errors were encountered:

dbluxo · 2020-10-07T11:25:17Z

Any idea what that might be?

phenixdotnet · 2020-10-16T09:42:08Z

Hi, Same issue here with the latest version of gotk:

gotk --version
gotk version 0.1.6

Error message from debug log:

{
    "level": "error",
    "ts": "2020-10-16T09:42:37.951Z",
    "logger": "controller",
    "msg": "Reconciler error",
    "reconcilerGroup": "source.toolkit.fluxcd.io",
    "reconcilerKind": "GitRepository",
    "controller": "gitrepository",
    "name": "gotk-system",
    "namespace": "gotk-system",
    "error": "git clone error: unknown error: remote: ",
    "stacktrace": "github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:246\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.9/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.9/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.9/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.9/pkg/util/wait/wait.go:90"
}

Issue appear just after GitLab upgrade to version 13.4.4 (With Gitaly version 13.4.4 and Git version 2.28.0)

hiddeco · 2020-10-16T09:52:23Z

Issue appear just after GitLab upgrade to version 13.4.4 (With Gitaly version 13.4.4 and Git version 2.28.0)

Are you implying that it worked on your previous <13.4.4 GitLab instance? If so, can you please share what this initial version was?

dbluxo · 2020-10-16T09:59:02Z

I got the error with GitLab 12.10.11

stefanprodan · 2020-10-16T10:13:04Z

I can't explain why this works on GitLab.com but not on self-hosted. Can someone test https repos on-prem to see if that works?

GitLab on-prem https with token auth:

download the latest gotk CLI
install the controllers

gotk install

create a GitLab token with access to a repo
add the repo using the token

gotk create source git test \
--url=https://your-gitlab/org/repo \
--branch=master \
--username=git \
--password=your-gitlab-token

phenixdotnet · 2020-10-16T12:27:53Z

@hiddeco : Yes source-controller was working before the update.
We upgraded from gitlab-ce_13.3.1-ce.0

@stefanprodan: Cloning with https instead of ssh work perfectly, so at least we have a workaround 👍
Additionally, I tried to clone the repo with ssh and the gotk private key, everything is working fine. (git client version: 2.28.0)

We have 2 EKS clusters with gotk deployed. I just checked the second and I can't understand why this one is working even after the source-controller pods was killed. Both gotk install are up to date (ghcr.io/fluxcd/source-controller:v0.1.1)

stefanprodan · 2020-10-16T12:32:09Z

@phenixdotnet we don't use git the binary but go-git the package.

I just checked the second and I can't understand why this one is working even after the source-controller pods was killed.

Maybe it's related to VPC/firewalls?

stefanprodan · 2020-10-16T12:35:43Z

@dbluxo can you please try HTTPS instead of SSH, using my example?

I guess we could just drop ssh support from gotk bootstrap gitlab and use https only if it works for everyone.

phenixdotnet · 2020-10-16T12:37:41Z

@stefanprodan : Yes I know that :)
Anyway I found the issue, it's not related to gotk, I'm just stupid ... There is a typo error in the git repository configuration. Fixing it has fixed the issue.

One thing: If the error message could be a little more explicit it could help with this kind of error :)

Thanks for your help !

dbluxo · 2020-10-16T12:38:50Z

@dbluxo can you please try HTTPS instead of SSH, using my example?

Yes, I will try it

stefanprodan · 2020-10-16T12:46:56Z

One thing: If the error message could be a little more explicit it could help with this kind of error :)

We take the error from go-git as it is but clearly doesn't help in this case. Would appending the git URL to the error help?

phenixdotnet · 2020-10-16T12:50:48Z

We take the error from go-git as it is but clearly doesn't help in this case. Would appending the git URL to the error help?

Well, actually the error message is empty : (git clone error: unknown error: remote: ) So yes I guess having at least the repo url could be useful :)

dbluxo · 2020-10-16T14:33:01Z

I can't explain why this works on GitLab.com but not on self-hosted. Can someone test https repos on-prem to see if that works?

GitLab on-prem https with token auth:
* download the latest gotk CLI

* install the controllers
gotk install
* create a GitLab token with access to a repo

* add the repo using the token
gotk create source git test \
--url=https://your-gitlab/org/repo \
--branch=master \
--username=git \
--password=your-gitlab-token

https repos on-prem are working fine

How to gotk bootstrap gitlab with only using https?

stefanprodan · 2020-10-16T16:31:18Z

How to gotk bootstrap gitlab with only using https?

There is no such thing, but I'll make it happen 😄

stefanprodan changed the title ~~git clone error: unknown error: remote:~~ GitLab self-hosted: git clone unknown error Oct 16, 2020

stefanprodan mentioned this issue Oct 16, 2020

Add Git URL to clone error message #170

Merged

stefanprodan mentioned this issue Oct 16, 2020

Add GitLab HTTPS auth to bootstrap options fluxcd/flux2#351

Merged

stefanprodan closed this as completed in fluxcd/flux2#351 Oct 17, 2020

squaremo mentioned this issue Mar 3, 2021

Tidy git clone errors #304

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitLab self-hosted: git clone unknown error #120

GitLab self-hosted: git clone unknown error #120

dbluxo commented Aug 28, 2020

dbluxo commented Oct 7, 2020

phenixdotnet commented Oct 16, 2020 •

edited

Loading

hiddeco commented Oct 16, 2020

dbluxo commented Oct 16, 2020

stefanprodan commented Oct 16, 2020

phenixdotnet commented Oct 16, 2020

stefanprodan commented Oct 16, 2020

stefanprodan commented Oct 16, 2020

phenixdotnet commented Oct 16, 2020

dbluxo commented Oct 16, 2020

stefanprodan commented Oct 16, 2020

phenixdotnet commented Oct 16, 2020

dbluxo commented Oct 16, 2020

stefanprodan commented Oct 16, 2020

GitLab self-hosted: git clone unknown error #120

GitLab self-hosted: git clone unknown error #120

Comments

dbluxo commented Aug 28, 2020

dbluxo commented Oct 7, 2020

phenixdotnet commented Oct 16, 2020 • edited Loading

hiddeco commented Oct 16, 2020

dbluxo commented Oct 16, 2020

stefanprodan commented Oct 16, 2020

phenixdotnet commented Oct 16, 2020

stefanprodan commented Oct 16, 2020

stefanprodan commented Oct 16, 2020

phenixdotnet commented Oct 16, 2020

dbluxo commented Oct 16, 2020

stefanprodan commented Oct 16, 2020

phenixdotnet commented Oct 16, 2020

dbluxo commented Oct 16, 2020

stefanprodan commented Oct 16, 2020

phenixdotnet commented Oct 16, 2020 •

edited

Loading