Porch failing to connect to remote Git private repo with self-signed SSL certs

[dgeorgievski]

As Porch has moved to a new repository, it was suggested to move this issue here as all new work will take place under Nephio

kptdev/kpt#4099

Expected behavior

kpt alpha repo register should register a new, remote Git repo as explained in Porch instructions:
https://kpt.dev/guides/porch-user-guide

Actual behavior

The two registered repos, app-health-5g-blueprint and app-health-5g-deployment, are not ready.

$ kpt alpha repo list
NAME                       TYPE   CONTENT   DEPLOYMENT   READY   ADDRESS
app-health-5g-blueprint    git    Package                False   https://my-gitlab.com/dimitar.georgievski/blueprint.git
app-health-5g-deployment   git    Package   true         False   https://my-gitlab.com/dimitar.georgievski/deployment.git
free5gc-packages           git    Package   false        True    https://github.com/nephio-project/free5gc-packages.git
mgmt                       git    Package   true         True    http://172.18.0.200:3000/nephio/mgmt.git
mgmt-staging               git    Package   false        True    http://172.18.0.200:3000/nephio/mgmt-staging.git
nephio-example-packages    git    Package   false        True    https://github.com/nephio-project/nephio-example-packages.git

Information

kpt version: 1.0.0-beta.38
Porch version: v0.0.20

Steps to reproduce the behavior

kpt and Porch were deployed as part of nephio sandbox deployment in a kind cluster installed on a VM box:
https://github.com/nephio-project/docs/tree/main/install-guide#installing-on-a-pre-provisioned-vm

I tried to register two private GitLab repositories with self-signed SSL certificate which caused the following error on the Porch server side

cannot fetch repository default/app-health-5g-blueprint: Get "https://my-gitlab..com/dimitar.georgievski/blueprint.git/info/refs?service=git-upload-pack": 
tls: failed to verify certificate: x509: certificate signed by unknown authority

Could Porch server be configured to either skip SSL validation of the remote Git server or deploy the custom CA/SSL certificate?

[dgeorgievski]

It is worth mentioning that as long as the not-ready repos are registered with Porch operations with other ready repos are not possible which should be considered a bug.

$ kpt alpha rpkg get --name nephio-workload-cluster -v 10
I1208 15:44:38.625379 2251690 loader.go:374] Config loaded from file:  /home/ubuntu/.kube/config
I1208 15:44:38.641047 2251690 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;g=meta.k8s.io;v=v1,application/json" -H "User-Agent: kpt/1.0.0-beta.38" 'https://127.0.0.1:34759/apis/porch.kpt.dev/v1alpha1/namespaces/default/packagerevisions?fieldSelector=spec.packageName%3Dnephio-workload-cluster'
I1208 15:44:38.641714 2251690 round_trippers.go:510] HTTP Trace: Dial to tcp:127.0.0.1:34759 succeed
I1208 15:44:38.700053 2251690 round_trippers.go:553] GET https://127.0.0.1:34759/apis/porch.kpt.dev/v1alpha1/namespaces/default/packagerevisions?fieldSelector=spec.packageName%3Dnephio-workload-cluster 500 Internal Server Error in 58 milliseconds
I1208 15:44:38.700113 2251690 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 15 ms ServerProcessing 42 ms Duration 58 ms
I1208 15:44:38.700132 2251690 round_trippers.go:577] Response Headers:
I1208 15:44:38.700161 2251690 round_trippers.go:580]     Content-Length: 339
I1208 15:44:38.700193 2251690 round_trippers.go:580]     Audit-Id: 0c956862-a3ea-4d16-a9b3-303314203984
I1208 15:44:38.700217 2251690 round_trippers.go:580]     Audit-Id: 0c956862-a3ea-4d16-a9b3-303314203984
I1208 15:44:38.700234 2251690 round_trippers.go:580]     Cache-Control: no-cache, private
I1208 15:44:38.700275 2251690 round_trippers.go:580]     Cache-Control: no-cache, private
I1208 15:44:38.700304 2251690 round_trippers.go:580]     Content-Type: application/json
I1208 15:44:38.700327 2251690 round_trippers.go:580]     Date: Fri, 08 Dec 2023 15:44:38 GMT
I1208 15:44:38.700349 2251690 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 6abfd417-66e8-45a9-9542-54d899bea11d
I1208 15:44:38.700372 2251690 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 6abfd417-66e8-45a9-9542-54d899bea11d
I1208 15:44:38.700395 2251690 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 18e84185-1b8a-43b9-a989-8cec2e934432
I1208 15:44:38.700417 2251690 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 18e84185-1b8a-43b9-a989-8cec2e934432
I1208 15:44:38.700530 2251690 request.go:1154] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"cannot fetch repository default/app-health-5g-blueprint: Get \"https://my-gitlab.com/dimitar.georgievski/blueprint.git/info/refs?service=git-upload-pack\": tls: failed to verify certificate: x5           509: certificate signed by unknown authority","code":500}
I1208 15:44:38.700661 2251690 request.go:1345] body was not decodable (unable to check for Status): no kind "Status" is registered for version "v1" in scheme "pkg/runtime/scheme.go:100"
Error: an error on the server ("unknown") has prevented the request from succeeding (get packagerevisions.porch.kpt.dev)

[electrocucaracha]

We have recently moved most of the packages from nephio-example-packages repository to catalog. During that migration, we have upgraded the Porch Server version to point to the latest image. Could you verify that version is also failing?

[johnbelamaric]

It will for sure. This is a feature request, really, not a bug.

[liamfallon]

Triaged
Triage Comment: Private CA is not supported, nor is self signed certs

[dgeorgievski]

I wanted to confirm that the solution with external Git CA bundle resolved the issue, and I can access private Git repos now.
https://github.com/nephio-project/porch/blob/main/docs/adding-external-git-ca-bundle.md

This is great, but it would be even better if Porch would support 'go-git' InsecureSkipTLS option which would obviate the need to use CA bundles.

I'd use the bundles in Prod clusters, but in development and CI pipelines, there is a need to frequently bring up and down Kind clusters, and the deployment of CA bundles would create unnecessary overhead. I will create a new issue for this request.

[efiacor]

I wanted to confirm that the solution with external Git CA bundle resolved the issue, and I can access private Git repos now. https://github.com/nephio-project/porch/blob/main/docs/adding-external-git-ca-bundle.md

This is great, but it would be even better if Porch would support 'go-git' InsecureSkipTLS option which would obviate the need to use CA bundles.

I'd use the bundles in Prod clusters, but in development and CI pipelines, there is a need to frequently bring up and down Kind clusters, and the deployment of CA bundles would create unnecessary overhead. I will create a new issue for this request.

Hi @dgeorgievski , thanks for confirming.
Yes, you are correct that adding the other optional flags to the go-git connection would be the optimal solution.
We had considered this and more but felt for R3, we would provide a quick fix.
Beyond R3, the plan would be to look to extend the Porch Repository Custom Resource to provide additional connection protocols and or security features.

[dgeorgievski]

We had considered this and more but felt for R3, we would provide a quick fix. Beyond R3, the plan would be to look to extend the Porch Repository Custom Resource to provide additional connection protocols and or security features.

Thank you, @efiacor. This is more than enough to get us going with the review of Porch ( and Nephio in the long term). We have high expectations about both. By "we", I mean Axyom.Core, which was recently spun off Casa Systems. I'd be glad to provide feedback.