Implementation of OCI-Catalog gRPC micro-service #6263
Assignees
Labels
component/apprepository
Issue related to kubeapps apprepository
kind/feature
An issue that reports a feature (approved) to be implemented
Milestone
Comments
absoludity
added
the
kind/proposal
ppbaena
added
kind/feature
absoludity
added a commit
that referenced
this issue
Jun 7, 2023
### Description of the change As per #6263, this PR adds an initally stubbed ocicatalog service gRPC service implementation. ### Benefits Can begin building the fully featured OCI Catalog for indexing OCI repositories. ### Applicable issues - Ref #6263 ### Additional information ``` $ grpcurl -proto ./proto/ocicatalog.proto -d '{"registry": "registry-1.docker.io"}' -plaintext "[::1]:50051" ocicatalog.OCICatalog.ListRepositoriesForRegistry { "registry": "registry-1.docker.io", "name": "repo-0" } { "registry": "registry-1.docker.io", "name": "repo-1" } { "registry": "registry-1.docker.io", "name": "repo-2" } { "registry": "registry-1.docker.io", "name": "repo-3" } { "registry": "registry-1.docker.io", "name": "repo-4" } { "registry": "registry-1.docker.io", "name": "repo-5" } { "registry": "registry-1.docker.io", "name": "repo-6" } { "registry": "registry-1.docker.io", "name": "repo-7" } { "registry": "registry-1.docker.io", "name": "repo-8" } { "registry": "registry-1.docker.io", "name": "repo-9" } $ grpcurl -proto ./proto/ocicatalog.proto -d '{"repository": {"registry": "registry-1.docker.io"}}' -plaintext "[::1]:50051" ocicatalog.OCICatalog.ListTagsForRepository { "name": "tag-0" } { "name": "tag-1" } { "name": "tag-2" } { "name": "tag-3" } { "name": "tag-4" } { "name": "tag-5" } { "name": "tag-6" } { "name": "tag-7" } { "name": "tag-8" } { "name": "tag-9" } ``` --------- Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Jun 9, 2023
### Description of the change Creating the Dockerfile for the stubbed service now so that we can start the process for getting an official image in the bitnami pipeline. I'll have another attempt at a static build for a scratch image before making the request. ### Benefits We can communicate a working Dockerfile for the official Bitnami image to be based on. ### Possible drawbacks <!-- Describe any known limitations with your change --> ### Applicable issues <!-- Enter any applicable Issues here (You can reference an issue using #) --> - ref #6263 ### Additional information ``` $ IMAGE_TAG=dev1 make kubeapps/oci-catalog ... $ docker run -p 50051:50051 docker.io/kubeapps/oci-catalog:dev1 ``` then in a separate terminal ``` $ grpcurl -proto ./proto/ocicatalog.proto -d '{"repository": {"registry": "registry-1.docker.io"}}' -plaintext "127.0.0.1:50051" ocicatalog.OCICatalog.ListTagsForRepository { "name": "tag-0" } { "name": "tag-1" } { "name": "tag-2" } { "name": "tag-3" } { "name": "tag-4" } { "name": "tag-5" } { "name": "tag-6" } { "name": "tag-7" } { "name": "tag-8" } { "name": "tag-9" } ``` --------- Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Jun 12, 2023
…6295) ### Description of the change Adds basic command options (`--help`, `--port`, `--version`). Splits code out to enable multiple providers later. ### Applicable issues <!-- Enter any applicable Issues here (You can reference an issue using #) --> - ref #6263 ### Additional information ``` $ RUST_LOG=info cargo run -- --port 9876 Finished dev [unoptimized + debuginfo] target(s) in 0.37s Running `target/debug/oci-catalog --port 9876` [2023-06-09T04:31:05Z INFO oci_catalog] listening for gRPC requests at 0.0.0.0:9876 ``` Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Jun 14, 2023
### Description of the change Adds implementation of send_repositories for the dockerhub implementation, replacing the stub. ### Benefits Actual results (see below). ### Applicable issues - ref #6263 ### Additional information Logs for request show two separate requests to hub.docker: ``` RUST_LOG=debug cargo run Finished dev [unoptimized + debuginfo] target(s) in 0.52s Running `target/debug/oci-catalog` [2023-06-13T01:54:29Z INFO oci_catalog] listening for gRPC requests at 0.0.0.0:50001 [2023-06-13T01:54:33Z DEBUG oci_catalog::providers::dockerhub] requesting: https://hub.docker.com/v2/namespaces/bitnamicharts/repositories/?page_size=100&ordering=name&content_types=helm [2023-06-13T01:54:33Z DEBUG reqwest::connect] starting new connection: https://hub.docker.com/ [2023-06-13T01:54:35Z DEBUG oci_catalog::providers::dockerhub] requesting: https://hub.docker.com/v2/namespaces/bitnamicharts/repositories/?content_types=helm&ordering=name&page=2&page_size=100 ``` Request shows grpc streaming the full result. ``` grpcurl -proto ./proto/ocicatalog.proto -d '{"registry": "registry-1.docker.io", "namespace": "bitnamicharts", "content_types":["helm"]}' -plaintext "0.0.0.0:50001" ocicatalog.OCICatalog.ListRepositoriesForRegistry { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "airflow" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "apache" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "apisix" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "appsmith" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "argo-cd" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "argo-workflows" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "aspnet-core" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "cassandra" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "cert-manager" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "clickhouse" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "common" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "concourse" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "consul" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "contour" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "contour-operator" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "discourse" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "dokuwiki" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "drupal" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "ejbca" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "elasticsearch" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "etcd" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "external-dns" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "flink" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "fluent-bit" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "fluentd" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "flux" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "ghost" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "gitea" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "grafana" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "grafana-loki" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "grafana-mimir" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "grafana-operator" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "grafana-tempo" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "haproxy" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "harbor" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "influxdb" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "jaeger" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "jasperreports" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "jenkins" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "joomla" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "jupyterhub" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "kafka" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "keycloak" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "kiam" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "kibana" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "kong" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "kubeapps" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "kube-prometheus" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "kubernetes-event-exporter" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "kube-state-metrics" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "logstash" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "magento" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "mariadb" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "mariadb-galera" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "mastodon" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "matomo" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "mediawiki" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "memcached" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "metallb" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "metrics-server" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "minio" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "mongodb" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "mongodb-sharded" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "moodle" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "multus-cni" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "mxnet" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "mysql" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "nats" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "nginx" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "nginx-ingress-controller" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "node-exporter" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "oauth2-proxy" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "odoo" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "opencart" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "osclass" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "parse" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "phpbb" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "phpmyadmin" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "pinniped" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "postgresql" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "postgresql-ha" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "prestashop" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "prometheus" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "pytorch" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "rabbitmq" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "rabbitmq-cluster-operator" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "redis" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "redis-cluster" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "redmine" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "schema-registry" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "sealed-secrets" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "solr" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "sonarqube" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "spark" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "spring-cloud-dataflow" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "suitecrm" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "supabase" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "tensorflow-resnet" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "thanos" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "tomcat" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "vault" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "wavefront" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "wavefront-hpa-adapter" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "wavefront-prometheus-storage-adapter" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "whereabouts" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "wildfly" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "wordpress" } { "registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "zookeeper" } ```
absoludity
added a commit
that referenced
this issue
Jun 15, 2023
### Description of the change After improving the error handling, this PR adds a strategy pattern for enabling the provider to be chosen based on the request. ### Benefits Other providers can be added easily. ### Applicable issues <!-- Enter any applicable Issues here (You can reference an issue using #) --> - ref #6263 --------- Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Jun 28, 2023
### Description of the change This PR adds an initial OCI check for `CatalogAvailable` which currently just checks a VMware application catalog specific index. This will later be generalised to support other registries. ### Benefits May provide a quick win for displaying a catalog for VAC registries. Next need to use this when validating an OCI registry in the UI, then use when syncing the registry. ### Applicable issues - ref #6263 Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Jun 29, 2023
…6342) ### Description of the change This PR updates the OCI app registry validation so that an empty list of repositories is allowed if a catalog is available. ### Benefits Step 2 of 3 in experiment to support VAC index. ### Applicable issues - ref #6263 Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Jun 30, 2023
### Description of the change Part 3/3 for experimental support of VAC indexes. ### Benefits Potentially quick win to display VAC OCI repos. ### Applicable issues - ref #6263 ### Additional information Still need to test locally. Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Jun 30, 2023
### Description of the change Two small fixes required when testing the VAC catalog for an OCI registry: 1. Ensure the UX no longer requires the filter field 2. Ensure in the backend the repositories are set before the CheckSum is calculated. ### Benefits With this change, Kubeapps displays the full catalog from a VAC-published OCI registry. - ref #6263 Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Aug 9, 2023
### Description of the change Adds the implementation for retrieving and sending tags for dockerhub repositories. ### Benefits Enables an initial integration to be tested. ### Possible drawbacks <!-- Describe any known limitations with your change --> ### Applicable issues <!-- Enter any applicable Issues here (You can reference an issue using #) --> - ref #6263 ### Additional information IRL test: ```console grpcurl -plaintext -proto ./proto/ocicatalog.proto -d '{ "repository": {"registry": "registry-1.docker.io", "namespace": "bitnamicharts", "name": "zookeeper" }}' 0.0.0.0:50001 ocicatalog.OCICatalog.ListTagsForRepository { "name": "11.4.10" } { "name": "11.4.9" } { "name": "11.4.8" } { "name": "11.4.7" } { "name": "11.4.6" } { "name": "11.4.5" } { "name": "11.4.4" } { "name": "11.4.3" } { "name": "11.4.2" } { "name": "11.4.1" } { "name": "11.3.2" } { "name": "11.3.1" } { "name": "11.2.1" } { "name": "11.1.6" } { "name": "11.1.5" } { "name": "11.1.4" } { "name": "11.1.3" } { "name": "11.1.2" } { "name": "11.1.1" } { "name": "11.0.3" } { "name": "11.0.2" } { "name": "11.0.1" } { "name": "11.0.0" } { "name": "10.2.5" } { "name": "10.2.4" } { "name": "10.2.3" } ``` --------- Signed-off-by: Michael Nelson <minelson@vmware.com>
This was referenced Aug 9, 2023
absoludity
added a commit
that referenced
this issue
Aug 10, 2023
### Description of the change While setting up the dev chart changes, just realised I hadn't yet updated to publish the dev image. ### Benefits Dev image for oci-catalog will be available. ### Applicable issues - ref #6263 ### Additional information <!-- If there's anything else that's important and relevant to your pull request, mention that information here.--> --------- Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Aug 14, 2023
) ### Description of the change Just a small copyright date update in proto files and mechanical re-generation with current buf. I was needing to run buf generate on my own work branch and was seeing distracting side-effects such as dates which had been manually (?) changed to 2023 in generated files being reverted back to 2022. So this PR just updates the dates in the proto files and regenerates so I won't have that noise in my own diff. ### Benefits Easier reviewing. ### Applicable issues - ref #6263 ### Additional information I'll land this once passing to continue. Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Aug 15, 2023
### Description of the change Adds the initial values and container for the oci-catalog to the kubeapps-apis service. Leaving in draft for the moment as there are a number of things I want to check IRL while playing: - ~~Use an actual grpc liveness check (to test that out, since we can remove the grpc health binary, as our oldest k8s version is now past 1.24)~~ Bitnami tests are still testing with 1.23 apparently. - Evaluate whether kubeapps-apis is really the right pod for having this side-car. We'll need to use the service when validating a repo from the UX (which is kubeapps-apis), but also during the sync job. It could also run as a separate pod, but not sure it's worth the resources. Note: I've got `ociCatalog.enabled` defaulting to false now, so no change to the chart output. ### Benefits Can start integration. ### Possible drawbacks ### Applicable issues - ref #6263 ### Additional Information ```console $ helm template ./chart/kubeapps --debug --set ociCatalog.enabled=true | grep -A 40 oci-catalog install.go:200: [debug] Original chart version: "" install.go:217: [debug] CHART PATH: /Users/minelson/dev/vmware/kubeapps/chart/kubeapps - name: oci-catalog image: docker.io/kubeapps/oci-catalog:latest imagePullPolicy: "IfNotPresent" securityContext: runAsNonRoot: true runAsUser: 1001 command: - /oci-catalog args: env: - name: OCI_CATALOG_PORT value: "50061" envFrom: ports: - name: grpc-http containerPort: 50061 livenessProbe: failureThreshold: 6 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 exec: command: ["grpc_health_probe", "-addr=:50061"] initialDelaySeconds: 10 readinessProbe: failureThreshold: 6 initialDelaySeconds: 0 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 exec: command: ["grpc_health_probe", "-addr=:50061"] initialDelaySeconds: 5 resources: limits: cpu: 250m memory: 256Mi requests: cpu: 25m memory: 32Mi volumeMounts: volumes: - name: clusters-config configMap: name: release-name-kubeapps-clusters-config - name: ca-certs emptyDir: {} ``` With these changes in dev: ``` k -n kubeapps logs kubeapps-internal-kubeappsapis-7f5cc7f98b-wnqq4 oci-catalog [2023-08-14T01:42:41Z INFO oci_catalog] listening for gRPC requests at 0.0.0.0:50061 ``` --------- Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Aug 15, 2023
…6618) ### Description of the change Follows on from #6595, updates the proto service files according to buf's lint, and generates the go client for use in the asset-syncer. Note: I added a separate `buf.yaml` and `buf.gen.yaml` for the oci-catalog service as buf doesn't let you refer to a proto outside of the `buf.yaml` root, so the other option would have been to move our existing `buf.yaml` and `buf.generate` into the Kubeapps root directory. I don't mind either way, but it made more sense to me for the oci-catalog service to have its own so that it is independent (it may be useful outside of Kubeapps, theoretically). ### Benefits The next PR can use the client without being cluttered by the extra diff. ### Applicable issues <!-- Enter any applicable Issues here (You can reference an issue using #) --> - ref: #6263 --------- Signed-off-by: Michael Nelson <minelson@vmware.com>
This was referenced Aug 15, 2023
absoludity
added a commit
that referenced
this issue
Aug 16, 2023
…6620) ### Description of the change First part of integration work to update the OCI repo support to use the oci-catalog service. This PR refactors the existing repository validation to remove the assumption of an `HTTPValidator` (since the gRPC validation doesn't use an httpclient etc.) ### Benefits Step towards adding grpc validation for OCI repositories using the new service. ### Possible drawbacks None ### Applicable issues - ref #6263 Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Aug 17, 2023
### Description of the change Follows on from #6620, uses the grpc Client to check if we can find repos for an OCI registry. While there I improved a couple of other things regarding the validation: - a context.Context can (and must) now be passed through to the validation (we should always pass the request context through subsequent requests); and - the getValidator function is now a method so that the different validators can have custom data set (such as the OCI catalog address, or the repo client getter). I've added a basic test double grpc service for the OCI catalog to use here in tests, and will undoubtedly improve this test double when adding the actual sync functionality to use the OCI catalog service next. ### Benefits OCI Catalog service can be used to check an OCI Registry, without impacting current behaviour. ### Possible drawbacks None that I'm aware of. ### Applicable issues - ref #6263 ### Additional information I'll test IRL locally and update here: Logs showing the fallback when the OCI catalog is not running: ``` E0816 03:49:15.487354 1 repositories_validation.go:354] unable to query OCI Catalog service at ":50061": error querying OCI Catalog for repos: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp :50061: connect: connection refused" E0816 03:49:16.462758 1 utils.go:400] unable to get catalog manifest: GET request to [https://registry-1.docker.io/v2/bitnamicharts/charts-index/manifests/latest] failed due to status [401]: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"bitnamicharts/charts-index","Action":"pull"}]}]} E0816 03:49:16.462894 1 repositories_validation.go:52] Failed repository validation validation: &{Code:400 Message:unable to determine the OCI catalog, you need to specify at least one repository} ``` and verified I can add the Bitnami catalog using the OCI `https://registry-1.docker.io/bitnamicharts` . That is, it passes validation and gets added. It doesn't yet sync - that'll be the next PR. --------- Signed-off-by: Michael Nelson <minelson@vmware.com>
This was referenced Aug 23, 2023
absoludity
added a commit
that referenced
this issue
Aug 24, 2023
…elm repositories. (#6680) ### Description of the change The code currently talks about OCI repositories and Helm repositories and Helm repositories that are OCI registries. This PR is just splitting out some renaming to avoid the confusion, so instead we have: - `Repo` model -> `AppRepository` which correspond to our `AppRepository` custom resource which can represent a Helm repository or an OCI registry (or a namespace within an OCI registry) which contains many oci repositories. - `Repo` interace -> `ChartCatalog` interface, as this is implemented by both Helm and OCI versions It contains another small refactor pulling out a `ocicatalog_client.NewClient` helper. Leaving as draft for now in case I come across any other straight renames/refactors in the current work. ### Applicable issues <!-- Enter any applicable Issues here (You can reference an issue using #) --> - ref #6263 ### Additional information <!-- If there's anything else that's important and relevant to your pull request, mention that information here.--> Signed-off-by: Michael Nelson <minelson@vmware.com>
absoludity
added a commit
that referenced
this issue
Aug 24, 2023
…6684) ### Description of the change This PR moves the validation code that checks the oci-catalog service into the existing OCIApiClient itself, so that we can use the same and encapsulate the functionality in the one place - the OCIApiClient - when adding the sync functionality. ### Benefits Functionality encapsulated in the one client. Also improves existing code to use contexts in network calls. Also removes the `AuthHeader` that was not being used, since some time ago we switched to creating the http client using the header, so the auth is present there already. ### Possible drawbacks None that I can see (let CI determine). ### Applicable issues - ref #6263 ### Additional information <!-- If there's anything else that's important and relevant to your pull request, mention that information here.--> --------- Signed-off-by: Michael Nelson <minelson@vmware.com>
|
So the last PR updates the sync code to use the OCI catalog, when available (and when the namespaced OCI registry - the AppRepository - does not specify the repos). But as discussed, since the rest of the sync code (such as pulling the charts and extracting the files etc.) uses the OCI Distribution Spec API, it currently requires credentials even for a public registry, such as the Bitnami catalog. I need to investigate separately whether it's possible for us to do this without creds using, for dockerhub for example, the separate dockerhub API, and spec/implement the functionality in the oci-catalog service. |
absoludity
added a commit
that referenced
this issue
Aug 24, 2023
### Description of the change This PR updates the sync code to also try the OCI Catalog service for app repositories where the repos aren't listed (after trying the VAC index). ### Benefits Finishes the work of #6263, although, as we've discussed, to benefit from this (ie. be able to add the public bitnami OCI repo), we need to add a couple more features, since currently it uses the existing OCI Distribution API which *requires* authentication. We'd like to ensure people don't need authentication to use a public OCI namespace, such as the bitnami catalog. ### Possible drawbacks ### Applicable issues <!-- Enter any applicable Issues here (You can reference an issue using #) --> - fixes #6263 ### Additional information <!-- If there's anything else that's important and relevant to your pull request, mention that information here.--> Signed-off-by: Michael Nelson <minelson@vmware.com>
This was referenced Aug 25, 2023
absoludity
added a commit
that referenced
this issue
Aug 29, 2023
### Description of the change
Updates the dev chart and snippets of code so that if the OCI-Catalog
service is enabled, the URL is passed all the way through to the sync
job.
Verified from the logs:
```
I0829 01:33:33.729325 1 root.go:32] "The component 'asset-syncer' has been configured with" serverOptions={"DatabaseURL":"kubeapps-postgresql:5432","DatabaseName":"assets","DatabaseUser":"postgres","DatabasePassword":"REDACTED","Debug":false,"Namespace":"default","OciRepositories":[],"TlsInsecureSkipVerify":false,"FilterRules":"","PassCredentials":false,"UserAgent":"asset-syncer/903124d9eb9976d7cad846331c20440132b143df (kubeapps/DEVEL)","UserAgentComment":"kubeapps/DEVEL","GlobalPackagingNamespace":"kubeapps","KubeappsNamespace":"","AuthorizationHeader":"","DockerConfigJson":"","OCICatalogURL":"kubeapps-internal-kubeappsapis:50061"}
I0829 01:33:34.926684 1 utils.go:495] Unable to find VAC index: .... Attempting OCI-Catalog
Error: error: GET request to [https://registry-1.docker.io/v2/bitnamicharts/airflow/tags/list] failed due to status [401]: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"bitnamicharts/airflow","Action":"pull"}]}]}
```
### Benefits
Can start next step of getting an anon token to continue the sync.
### Possible drawbacks
### Applicable issues
- ref #6706
### Additional information
This should really have been done as part of the previous #6263 where I
updated the sync code to use the new service, but didn't hook it up.
---------
Signed-off-by: Michael Nelson <minelson@vmware.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary
Enable Kubeapps to display a catalog of apps for an OCI registry or a namespace of an OCI registry. The proposed implementation is for a stateless gRPC micro-service that can be run (though is not restricted to run) as a side-car of existing the asset-syncer job to provide lists of repositories for a (namespaced) registry, regardless of the registry provider.
Background and rationale
See #6179 for the problem/investigation as well as the Kubeapps OCI-Catalog proposal for more information.
Description
This issue tracks the implementation of the functionality. I'm listing the tasks here, which we may expand into subtasks as needed, but initially having them here is fine.
Implement OCI Catalog service
ListRepositoriesForRegistryandListTagsForRepositoryListRepositoriesForRegistryandListTagsForRepositoryfor dockerhub (at this point, the integration tasks below can begin too)Integrate OCI Catalog service
Switch existing code for obtaining tags to use the gRPC call instead.Additional changes to the add AppRepository form functionality (we may want to adjust the filter so that it operates as a filter only rather than specification of available repos). I suspect we may require other small changes too (to hint which provider should be used for ambiguous cases, or provide extra endpoint).Ensure integration for single Kubeapps installation with one dockerhub OCI registry (bitnamicharts namespace).Acceptance criteria
See the last integration task. When the catalog lists the available apps correctly from all three OCI registries then this task is considered finished.
The text was updated successfully, but these errors were encountered: