Helm: Update chart's on-disk dependencies (k8s:helm-dependency-update)

[jmesnil]

Describe the bug

Requires/blocked by #2392

k8s:helm-dependency-update https://helm.sh/docs/helm/helm_dependency_update

As an user, I want to create a Helm chart with JKube that has a dependency.

Eclipse JKube version

1.11.0

Component

Kubernetes Maven Plugin

Apache Maven version

other (please specify in additional context)

Gradle version

None

Steps to reproduce

Use my fork of https://github.com/rohankanojia-forks/eclipse-jkube-helm-demo which uses the kubernetes-maven-plugin instead of the openshift-maven-plugin and has a dependency on the wildlfly Helm chart.

git clone https://github.com/jmesnil/eclipse-jkube-helm-demo
cd eclipse-jkube-helm-demo/jkube-helm-maven
mvn clean k8s:resource k8s:helm -Pxml-configuration
helm install -g target/jkube/helm/kubernetes/

Installation of the Helm chart fails with the error:

Error: INSTALLATION FAILED: An error occurred while checking for chart dependencies. You may need to run `helm dependency build` to fetch missing dependencies: found in Chart.yaml, but missing in charts/ directory: wildfly

Content of the Chart.yaml file is correct (with a wildfly dependency):

$ cat target/jkube/helm/kubernetes/Chart.yaml
---
---
apiVersion: v1
name: jkube-helm-maven
home: https://www.eclipse.org/jkube/
sources:
- https://github.com/eclipse/jkube
version: 1.0.0-SNAPSHOT
description: JKube Helm Maven demo Helm Chart (XML)
keywords:
- eclipse
- jkube
- kubernetes
- maven
maintainers:
- name: Maintainer1
  email: maintainer1@maintainer1.org
  url: maintainer1.org
- name: Maintainer2
  email: maintainer2@maintainer2.org
  url: maintainer2.org
icon: https://helm.sh/img/helm.svg
dependencies:
- name: wildfly
  version: 1.3.0
  repository: https://docs.wildfly.org/wildfly-charts/

Expected behavior

The Helm Chart is successfully installed.

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

1.25.3

Environment

macOS

Eclipse JKube Logs

[INFO] --- k8s:1.11.0:resource (default-cli) @ jkube-helm-maven ---
[INFO] k8s: Running generator quarkus
[INFO] k8s: quarkus: Using Docker image quay.io/jkube/jkube-java:0.0.17 as base / builder
[INFO] k8s: Using resource templates from /Users/jmesnil/tmp/eclipse-jkube-helm-demo/jkube-helm-maven/src/main/jkube
[INFO] k8s: jkube-controller: Adding a default Deployment
[INFO] k8s: jkube-service: Adding a default service 'jkube-helm-maven' with ports [8080]
[INFO] k8s: jkube-service-discovery: Using first mentioned service port '8080'
[INFO] k8s: jkube-revision-history: Adding revision history limit to 2
[INFO] k8s: validating /Users/jmesnil/tmp/eclipse-jkube-helm-demo/jkube-helm-maven/target/classes/META-INF/jkube/kubernetes/jkube-helm-maven-deployment.yml resource
[INFO] k8s: validating /Users/jmesnil/tmp/eclipse-jkube-helm-demo/jkube-helm-maven/target/classes/META-INF/jkube/kubernetes/jkube-helm-maven-service.yml resource
[INFO]
[INFO] --- k8s:1.11.0:helm (default-cli) @ jkube-helm-maven ---
[INFO] k8s: Creating Helm Chart "jkube-helm-maven" for Kubernetes

Sample Reproducer Project

https://github.com/jmesnil/eclipse-jkube-helm-demo/tree/main/jkube-helm-maven

Additional context

The installation is failing as JKube does not update the dependencies in the Chart directory.

When the Helm chart is generated it contains:

$ tree target/jkube/helm/kubernetes/
target/jkube/helm/kubernetes/
├── Chart.yaml
├── README.md
├── templates
│   ├── jkube-helm-maven-deployment.yaml
│   └── jkube-helm-maven-service.yaml
└── values.yaml

To be able to install the chart with its wildfly dependency, I must run helm dependency update:

$ helm dependency update ./target/jkube/helm/kubernetes
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://docs.wildfly.org/wildfly-charts/" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading wildfly from repo https://docs.wildfly.org/wildfly-charts/
Deleting outdated charts

The content of the Chart is then correct:

$ tree target/jkube/helm/kubernetes
target/jkube/helm/kubernetes
├── Chart.yaml
├── README.md
├── charts
│   └── wildfly-1.3.0.tgz
├── requirements.lock
├── templates
│   ├── jkube-helm-maven-deployment.yaml
│   └── jkube-helm-maven-service.yaml
└── values.yaml

And I can finally install it:

$ helm install -g target/jkube/helm/kubernetes/ --set wildfly.build.uri=https://github.com/wildfly-up-and-running/wuar-demo --set wildfly.build.enabled=false --set wildfly.deploy.route.enabled=false
NAME: kubernetes-1680096807
LAST DEPLOYED: Wed Mar 29 15:33:27 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

(Don't pay attention to the --set values, they are required by the wildfly Helm Chart to deploy on Kubernetes instead of OpenShift)

At first glance, the code at https://github.com/eclipse/jkube/blob/1b6af6b4f49f25cd0db4df6554df6566a28bf174/jkube-kit/resource/helm/src/main/java/org/eclipse/jkube/kit/resource/helm/HelmService.java#L80 is missing the helm dependency update execution to provide an installable Helm Chart.

[rohanKanojia]

Did you run helm dependency update before installing the chart?

[jmesnil]

Did you run helm dependency update before installing the chart?

No. As I mentioned, if I run it before I install from target/jkube/helm/kubernetes, it works.

The doc at https://www.eclipse.org/jkube/docs/kubernetes-maven-plugin/#jkube:helm does not mention to run helm dep up so I assumed that JKube would handle that step when I specify dependencies.

Additionally, the Helm archive generated by JKube is not correct and that one can not be updated (only unpacked charts can be updated)

[rohanKanojia]

@jmesnil : I did not understand this comment correctly.

Additionally, the Helm archive generated by JKube is not correct and that one can not be updated (only unpacked charts can be updated)

I tried doing helm dependency update with some other chart downloaded from the internet. But I got this error message:

~ : $ helm dependency update chartmuseum-2.5.0.tgz 
Error: only unpacked charts can be updated

I'm using Helm v3.10.3 . Which version of helm are you using?

[jmesnil]

@rohanKanojia Indeed, a Helm archive that contains dependencies can not be updated.

The issue with the archive generated by JKube is that it does not contain the dependencies specified in the pom.xml and is not a valid Helm archive.

The workaround is:

1. generate the Helm tgz with JKube
2. unarchive the tgz
3. run helm dep up
4. archive again the tgz with the pulled dependencies

[rohanKanojia]

@jmesnil : I'm not sure if I understand correctly. Did you specify dependencies in plugin configuration helm > dependencies section?

[rohanKanojia]

ohk, you mean jkube doesn't include the dependent charts.

At the moment we only include references to dependencies in generated Chart.yaml

[jmesnil]

ohk, you mean jkube doesn't include the dependent charts.

Exactly. So the tgz provided by JKube is not installable if it contains dependencies. I'm not sure how you want to fix it (as it could require JKube to do something similar to helm dep up before generating the archive) but this might at lease be called out in the documentation, wdyt?

[rohanKanojia]

We don't rely on helm CLI for generating charts. Need to check how Helm downloads and extracts these dependencies and see if we can emulate it in jkube.

[rohanKanojia]

@jmesnil : Do you know if when we build a standard helm chart with dependencies using helm, it includes the dependencies already, or helm dependency update step is required with helm CLI as well?

[jmesnil]

@rohanKanojia the dependencies are not included when you want to package a Helm archive, you have to run helm dep up to pull the dependencies.

As an example, starting from a Helm chart that has dependencies specified in Chart.yaml:

$ helm package chart
Error: found in Chart.yaml, but missing in charts/ directory: eap74

$ helm dep up chart
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jboss-eap" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading eap74 from repo https://jbossas.github.io/eap-charts/
Deleting outdated charts

$ helm package chart
Successfully packaged chart and saved it to: /Users/jmesnil/Developer/jboss-eap-up-and-running/eap7-with-postgres/eap7-with-postgresql-1.0.0.tgz

[manusa]

This is being worked on.

However, just a quick note.

The helm install command does provide a --dependency-update flag intended just for this purpose.

The following command:

helm install --dependency-update  -g target/jkube/helm/kubernetes/

should work just fine.

[manusa]

Helm: Update chart's on-disk dependencies (k8s:helm-dependency-update)

#2110

Description

As an Eclipse JKube user, I want to be able to update the chart's on-disk dependencies, so that I can properly install charts with dependencies.

Proposal

JKube Kit

JKube Kit exposes a new method dependencyUpdate that updates the chart's on-disk dependencies.

The chart should have been previously generated using the generateHelmCharts method.

• The update is performed by executing helm-java dependency update command.
• At this point, there are no options to be provided through HelmConfig. However, the method accepts a HelmConfg argument to infer the chart's location.
• If the update succeeds. the output of the update command is logged in white ([[W]]).
• If the update fails, error message is logged.

Maven plugins

The Kubernetes Maven Plugin and OpenShift Maven Plugin expose a new helm-dependency-update goal in a HelmDependencyUpdateMojo and OpenshiftHelmDependencyUpdateMojo that extends the basic HelmMojo (following a similar approach to that of HelmPushMojo).

The mojos are bound by default to the LifecyclePhase.INSTALL phase.

Tests should be added for each of the mojos and also for KubernetesPluginTest, OpenShiftPluginTest, and GeneratedPluginDescriptorTest.

Gradle plugins

The Kubernetes Gradle Plugin and OpenShift Gradle Plugin expose a new helmDependencyUpdate task in a KubernetesHelmDependencyUpdateTask and OpenShiftHelmDependencyUpdateTask that extends the basic AbstractJKubeTask task.

Individual tests should be added for the new tasks and also for the plugins to verify task precedence.

An integration test proves that the task can be executed and required dependencies are downloaded.

Documentation

The documentation exposes for each of the Maven, Gradle, Kubernetes, and OpenShift plugins the new dependency update goal/task.