[aws-eks] allow specifying timeouts for HelmCharts

[gibbster]

For long running helm charts the creation times out.

Reproduction Steps

I had this issue with the following resource (based on a minimal cluster defined through the CDK Cluster component).

new eks.HelmChart(this, 'spinnaker', {
  cluster,
  chart: 'spinnaker',
  repository: 'https://kubernetes-charts.storage.googleapis.com/',
  version: '2.0.0-rc3',
  namespace: 'spinnaker'
});

Error Log

3/9 | 3:05:54 PM | CREATE_FAILED | Custom::AWSCDK-EKS-HelmChart | spinnaker/Resource/Default (spinnakerC4B2DB2C) Failed to create resource. TimeoutError: Connection timed out after 120000ms

Environment

• **CLI Version :1.41.0 (build 9e071d2)
• **Framework Version: 1.41.0
• **OS : macOS 10.14.6
• **Language : typescript

Other

The helm command has an option for timeout, and in fact the chart I was trying to run (stable/spinnaker) suggests a timeout of 600s: https://github.com/helm/charts/blob/master/stable/spinnaker/README.md. I believe that the chart should add timeout as a option:

aws-cdk/packages/@aws-cdk/aws-eks/lib/helm-chart.ts

Lines 8 to 50 in 86eac6a

	export interface HelmChartOptions {
	/**
	* The name of the chart.
	*/
	readonly chart: string;
	/**
	* The name of the release.
	* @default - If no release name is given, it will use the last 53 characters of the node's unique id.
	*/
	readonly release?: string;
	/**
	* The chart version to install.
	* @default - If this is not specified, the latest version is installed
	*/
	readonly version?: string;
	/**
	* The repository which contains the chart. For example: https://kubernetes-charts.storage.googleapis.com/
	* @default - No repository will be used, which means that the chart needs to be an absolute URL.
	*/
	readonly repository?: string;
	/**
	* The Kubernetes namespace scope of the requests.
	* @default default
	*/
	readonly namespace?: string;
	/**
	* The values to be used by the chart.
	* @default - No values are provided to the chart.
	*/
	readonly values?: {[key: string]: any};
	/**
	* Whether or not Helm should wait until all Pods, PVCs, Services, and minimum number of Pods of a
	* Deployment, StatefulSet, or ReplicaSet are in a ready state before marking the release as successful.
	* @default - Helm will not wait before marking release as successful
	*/
	readonly wait?: boolean;
	}

1.41.0 (build 9e071d2)

---

This is 🐛 Bug Report

[eladb]

It would be possible to allow timeouts of up to 15m. @pahud is this something you'd be interested to pick up?

[pahud]

@eladb Sure!

The interesting thing is the Kubelctl Provider default timeout is 15min now.

aws-cdk/packages/@aws-cdk/aws-eks/lib/kubectl-provider.ts

Line 26 in 99e7330

timeout: Duration.minutes(15),

And the error message provided doesn't seem to be a lambda timeout

3/9 | 3:05:54 PM | CREATE_FAILED | Custom::AWSCDK-EKS-HelmChart | spinnaker/Resource/Default (spinnakerC4B2DB2C) Failed to create resource. TimeoutError: Connection timed out after 120000ms

I'll try re-produce it in my environment and check the logs.

[eduardomourar]

We are getting the same error while running:

cluster.addChart('Prometheus', {
  chart: 'prometheus-operator',
  repository: 'https://kubernetes-charts.storage.googleapis.com/',
  version: '8.13.8',
  namespace: 'default'
});

[eduardomourar]

I can confirm that there is a timeout from the AWS SDK (as specified here) and it can solved by adding to the provider framework code:

lambda.config.update({httpOptions: { timeout: 900000 }});

I will make a PR soon to fix this.

[pahud]

It sounds like an underlying http timeout between the provider framework and the kubectl provider which executes the helm command. Is the fix working in your environment? @eduardomourar

[eduardomourar]

Yes, so far so good. My helm charts are taking about 3 minutes to deploy and without any issue now.

I believe the issue is that the invokeFunction used by the provider framework has this 2 minutes timeout, so it was not honoring the overall timeout (15 minutes) set for the target lambda.