Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update cluster autoscaler addon documentation for newly added flags #1714

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 39 additions & 1 deletion content/kubeone/main/guides/autoscaler-addon/_index.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,43 @@ addons:

If you wish to change some of the properties, such as timeout for scaling up/down, you’ll need to provide the appropriate command-line flags to cluster-autoscaler. For that, you’ll need to override the cluster-autoscaler addon embedded in the KubeOne binary with your addon.

To find out how to override embedded add-ons, please check the [Addons document][embedded-addons]. For more information regarding available configuration parameters and options, please check the [Cluster Autoscaler FAQ][ca-faq].
To find out how to override embedded add-ons, please check the [Addons document][embedded-addons].

If you are facing issue similar to the one described over [here][enforce-node-group-min-size] in the [Cluster Autoscaler FAQ][ca-faq], pass down the param `CLUSTER_AUTOSCALER_ENFORCE_NODE_GROUP_MIN_SIZE` to the cluster autoscaler addon as shown below:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you are facing issue similar to the one described over [here][enforce-node-group-min-size] in the [Cluster Autoscaler FAQ][ca-faq], pass down the param `CLUSTER_AUTOSCALER_ENFORCE_NODE_GROUP_MIN_SIZE` to the cluster autoscaler addon as shown below:
If you're facing an issue where your [cluster is below the minimum or above the maximum number of nodes, but the cluster-autoscaler is not fixing it][ca-faq], you need to pass down the `CLUSTER_AUTOSCALER_ENFORCE_NODE_GROUP_MIN_SIZE` parameter to the cluster-autoscaler addon as shown below:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!


```yaml
apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
kubernetes: '1.29.4' ## kubernetes version
cloudProvider: ## This field is sourced automatically if terraform is used for the cluster
aws: {}
addons:
enable: true
addons:
- name: cluster-autoscaler
params:
CLUSTER_AUTOSCALER_ENFORCE_NODE_GROUP_MIN_SIZE: "true"
```

If you are facing issue similar to the one described over [here][balance-similar-node-groups] in the [Cluster Autoscaler FAQ][ca-faq], pass down the param `CLUSTER_AUTOSCALER_BALANCE_SIMILAR_NODE_GROUP` to the cluster autoscaler addon as shown below:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you are facing issue similar to the one described over [here][balance-similar-node-groups] in the [Cluster Autoscaler FAQ][ca-faq], pass down the param `CLUSTER_AUTOSCALER_BALANCE_SIMILAR_NODE_GROUP` to the cluster autoscaler addon as shown below:
If you're running a cluster with nodes in the multiple zones for the HA purposes, consider setting the `CLUSTER_AUTOSCALER_BALANCE_SIMILAR_NODE_GROUP` parameter to `true` as shown below. For more information about this parameter, check the [Cluster Autoscaler FAQ][balance-similar-node-groups].

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, to make it easier to follow the document, maybe we should provide just one YAML example with both arguments set, eventually with a comment that you can remove the parameter that you don't need.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!


```yaml
apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
kubernetes: '1.29.4' ## kubernetes version
cloudProvider: ## This field is sourced automatically if terraform is used for the cluster
aws: {}
addons:
enable: true
addons:
- name: cluster-autoscaler
params:
CLUSTER_AUTOSCALER_BALANCE_SIMILAR_NODE_GROUP: "true"
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we put this in the step 1 instead, in an information or a warning block? This step is more about modifying the cluster-autoscaler itself (i.e. YAML manifests), but this is more about additionally configuring the cluster-autoscaler addon.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!


For more information regarding available configuration parameters and options, please check the [Cluster Autoscaler FAQ][ca-faq].

### Step 3: Deploying Kubernetes Cluster Autoscaler

Expand Down Expand Up @@ -212,3 +248,5 @@ That is it! You have successfully deployed Kubernetes autoscaler on the KubeOne
[embedded-addons]: {{< ref "../../guides/addons/#overriding-embedded-eddons" >}}
[ca-faq]: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md
[ca-faq-what-is]: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-is-cluster-autoscaler
[enforce-node-group-min-size]: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#my-cluster-is-below-minimum--above-maximum-number-of-nodes-but-ca-did-not-fix-that-why
[balance-similar-node-groups]: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you instead use permalinks here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!

40 changes: 39 additions & 1 deletion content/kubeone/v1.8/guides/autoscaler-addon/_index.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,43 @@ addons:

If you wish to change some of the properties, such as timeout for scaling up/down, you’ll need to provide the appropriate command-line flags to cluster-autoscaler. For that, you’ll need to override the cluster-autoscaler addon embedded in the KubeOne binary with your addon.

To find out how to override embedded add-ons, please check the [Addons document][embedded-addons]. For more information regarding available configuration parameters and options, please check the [Cluster Autoscaler FAQ][ca-faq].
To find out how to override embedded add-ons, please check the [Addons document][embedded-addons].

If you are facing issue similar to the one described over [here][enforce-node-group-min-size] in the [Cluster Autoscaler FAQ][ca-faq], pass down the param `CLUSTER_AUTOSCALER_ENFORCE_NODE_GROUP_MIN_SIZE` to the cluster autoscaler addon as shown below:

```yaml
apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
kubernetes: '1.29.4' ## kubernetes version
cloudProvider: ## This field is sourced automatically if terraform is used for the cluster
aws: {}
addons:
enable: true
addons:
- name: cluster-autoscaler
params:
CLUSTER_AUTOSCALER_ENFORCE_NODE_GROUP_MIN_SIZE: "true"
```

If you are facing issue similar to the one described over [here][balance-similar-node-groups] in the [Cluster Autoscaler FAQ][ca-faq], pass down the param `CLUSTER_AUTOSCALER_BALANCE_SIMILAR_NODE_GROUP` to the cluster autoscaler addon as shown below:

```yaml
apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster
versions:
kubernetes: '1.29.4' ## kubernetes version
cloudProvider: ## This field is sourced automatically if terraform is used for the cluster
aws: {}
addons:
enable: true
addons:
- name: cluster-autoscaler
params:
CLUSTER_AUTOSCALER_BALANCE_SIMILAR_NODE_GROUP: "true"
```

For more information regarding available configuration parameters and options, please check the [Cluster Autoscaler FAQ][ca-faq].

### Step 3: Deploying Kubernetes Cluster Autoscaler

Expand Down Expand Up @@ -212,3 +248,5 @@ That is it! You have successfully deployed Kubernetes autoscaler on the KubeOne
[embedded-addons]: {{< ref "../../guides/addons/#overriding-embedded-eddons" >}}
[ca-faq]: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md
[ca-faq-what-is]: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-is-cluster-autoscaler
[enforce-node-group-min-size]: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#my-cluster-is-below-minimum--above-maximum-number-of-nodes-but-ca-did-not-fix-that-why
[balance-similar-node-groups]: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This section deals with the usage of Kubernetes Cluster Autoscaler in a KKP User

## What is a Cluster Autoscaler in Kubernetes?

Kubernetes Cluster Autoscaler is a tool that automatically adjusts the size of the worker’s node up or down depending on the consumption. This means that the autoscaler, for example, automatically scale up a cluster by increasing the node count when there are not enough node resources for cluster workload scheduling and scale down when the node resources have continuously staying idle, or there are more than enough node resources available for cluster workload scheduling. In a nutshell, it is a component that automatically adjusts the size of a Kubernetes cluster so that all pods have a place to run and there are no unneeded nodes.
Kubernetes Cluster Autoscaler is a tool that automatically adjusts the size of the worker’s node up or down depending on the consumption. This means that the cluster autoscaler, for example, automatically scale up a cluster by increasing the node count when there are not enough node resources for cluster workload scheduling and scale down when the node resources have continuously staying idle, or there are more than enough node resources available for cluster workload scheduling. In a nutshell, it is a component that automatically adjusts the size of a Kubernetes cluster so that all pods have a place to run and there are no unneeded nodes.

## KKP Cluster Autoscaler Usage

Expand Down Expand Up @@ -43,7 +43,7 @@ kube-proxy-tstvd 1/1 Running 0 21m
node-local-dns-4p8jr 1/1 Running 0 21m
```

As shown above, the autoscaler is not part of the running Kubernetes components within the namespace.
As shown above, the cluster autoscaler is not part of the running Kubernetes components within the namespace.

**Step 3**

Expand Down Expand Up @@ -82,7 +82,7 @@ coredns-666448b887-s8wv8 1/1 Running 0 36m
coredns-666448b887-vldzz 1/1 Running 0 36m
```

As shown above, the autoscaler has been provisioned and running.
As shown above, the cluster autoscaler has been provisioned and running.


## Annotating MachineDeployments for Autoscaling
Expand Down Expand Up @@ -171,7 +171,7 @@ Metadata:
……………………
```

As shown above, the MachineDeployment has been annotated with a minimum of 1 and a maximum of 5. Therefore, the autoscaler will consider only the annotated MachineDeployment on the cluster.
As shown above, the MachineDeployment has been annotated with a minimum of 1 and a maximum of 5. Therefore, the cluster autoscaler will consider only the annotated MachineDeployment on the cluster.

## Edit KKP Autoscaler

Expand All @@ -187,7 +187,16 @@ You can delete autoscaler from where you edit it above and select delete.
![Delete Autoscaler](images/delete-autoscaler.png?classes=shadow,border "Delete Autoscaler")


Once it has been deleted, you can check the cluster to ensure that the autoscaler has been deleted using the command `kubectl get pods -n kube-system`.
Once it has been deleted, you can check the cluster to ensure that the cluster autoscaler has been deleted using the command `kubectl get pods -n kube-system`.


## Customize KKP Autoscaler

You can customize the cluster autoscaler addon in order to override the cluster autoscaler deployment definition to set or pass the required flag(s) by following the instructions provided over [here](https://docs.kubermatic.com/kubermatic/v2.25/architecture/concept/kkp-concepts/addons/#custom-addons), if you are facing any of the below mentioned issues as described in the [cluster autoscaler FAQs](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md):

* [My cluster is below minimum / above maximum number of nodes, but CA did not fix that! Why?](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#my-cluster-is-below-minimum--above-maximum-number-of-nodes-but-ca-did-not-fix-that-why)

* [I'm running cluster with nodes in multiple zones for HA purposes. Is that supported by Cluster Autoscaler?](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler)


## Summary
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This section deals with the usage of Kubernetes Cluster Autoscaler in a KKP User

## What is a Cluster Autoscaler in Kubernetes?

Kubernetes Cluster Autoscaler is a tool that automatically adjusts the size of the worker’s node up or down depending on the consumption. This means that the autoscaler, for example, automatically scale up a cluster by increasing the node count when there are not enough node resources for cluster workload scheduling and scale down when the node resources have continuously staying idle, or there are more than enough node resources available for cluster workload scheduling. In a nutshell, it is a component that automatically adjusts the size of a Kubernetes cluster so that all pods have a place to run and there are no unneeded nodes.
Kubernetes Cluster Autoscaler is a tool that automatically adjusts the size of the worker’s node up or down depending on the consumption. This means that the cluster autoscaler, for example, automatically scale up a cluster by increasing the node count when there are not enough node resources for cluster workload scheduling and scale down when the node resources have continuously staying idle, or there are more than enough node resources available for cluster workload scheduling. In a nutshell, it is a component that automatically adjusts the size of a Kubernetes cluster so that all pods have a place to run and there are no unneeded nodes.

## KKP Cluster Autoscaler Usage

Expand Down Expand Up @@ -43,7 +43,7 @@ kube-proxy-tstvd 1/1 Running 0 21m
node-local-dns-4p8jr 1/1 Running 0 21m
```

As shown above, the autoscaler is not part of the running Kubernetes components within the namespace.
As shown above, the cluster autoscaler is not part of the running Kubernetes components within the namespace.

**Step 3**

Expand Down Expand Up @@ -82,7 +82,7 @@ coredns-666448b887-s8wv8 1/1 Running 0 36m
coredns-666448b887-vldzz 1/1 Running 0 36m
```

As shown above, the autoscaler has been provisioned and running.
As shown above, the cluster autoscaler has been provisioned and running.


## Annotating MachineDeployments for Autoscaling
Expand Down Expand Up @@ -171,7 +171,7 @@ Metadata:
……………………
```

As shown above, the MachineDeployment has been annotated with a minimum of 1 and a maximum of 5. Therefore, the autoscaler will consider only the annotated MachineDeployment on the cluster.
As shown above, the MachineDeployment has been annotated with a minimum of 1 and a maximum of 5. Therefore, the cluster autoscaler will consider only the annotated MachineDeployment on the cluster.

## Edit KKP Autoscaler

Expand All @@ -187,7 +187,16 @@ You can delete autoscaler from where you edit it above and select delete.
![Delete Autoscaler](/img/kubermatic/v2.25/tutorials/cluster-autoscaler/delete-autoscaler.png?classes=shadow,border "Delete Autoscaler")


Once it has been deleted, you can check the cluster to ensure that the autoscaler has been deleted using the command `kubectl get pods -n kube-system`.
Once it has been deleted, you can check the cluster to ensure that the cluster autoscaler has been deleted using the command `kubectl get pods -n kube-system`.


## Customize KKP Autoscaler
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as for the KubeOne document, I'd recommend using permalinks here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!


You can customize the cluster autoscaler addon in order to override the cluster autoscaler deployment definition to set or pass the required flag(s) by following the instructions provided over [here](https://docs.kubermatic.com/kubermatic/v2.25/architecture/concept/kkp-concepts/addons/#custom-addons), if you are facing any of the below mentioned issues as described in the [cluster autoscaler FAQs](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about something like this:

Suggested change
You can customize the cluster autoscaler addon in order to override the cluster autoscaler deployment definition to set or pass the required flag(s) by following the instructions provided over [here](https://docs.kubermatic.com/kubermatic/v2.25/architecture/concept/kkp-concepts/addons/#custom-addons), if you are facing any of the below mentioned issues as described in the [cluster autoscaler FAQs](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md):
You can customize the cluster autoscaler addon in order to override the cluster autoscaler deployment definition to set or pass the required flag(s) by following the instructions provided [in the Addons document](https://docs.kubermatic.com/kubermatic/v2.25/architecture/concept/kkp-concepts/addons/#custom-addons).
### Suggestions and Known Issues
Some of [the know issues in the cluster-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md) can be mitigated by adjusting the cluster-autoscaler deployment to pass additional flags, for example:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also please use ref to access other KKP documents, such as {{< ref "../../architecture/concept/kkp-concepts/addons/#custom-addons" >}} (I didn't test this URL, please make sure to test it and fix it if needed).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done!


* [My cluster is below minimum / above maximum number of nodes, but CA did not fix that! Why?](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#my-cluster-is-below-minimum--above-maximum-number-of-nodes-but-ca-did-not-fix-that-why)

* [I'm running cluster with nodes in multiple zones for HA purposes. Is that supported by Cluster Autoscaler?](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler)


## Summary
Expand Down