[Feature] Add the ability to set custom k3s agent args

[sambonbonne]

Hello,

Like we can set the k3s_exec_server_args variable for the k3s server command, I propose to add a k3s_exec_agent_args variable for the k3s agent command.

I tested it on fixed and autoscaler nodepools, with some kubelet-arg (to reserver resources for system and kube) and it works as expected.

Let me know if I can improve this PR!

[clementguillot]

Very good and useful feature, I hope it will be accepted!

[mysticaltech]

@sambonbonne @clementguillot This seems nice! Merci les amis.

@kube-hetzner/core For context:

You:

I am using k3s

Let's say I give this arg to k3s agent: --kubelet-arg kube-reserved=cpu=100m,memory=200Mi,ephemeral-storage=1Gi

And also have this kubelet_arg : "cloud-provider=external", "volume-plugin-dir=/var/lib/kubelet/volumeplugins" in the config.yaml file.

What will happen, what is the final kubelet-arg considered by k3s?

ChatGPT:

In k3s, you can specify kubelet arguments in two ways:

1. Inline with the command when starting the k3s server or agent using the --kubelet-arg command-line flag.
2. Within the kubelet_arg section of the config.yaml file.

In your case, you are providing kubelet arguments through both methods.

The way k3s processes these arguments is that it will combine both sets of arguments into a final set that gets passed to the kubelet. If there are conflicting arguments (i.e., the same argument specified in both the command line and the config file), the value from the command line takes precedence. This is typical behavior in many systems where command-line arguments override configuration file settings.

So, in your case:

• The argument kube-reserved=cpu=100m,memory=200Mi,ephemeral-storage=1Gi specified via the --kubelet-arg flag on the command line will be accepted.
• The arguments cloud-provider=external and volume-plugin-dir=/var/lib/kubelet/volumeplugins specified in the config.yaml file under kubelet_arg will also be accepted.

These arguments will be merged and passed to the kubelet. Remember, if there is a conflict between command line and config.yaml, the command-line argument will take precedence. In your case, as there are no conflicts, all your specified arguments will be considered by kubelet.

[mysticaltech]

Will leave it another day to give the chance for the team to comment if they want, but looks good to me.

[sambonbonne]

Tanks for the merge 🚀

[schlichtanders]

@sambonbonne a bit late to the game, I also wanted something similar, however need it slightly different, namely using double quotes instead of single quotes

-     "curl ... | ... INSTALL_K3S_EXEC='agent ${var.k3s_exec_agent_args}' sh -"
+     "curl ... | ... INSTALL_K3S_EXEC=\"agent ${var.k3s_exec_agent_args}\" sh -"

It would also be great to change it for k3s_exec_server_args, but that may break others configs... only in rare cases.

As this was just merged, I guess no one really uses this yet, so it could still be changed without effecting others.

Benefits:

• you can run arbitrary code when the agent starts
• see Automatically apply taint during node self-registering if IP is blacklisted #839 for one example case

What do you think?

[sambonbonne]

@schlichtanders I used single quotes because var.k3s_exec_server_args use single quotes too (see

terraform-hcloud-kube-hetzner/locals.tf

Line 52 in 357ff14

"curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_CHANNEL=${var.initial_k3s_channel} INSTALL_K3S_EXEC='server ${var.k3s_exec_server_args}' sh -"

).

I'm not really a maintainer of this repository (so don't take my words as an official answer) but I see two disadvantages by changing the single quotes to double quotes:

• if the behavior of k3s_exec_agent_args is changed, then we should change the behavior of k3s_exec_server_args in order to prevent confusion between the behavior of the two variables (which may break the configuration for some users)
• adding the ability to run arbitrary code seem very insecure to me (but it may not be a deal breaker)

Maybe a better solution would be to have a specific variable for arbitrary code?

[mysticaltech]

@schlichtanders Normally " within " will not work, it needs to be different kinds encompassing each other, but there is an alternative which is to use <<EOT ... EOT.

If done well, I do not think this will break anything, PR welcome, and FYI GPT-4 is quite good at this stuff lol (if you need a non-human hand).

And as @sambonbonne mentioned, if you do submit a PR, please make sure to harmozine k3s_exec_server_args with the changes too.