Longhorn dependencies missing, flexible way to define extra packages to install on worker nodes

[acdoussan]

FYI: This is probably more a feature request than a bug

Expected Behavior

When installing longhorn on a cluster using helm and flux, the install works

Current Behavior

https://forums.rancher.com/t/k8s-rancheros-longhorn/9919/2

the longhorn-manager pods throw the above error

Steps to Reproduce

1. Create a new cluster
2. Install flux
3. try to install longhorn, get install failed

Context (variables)

Operating system: macOS

Hardware: n/a

Variables Used

n/a

all.yml

k3s_version: ""
ansible_user: NA
systemd_dir: ""

flannel_iface: ""

apiserver_endpoint: ""

k3s_token: "NA"

extra_server_args: ""
extra_agent_args: ""

kube_vip_tag_version: ""

metal_lb_speaker_tag_version: ""
metal_lb_controller_tag_version: ""

metal_lb_ip_range: ""

Hosts

host.ini

[master]
IP.ADDRESS.ONE
IP.ADDRESS.TWO
IP.ADDRESS.THREE

[node]
IP.ADDRESS.FOUR
IP.ADDRESS.FIVE

[k3s_cluster:children]
master
node

Possible Solution

I went onto the hosts and ran apt-get install open-iscsi and the pods started to launch successfully.

I am thinking that it's probably a good idea to create a new variable that is a list of additional packages to install when provisioning the cluster, maybe as part of the prereq tasks? Should probably only be for worker nodes. Just want to make sure you think this belongs here before I bother thinking more deeply about this.

• I've checked the General Troubleshooting Guide

[timothystewart6]

Hey, thanks! I am aware of this and yeah, this is required to run Longhorn. Unfortunately I don't want to add too many dependencies for things downstream (items you install after). You can see how this will get out of handle really quickly. Here are the 2 playbooks I have to install Longhorn dependencies:

open-scsi.yaml

---
- name: Install latest open-iscsi on all hosts
  hosts: "*"
  tasks:
    - name: Install open-iscsi
      apt:
        name: open-iscsi
        state: present
        update_cache: true
      become: true

(optional if you are going to use nfs with Longhorn)

nfs-common.yaml

---
- name: Install latest nfs-common on all hosts
  hosts: "*"
  tasks:
    - name: Install nfs-common
      apt:
        name: nfs-common
        state: present
        update_cache: true
      become: true

I hope this helps!

[acdoussan]

Yeah, totally agree they definitely shouldn't be installed by default. I was more thinking along the lines of a task that just passes a variable to ansibles package builtin. This way the user of the playbook decides what extra packages to install on their cluster worker nodes, or if they don't want any at all they can just leave the variable as an empty array.

https://docs.ansible.com/ansible/latest/collections/ansible/builtin/package_module.html

Side thought, whats nfs-common needed for? like making disks available over nfs? Maybe I should just go poke around on the longhorn docs :P

[acdoussan]

oh, probably for the backups huh

[sleiner]

In general, if you need some tasks before or after the cluster setup of this playbook, why don't you use regular Ansible tasks and roles? There, you get the full power and flexibility of what Ansible can do: Installing software, writing config files, whatever 😄

In contrast, including a single task for package installation seems to me like an overly specific thing to add to this playbook.

[acdoussan]

Why should the users of this playbook have to figure it out themselves, instead of the playbook making it easy for them, when we know this is a common use case?

I'm not saying individual tasks for nfs or open iscsi, this is pretty much what im thinking

all.yml

extra_worker_packages:
- open-iscsi # for longhorn
- nfs-common # for longhorn backups

and then in one of the task files

- name: install extra worker dependencies
  package:
    name: "{{ extra_worker_packages }}"
    state: present

The default example would just have an empty list for extra_worker_packages, and users can decide what extra dependencies they need on their nodes

In contrast, including a single task for package installation seems to me like an overly specific thing to add to this playbook.

This playbook is meant to set up a k3s cluster that is ready for my workloads, right? I wouldn't classify a cluster as ready if it is missing dependencies I know that I need.

[sleiner]

Why should the users of this playbook have to figure it out themselves, instead of the playbook making it easy for them, when we know this is a common use case?

I do not know how common this is (as far as I am aware, you are at least the first one to ask here). I would agree that having some sort of surrounding tasks is likely, but and what that will be may be different for everyone (e.g. installing some package, applying some manifest, writing some configuration, executing some command, ...).

---

This playbook is meant to set up a k3s cluster that is ready for my workloads, right?

I think this is the core point of our discussion: I don't agree with this statement. My assumption of this project's objective is a bit more narrow and better described by the first sentence of the current README file:

This playbook will build an HA Kubernetes cluster with k3s, kube-vip and MetalLB via ansible.

IMO, the objective should not be:

Set up everything needed in order for me (or you) to start deploying their workloads, whatever those may be.

If that was the objective, I would agree with you that we should add something like this. But then we would also need to add a generic way to apply some manifests (maybe you need some CRDs before adding your workload?), add some parameter to a config file, execute some command on the shell, .... Also: Does the order of those actions matter now? (e.g. do we run the custom command before or after installing the packages) If we aim to satisfy all of those by explicit features of this project, we will end up building a worse version of Ansible on top of Ansible.

To come back to the objective: I think, designing something that sets up my cluster (to whatever extent) is my responsibility and the same thing applies to you. This project can be a (large) part of that by setting up a HA k3s setup, but everything beyond its scope is my responsibility again. To empower users, the project should take care to compose well with other things a user might have or build and possibly provide some guidance (docs, blog posts, ...) that show how you can integrate it nicely into your custom workflow and infrastructure.

[acdoussan]

I understand your concern about scope creep, but I don't think this is anywhere near as dramatic of a stepping stone as you make it out to be. One could make the same argument that you're "trying to reimplement the underlying technology" about any new framework or feature added to an existing framework. I think we should focus on what the best experience is for the user and work backward from that.

This playbook sets up the hosts to run a Kubernetes cluster, installing dependencies on these hosts is a normal thing to need to do. IMO having a separate playbook/role that just does this is overly specific and awkward. I don't want to have to run two different playbooks to set up my cluster, I just want to run one and then start deploying my applications.

If I need to make a change to a framework or library that I'm using, I default to contributing it back. I'm going to implement the above this weekend. @timothystewart6 would you accept a PR for what has been outlined, or should I just throw it on my personal branch on my fork and move on?

[hdensity]

Sorry to bud in on this discussion. Just one question: does a general requirement exist to install additional node packages, or is it specifically related to Longhorn?

If it's the latter, you can install the packages using the official helm charts available here: https://github.com/longhorn/longhorn/tree/master/deploy/prerequisite

As an alternative to this proposal, we could add an empty (and then git-ignored) playbook, where the user can add additional steps. Or a playbook which copies all files (e.g. helm charts, HelmChart(Config) files, etc.) from a directory to the first master. Just an idea ;)

[acdoussan]

Didn't know those were an option, interesting.

My knee-jerk reaction was that if it's possible for longhorn it's probable for other services, especially custom workloads. I honestly do not have a lot of experience with k3s yet though, if someone else knows better.

I was honestly pretty surprised that the service can even tell what is installed on the host, feels like it breaks the whole containerization point of docker. My longhorn install does use helm though, I just don't have those other helm charts, and honestly, I don't know how I feel about having a helm chart modify the underlying system like that as a default solution.

[hdensity]

That wasn't a knee-jerk reaction, it's a good idea. And I don't know enough about k3s do be able to foresee other use-cases where additional node packages are required.

And I understand your reservations as well. Some similar approaches exist though, e.g. flannel installed from a chart on k8s, requiring iptables. (btw. in a rootless installation of k3s, those charts would fail. On the other hand this is not recommended, experimental and doesn't apply to k3s-ansible anyway, just an fyi).

My perspective is the following:

• those packages are not required to run k3s
• some packages require additional setup, e.g. iscsi (enabling and starting a service, a call to modprobe)
• helm charts (can) setup everything required to run an application (rbac rules, deployments, daemon sets, config maps, etc. pp.) and apparently install package requirements as well.
• the charts i mentioned above are well maintained and officially supported, less work for the user

So to me, your proposal is very interesting, but doesn't cover it's original intent completely (installation but no configuration of iscsi and nfs). And imho, that's outside of the scope of k3s-ansible.

Still, this is just an opinion. What do you think?

[acdoussan]

I just wanted to add some closing thoughts while this is still relatively fresh for anyone else that comes along.

The premise @sleiner proposed of just having your own tasks is on the right track, however, it's not easily accomplishable with the current setup without some drawbacks to consider. The following options are assuming you make your changes on your own branch so that you can pull the latest changes from upstream and then rebase off of them. These options are:

1. Add your own playbook file similar to site.yml and reset.yml, and define your custom tasks there

Pros:

• simple, shouldn't cause any conflicts

Cons:

• you now have to run two ansible commands to set up your environment, one to install k3s, another to run your custom tasks
  • You could solve this by creating an aggregate playbook but now you are at risk of having the interface change for the project and needing to think about how to update this file to accommodate for that (IMO, pretty low chance here)

2. Create your own role and add it to the main playbook / just add tasks to the main playbook

Pros:

• even simpler, only one command to run and it matches the upstream

Cons:

• you are now at risk of merge conflicts or similar interface changes to the first option when pulling updates

3. Create a PR to add a new task to the main playbook to include an "extra" tasks file

Pros:

• should avoid conflicts altogether, the file name is designated and no one should really mess with that

Cons:

• more work
• needs to be tested with molecule, I've found this "framework" to be pretty convoluted and hard to work with / understand
• need to make sure this change will be accepted, or risk wasted time

4. Implement #198

Pros:

• The roles are now the project interface, and you can create your own playbook rather than forking this one, and not worry at all about conflicts.
• Updates are just pulled via ansible galaxy, and you can actually fix yourself to an older version if a breaking change is made that you don't want to deal with.

Cons:

• a lot more work
• need to either create your own ansible galaxy account and publish it under that or work with @timothystewart6 to get the right values to publish it under his
• need to set up some sort of release process with versioning, and new changes would probably need to have more thought into backward compatibility
• need some new project template for people to get started from scratch, since you are no longer forking the project

5. Use the prerequisite charts @hdensity pointed out / create your own k3s resources to install dependencies

https://github.com/longhorn/longhorn/tree/master/deploy/prerequisite

I don't really view this as an option, ansible is meant for managing hosts and k3s is meant for managing applications. This is sort of a grey mix of the two, and while I don't like it, maybe you think differently.

---

If I were willing to put in the work, option 4 is probably the best path, but for now, I'm probably going to go with option 1. Hope this helps 🙂

[sleiner]

Thanks for the overview, @acdoussan! :-) I would like to add another option which I have been using for a few months now and that I had briefly mentioned in another discussion:

6. Using git submodules

Have an own repository (with no fork relation to this one) which includes your own playbook - then include the k3s-ansible playbook as a git submodule and use Ansible's import_playbook to include all of the tasks from this repo's plays, e.g.:

# You can add any tasks or roles before...
- name: Linux basics
  hosts: linux
  roles:
    - role: nice_shell

- name: Set up k3s cluster
  ansible.builtin.import_playbook: ../k3s-ansible/site.yaml

# ... or after
- name: Linux basics
  hosts: k3s-runner
  roles:
    - role: longhorn

Pros:

• Clear separation between k3s-ansible and your own code
• Merge conflicts between k3s-ansible and your code cannot happen
• Easy to try out patches to k3s-ansible (e.g. no need to release anything to Ansible Galaxy etc.)
• You are (by design of Git submodules) locking the exact version of k3s-ansible that you want to use at any time
• Lots of customization and integration into your own workflows is possible (e.g. there is no need to run two Ansible commands in order to set up a cluster exactly as you want it)

Cons:

• This is approach uses quite a few advanced features, so it might take some time to wrap your head around it. Other approaches might be more idiomatic in Ansible.
• You need to deal with Git submodules. I would also call those an "advanced feature" of Git and they definitely have some usability problems.

Overall, this approach has taken me a bit time to come up with and set up properly, but after that it was really easy to keep up with upstream changes and just overall quite stable and ergonomic (to me).