Removing a node executes network_plugin/cni dependency when it should not

[jwitko]

Environment:

• Cloud provider or hardware configuration:
  Bare metal, standard hardware x86_64

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
  Alma 9

• Version of Ansible (ansible --version):

ansible [core 2.12.5]
  config file = /home/jwitkowski/ETO/et-k8s-logging-zone/kubespray/kubespray/ansible.cfg
  configured module search path = ['/home/jwitkowski/ETO/et-k8s-logging-zone/kubespray/kubespray/library']
  ansible python module location = /home/jwitkowski/.venvs/et-k8s-logging-zone/lib/python3.8/site-packages/ansible
  ansible collection location = /home/jwitkowski/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/jwitkowski/.venvs/et-k8s-logging-zone/bin/ansible
  python version = 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
  jinja version = 2.11.3
  libyaml = True

• Version of Python (python --version):

Python 3.8.10

Kubespray version (commit) (git rev-parse --short HEAD):
v2.20.0

Network plugin used:
cilium

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

Command used to invoke ansible:
Just ran remove-node.yml

Output of ansible run:

TASK [network_plugin/cni : CNI | Copy cni plugins] **************************************************************************************************************************************************************
fatal: [zpa-ze2101-24.sjc4.prod.zpath.net]: FAILED! => {"changed": false, "msg": "Source '/tmp/releases/cni-plugins-linux-amd64-v1.1.1.tgz' does not exist"}

Anything else do we need to know:

Running remove-node.yml does an include_role: on network_plugin/<your-network-plugin> with a tasks_from: reset. All of the potential roles that can be loaded include a dependency for network_plugin/cni which will fail because the download role is not included in this playbook (nor should it be).

[yankay]

HI @jwitko

I try to reproduce the issue, but cannot.

Would you please upload the full logs.

[jwitko]

@yankay I do not have the full logs, I did not save them. I would image your test worked because the nodes were fresh and had not cleared the default releases_cache directory of local_release_dir: "/tmp/releases" yet? If that is the case then the task https://github.com/kubernetes-sigs/kubespray/blob/master/roles/network_plugin/cni/tasks/main.yml#L10 would still work. If this directory has been emptied by the system (which happens at various stages in various methods across different OS') then this task would fail because that file would no longer be there.

I'm not sure how you tried to reproduce but a valid test case would be:

1. Deploy a kubespray cluster with any amount of nodes
2. Delete the contents of /tmp/releases/*
3. Run a standard node removal on any node where /tmp/releases/* was cleared out

[yankay]

Thanks @jwitko
I have reproduce the issue with your suggestion :-)
And there has a CI #9562 for the issue now.