Unable to pull container images from insecure registry

[jpm38]

Environment:

• Cloud provider or hardware configuration:
  bare metal

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
  cat /etc/redhat-release
  Red Hat Enterprise Linux release 8.4 (Ootpa)

• Version of Ansible (ansible --version):
  ansible 2.9.25

• Version of Python (python --version):
  Python 3.6.8

Kubespray version (commit) (git rev-parse --short HEAD):
kubespray-2.18.1

Network plugin used:
flannel

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

Command used to invoke ansible:

Output of ansible run:

Anything else do we need to know:

I want to deploy a kubernetes cluster in offline mode, using containerd as CRI.
I have deployed a docker registry in insecure mode on my management node name "cypher1" and then configure in kubespray inventory :

inventory/cluster1/group_vars/all/containerd.yml:
...
containerd_insecure_registries:
"cypher1:5000": "cypher1:5000"

When deploying my cluster using: "ansible-playbook cluster.yml -i inventory/cluster1/cluster.ini"

I'm facing the following error:

TASK [download : debug] *******************************************************************************************************************************************************************************************
ok: [cypher2] => {
"msg": "Pull cypher1:5000/coreos/flannel:v0.15.1-amd64 required is: True"
}
ok: [cypher3] => {
"msg": "Pull cypher1:5000/coreos/flannel:v0.15.1-amd64 required is: True"
}
ok: [cypher4] => {
"msg": "Pull cypher1:5000/coreos/flannel:v0.15.1-amd64 required is: True"
}
ok: [cypher5] => {
"msg": "Pull cypher1:5000/coreos/flannel:v0.15.1-amd64 required is: True"
}
ok: [cypher6] => {
"msg": "Pull cypher1:5000/coreos/flannel:v0.15.1-amd64 required is: True"
}
ok: [cypher7] => {
"msg": "Pull cypher1:5000/coreos/flannel:v0.15.1-amd64 required is: True"
}
Monday 11 April 2022 15:56:34 +0200 (0:00:00.111) 0:03:13.088 **********
Monday 11 April 2022 15:56:34 +0200 (0:00:00.160) 0:03:13.248 **********
Monday 11 April 2022 15:56:35 +0200 (0:00:00.081) 0:03:13.330 **********
Monday 11 April 2022 15:56:35 +0200 (0:00:00.086) 0:03:13.417 **********
FAILED - RETRYING: download_container | Download image if required (4 retries left).
FAILED - RETRYING: download_container | Download image if required (4 retries left).
FAILED - RETRYING: download_container | Download image if required (4 retries left).
FAILED - RETRYING: download_container | Download image if required (4 retries left).
FAILED - RETRYING: download_container | Download image if required (4 retries left).
FAILED - RETRYING: download_container | Download image if required (3 retries left).
FAILED - RETRYING: download_container | Download image if required (3 retries left).
FAILED - RETRYING: download_container | Download image if required (3 retries left).
FAILED - RETRYING: download_container | Download image if required (2 retries left).
FAILED - RETRYING: download_container | Download image if required (3 retries left).
FAILED - RETRYING: download_container | Download image if required (3 retries left).
FAILED - RETRYING: download_container | Download image if required (1 retries left).
FAILED - RETRYING: download_container | Download image if required (2 retries left).
FAILED - RETRYING: download_container | Download image if required (2 retries left).

TASK [download_container | Download image if required] ************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ValueError: No closing quotation
fatal: [cypher2 -> cypher2]: FAILED! => {"attempts": 4, "changed": false, "module_stderr": "Traceback (most recent call last):\n File "", line 102, in \n File "", line 94, in _ansiballz_main\n File "", line 40, in invoke_module\n File "/usr/lib64/python3.6/runpy.py", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib64/python3.6/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_command_payload_pum5b0jo/ansible_command_payload.zip/ansible/modules/commands/command.py", line 344, in \n File "/tmp/ansible_command_payload_pum5b0jo/ansible_command_payload.zip/ansible/modules@@@

Looking into the code, the fixe is quite obvious:

change the following line:

defaults/main.yml:nerdctl_extra_flags: '{%- if containerd_insecure_registries is defined and containerd_insecure_registries|length>0 -%} \" --insecure-registry"{%- else -%}{%- endif -%}'

to

defaults/main.yml:nerdctl_extra_flags: '{%- if containerd_insecure_registries is defined and containerd_insecure_registries|length>0 -%} --insecure-registry{%- else -%}{%- endif -%}'

[cristicalin]

This should have been fixed by #8695 and #8699

[oomichi]

This should have been fixed by #8695 and #8699

Yeah, @cristicalin is right.

#8699 is the same as the suggested change in this issue message, and the pull request has been merged.
Feel free to re-open if still facing the same issue on release-2.18 branch.

/close

[k8s-ci-robot]

@oomichi: Closing this issue.

In response to this:

This should have been fixed by #8695 and #8699

Yeah, @cristicalin is right.

#8699 is the same as the suggested change in this issue message, and the pull request has been merged.
Feel free to re-open if still facing the same issue on release-2.18 branch.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.