Skip to content
This repository has been archived by the owner on Jan 8, 2023. It is now read-only.

Fix the waiting for a rootfull container creation #83

Merged
merged 4 commits into from
Nov 2, 2021
Merged

Fix the waiting for a rootfull container creation #83

merged 4 commits into from
Nov 2, 2021

Conversation

0b11stan
Copy link
Contributor

@0b11stan 0b11stan commented Sep 14, 2021

It's a fix for #77.
You can try it with the following:

python -m venv venv
source venv/bin/activate
pip install 'molecule[lint]'
pip install 'git+https://github.com/0b11stan/molecule-podman.git@fix/wait-rootfull'
molecule init role -d podman poc && cd poc && rm -r meta
cat > molecule/default/molecule.yml <<EOF
---
dependency:
  name: galaxy
driver:
  name: podman
platforms:
  - name: instance
    image: docker.io/pycontribs/centos:8
    pre_build_image: true
    rootless: false
provisioner:
  name: ansible
  inventory:
    host_vars:
      localhost:
        ansible_become_password: ******
verifier:
  name: ansible
EOF
molecule create

However, the converge step for rootfull containers is still not working, I'll create a new issue for that.

@@ -178,6 +178,7 @@
changed_when: true

- name: Wait for instance(s) creation to complete
become: "{{ not item.item.rootless | default(omit) }}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be the ignorant here but my impression was that become was supposed to be used on looped command and not on the async_status, which I doubt it really needs root access to check already running tasks.

@ssbarnea ssbarnea added the bug This issue/PR relates to a bug. label Oct 27, 2021
Copy link
Member

@ssbarnea ssbarnea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not get any reply on my original question. If that becomes still for very long, I will need to close it.

@0b11stan
Copy link
Contributor Author

0b11stan commented Oct 31, 2021

I'm sorry for the late reply, I did not catch the github notification. I understand that my last explanation was too brief and unclear. I'll try to expose why I think the async_status really needs root access to check already running tasks.

The bug (tested from ansible-community/molecule-podman:main)

Here is an execution of a molecule -vvv create command on the main branch (up to date with upstream) :

TASK [Create molecule instance(s)] *********************************************
task path: /tmp/test/main/venv/lib/python3.9/site-packages/molecule_podman/playbooks/create.yml:142
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: tristan
<127.0.0.1> EXEC /bin/sh -c 'echo ~tristan && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/tristan/.ansible/tmp `"&& mkdir "` echo /home/tristan/.ansible/tmp/ansible-tmp-1635665533.767641-32637-252596935354781 `" && echo ansible-tmp-1635665533.767641-32637-252596935354781="` echo /home/tristan/.ansible/tmp/ansible-tmp-1635665533.767641-32637-252596935354781 `" ) && sleep 0'
Using module file /usr/lib/python3.9/site-packages/ansible/modules/command.py
<127.0.0.1> PUT /home/tristan/.ansible/tmp/ansible-local-323843y7d67j4/tmpdvueqt10 TO /home/tristan/.ansible/tmp/ansible-tmp-1635665533.767641-32637-252596935354781/AnsiballZ_command.py
<127.0.0.1> PUT /home/tristan/.ansible/tmp/ansible-local-323843y7d67j4/tmpbguxslsr TO /home/tristan/.ansible/tmp/ansible-tmp-1635665533.767641-32637-252596935354781/async_wrapper.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/tristan/.ansible/tmp/ansible-tmp-1635665533.767641-32637-252596935354781/ /home/tristan/.ansible/tmp/ansible-tmp-1635665533.767641-32637-252596935354781/AnsiballZ_command.py /home/tristan/.ansible/tmp/ansible-tmp-1635665533.767641-32637-252596935354781/async_wrapper.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=sffbhxyqozulkfvnvkbfxjrhwervrjze] password:" -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-sffbhxyqozulkfvnvkbfxjrhwervrjze ; ANSIBLE_ASYNC_DIR='"'"'"'"'"'"'"'"'~/.ansible_async'"'"'"'"'"'"'"'"' /usr/bin/python /home/tristan/.ansible/tmp/ansible-tmp-1635665533.767641-32637-252596935354781/async_wrapper.py 339676911481 7200 /home/tristan/.ansible/tmp/ansible-tmp-1635665533.767641-32637-252596935354781/AnsiballZ_command.py _'"'"' && sleep 0'
changed: [localhost] => (item=instance) => {
    "ansible_job_id": "339676911481.32650",
    "ansible_loop_var": "item",
    "changed": true,
    "finished": 0,
    "item": {
        "image": "docker.io/pycontribs/centos:8",
        "name": "instance",
        "pre_build_image": true,
        "rootless": false
    },
    "results_file": "/root/.ansible_async/339676911481.32650",
    "started": 1
}

TASK [Wait for instance(s) creation to complete] *******************************
task path: /tmp/test/main/venv/lib/python3.9/site-packages/molecule_podman/playbooks/create.yml:180
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: tristan
<127.0.0.1> EXEC /bin/sh -c 'echo ~tristan && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/tristan/.ansible/tmp `"&& mkdir "` echo /home/tristan/.ansible/tmp/ansible-tmp-1635665534.0982563-32657-135020011412783 `" && echo ansible-tmp-1635665534.0982563-32657-135020011412783="` echo /home/tristan/.ansible/tmp/ansible-tmp-1635665534.0982563-32657-135020011412783 `" ) && sleep 0'
Using module file /usr/lib/python3.9/site-packages/ansible/modules/async_status.py
<127.0.0.1> PUT /home/tristan/.ansible/tmp/ansible-local-323843y7d67j4/tmpxitbvlbz TO /home/tristan/.ansible/tmp/ansible-tmp-1635665534.0982563-32657-135020011412783/AnsiballZ_async_status.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/tristan/.ansible/tmp/ansible-tmp-1635665534.0982563-32657-135020011412783/ /home/tristan/.ansible/tmp/ansible-tmp-1635665534.0982563-32657-135020011412783/AnsiballZ_async_status.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /home/tristan/.ansible/tmp/ansible-tmp-1635665534.0982563-32657-135020011412783/AnsiballZ_async_status.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/tristan/.ansible/tmp/ansible-tmp-1635665534.0982563-32657-135020011412783/ > /dev/null 2>&1 && sleep 0'
failed: [localhost] (item=instance) => {
    "ansible_job_id": "339676911481.32650",
    "ansible_loop_var": "item",
    "attempts": 1,
    "changed": false,
    "finished": 1,
    "invocation": {
        "module_args": {
            "_async_dir": "/home/tristan/.ansible_async",
            "jid": "339676911481.32650",
            "mode": "status"
        }
    },
    "item": {
        "ansible_job_id": "339676911481.32650",
        "ansible_loop_var": "item",
        "changed": true,
        "failed": false,
        "finished": 0,
        "item": {
            "image": "docker.io/pycontribs/centos:8",
            "name": "instance",
            "pre_build_image": true,
            "rootless": false
        },
        "results_file": "/root/.ansible_async/339676911481.32650",
        "started": 1
    },
    "msg": "could not find job",
    "started": 1
}

This execution is failing with the error message: could not find job.

You see that for the Create molecule instance task, molecule is calling sudo (as expected for a rootfull container) :

/bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=sffbhxyqozulkfvnvkbfxjrhwervrjze] password:" -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-sffbhxyqozulkfvnvkbfxjrhwervrjze ; ANSIBLE_ASYNC_DIR='"'"'"'"'"'"'"'"'~/.ansible_async'"'"'"'"'"'"'"'"' /usr/bin/python /home/tristan/.ansible/tmp/ansible-tmp-1635665533.767641-32637-252596935354781/async_wrapper.py 339676911481 7200 /home/tristan/.ansible/tmp/ansible-tmp-1635665533.767641-32637-252596935354781/AnsiballZ_command.py _'"'"' && sleep 0'

And the ANSIBLE_ASYNC_DIR is set to ~/.ansible_async. As a consequence, the job description file ends in /root/.ansible_async :

"results_file": "/root/.ansible_async/339676911481.32650"

However, the Wait for instance task never uses sudo. Then, you can see in the module_args that it is looking for an async_dir under /home/tristan :

"module_args": {
    "_async_dir": "/home/tristan/.ansible_async",
    "jid": "339676911481.32650",
    "mode": "status"
}

But during this task execution, the item (from the server.results variable), still has it's task file pointing to /root/.ansible_async/339676911481.32650.

The fix (tested from 0b11stan/molecule-podman:fix/wait-rootfull)

That's why in my fix, I'm explicitly saying to the async_status module that I want to look for the task file in the /root/ dir.

Here is an execution of a molecule -vvv create command on the fix branch (up to date with upstream) :

TASK [Create molecule instance(s)] *********************************************
task path: /tmp/test/fix/venv/lib/python3.9/site-packages/molecule_podman/playbooks/create.yml:142
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: tristan
<127.0.0.1> EXEC /bin/sh -c 'echo ~tristan && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/tristan/.ansible/tmp `"&& mkdir "` echo /home/tristan/.ansible/tmp/ansible-tmp-1635666625.5814912-53263-187304956210369 `" && echo ansible-tmp-1635666625.5814912-53263-187304956210369="` echo /home/tristan/.ansible/tmp/ansible-tmp-1635666625.5814912-53263-187304956210369 `" ) && sleep 0'
Using module file /usr/lib/python3.9/site-packages/ansible/modules/command.py
<127.0.0.1> PUT /home/tristan/.ansible/tmp/ansible-local-53011l872041p/tmpe7czgq8y TO /home/tristan/.ansible/tmp/ansible-tmp-1635666625.5814912-53263-187304956210369/AnsiballZ_command.py
<127.0.0.1> PUT /home/tristan/.ansible/tmp/ansible-local-53011l872041p/tmp_c0zkpya TO /home/tristan/.ansible/tmp/ansible-tmp-1635666625.5814912-53263-187304956210369/async_wrapper.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/tristan/.ansible/tmp/ansible-tmp-1635666625.5814912-53263-187304956210369/ /home/tristan/.ansible/tmp/ansible-tmp-1635666625.5814912-53263-187304956210369/AnsiballZ_command.py /home/tristan/.ansible/tmp/ansible-tmp-1635666625.5814912-53263-187304956210369/async_wrapper.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=phasvamjwpwlrxgilbwujjxtahosrxja] password:" -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-phasvamjwpwlrxgilbwujjxtahosrxja ; ANSIBLE_ASYNC_DIR='"'"'"'"'"'"'"'"'~/.ansible_async'"'"'"'"'"'"'"'"' /usr/bin/python /home/tristan/.ansible/tmp/ansible-tmp-1635666625.5814912-53263-187304956210369/async_wrapper.py 954204873193 7200 /home/tristan/.ansible/tmp/ansible-tmp-1635666625.5814912-53263-187304956210369/AnsiballZ_command.py _'"'"' && sleep 0'
changed: [localhost] => (item=instance) => {
    "ansible_job_id": "954204873193.53278",
    "ansible_loop_var": "item",
    "changed": true,
    "finished": 0,
    "item": {
        "image": "docker.io/pycontribs/centos:8",
        "name": "instance",
        "pre_build_image": true,
        "rootless": false
    },
    "results_file": "/root/.ansible_async/954204873193.53278",
    "started": 1
}

TASK [Wait for instance(s) creation to complete] *******************************
task path: /tmp/test/fix/venv/lib/python3.9/site-packages/molecule_podman/playbooks/create.yml:180
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: tristan
<127.0.0.1> EXEC /bin/sh -c 'echo ~tristan && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/tristan/.ansible/tmp `"&& mkdir "` echo /home/tristan/.ansible/tmp/ansible-tmp-1635666625.9340813-53285-226305867506300 `" && echo ansible-tmp-1635666625.9340813-53285-226305867506300="` echo /home/tristan/.ansible/tmp/ansible-tmp-1635666625.9340813-53285-226305867506300 `" ) && sleep 0'
Using module file /usr/lib/python3.9/site-packages/ansible/modules/async_status.py
<127.0.0.1> PUT /home/tristan/.ansible/tmp/ansible-local-53011l872041p/tmpbhs54y8q TO /home/tristan/.ansible/tmp/ansible-tmp-1635666625.9340813-53285-226305867506300/AnsiballZ_async_status.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/tristan/.ansible/tmp/ansible-tmp-1635666625.9340813-53285-226305867506300/ /home/tristan/.ansible/tmp/ansible-tmp-1635666625.9340813-53285-226305867506300/AnsiballZ_async_status.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=ypgqrabzafrjjvaterlaxbutdkvdwapn] password:" -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-ypgqrabzafrjjvaterlaxbutdkvdwapn ; /usr/bin/python /home/tristan/.ansible/tmp/ansible-tmp-1635666625.9340813-53285-226305867506300/AnsiballZ_async_status.py'"'"' && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/tristan/.ansible/tmp/ansible-tmp-1635666625.9340813-53285-226305867506300/ > /dev/null 2>&1 && sleep 0'
FAILED - RETRYING: Wait for instance(s) creation to complete (300 retries left).Result was: {
    "ansible_job_id": "954204873193.53278",
    "attempts": 1,
    "changed": false,
    "finished": 0,
    "invocation": {
        "module_args": {
            "_async_dir": "/root/.ansible_async",
            "jid": "954204873193.53278",
            "mode": "status"
        }
    },
    "retries": 301,
    "started": 1
}
<127.0.0.1> EXEC /bin/sh -c 'echo ~tristan && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/tristan/.ansible/tmp `"&& mkdir "` echo /home/tristan/.ansible/tmp/ansible-tmp-1635666631.2740889-53285-82026741475788 `" && echo ansible-tmp-1635666631.2740889-53285-82026741475788="` echo /home/tristan/.ansible/tmp/ansible-tmp-1635666631.2740889-53285-82026741475788 `" ) && sleep 0'
Using module file /usr/lib/python3.9/site-packages/ansible/modules/async_status.py
<127.0.0.1> PUT /home/tristan/.ansible/tmp/ansible-local-53011l872041p/tmp0zptsz7s TO /home/tristan/.ansible/tmp/ansible-tmp-1635666631.2740889-53285-82026741475788/AnsiballZ_async_status.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/tristan/.ansible/tmp/ansible-tmp-1635666631.2740889-53285-82026741475788/ /home/tristan/.ansible/tmp/ansible-tmp-1635666631.2740889-53285-82026741475788/AnsiballZ_async_status.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'sudo -H -S  -p "[sudo via ansible, key=lhlcohgtoqqptsljmsmcqtjzcwrhfizj] password:" -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-lhlcohgtoqqptsljmsmcqtjzcwrhfizj ; /usr/bin/python /home/tristan/.ansible/tmp/ansible-tmp-1635666631.2740889-53285-82026741475788/AnsiballZ_async_status.py'"'"' && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/tristan/.ansible/tmp/ansible-tmp-1635666631.2740889-53285-82026741475788/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => (item=instance) => {
    "ansible_job_id": "954204873193.53278",
    "ansible_loop_var": "item",
    "attempts": 2,
    "changed": true,
    "cmd": [
        "/usr/bin/podman",
        "run",
        "-d",
        "--name",
        "instance",
        "--hostname=instance",
        "docker.io/pycontribs/centos:8",
        "bash",
        "-c",
        "while true; do sleep 10000; done"
    ],
    "delta": "0:00:00.275167",
    "end": "2021-10-31 08:50:26.256434",
    "finished": 1,
    "invocation": {
        "module_args": {
            "_raw_params": "/usr/bin/podman    run -d --name \"instance\"                  --hostname=instance  docker.io/pycontribs/centos:8 bash -c \"while true; do sleep 10000; done\"\n",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": false
        }
    },
    "item": {
        "ansible_job_id": "954204873193.53278",
        "ansible_loop_var": "item",
        "changed": true,
        "failed": false,
        "finished": 0,
        "item": {
            "image": "docker.io/pycontribs/centos:8",
            "name": "instance",
            "pre_build_image": true,
            "rootless": false
        },
        "results_file": "/root/.ansible_async/954204873193.53278",
        "started": 1
    },
    "msg": "",
    "rc": 0,
    "start": "2021-10-31 08:50:25.981267",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "4ac9830d0d419a62535e526948f53962941d51d2e6ff683302c072b90204f6cd",
    "stdout_lines": [
        "4ac9830d0d419a62535e526948f53962941d51d2e6ff683302c072b90204f6cd"
    ]
}

You can see that now the Wait for instance task is running sudo and starts with the following module arguments:

"module_args": {
    "_async_dir": "/root/.ansible_async",
    "jid": "954204873193.53278",
    "mode": "status"
}

Because the _async_dir in async_status's module arguments is matching the result_file's path in it's item (result of the command module), it can find the job file and do not end with could not find job error message.

Resulting in a successfull execution.

I hope it's answering your question and that I did not miss anything. However, I'm not an expert in how async_status and command "jobs" works so tell me if you see an other way to solve this issue.

@ssbarnea ssbarnea merged commit b4da8ca into ansible-community:main Nov 2, 2021
@0b11stan 0b11stan deleted the fix/wait-rootfull branch November 2, 2021 13:58
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug This issue/PR relates to a bug.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants