Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

molecule fails in GitHub actions with: ansible_compat.errors.InvalidPrerequisiteError: Collection 'community.docker' not found in '['/home/runner/.ansible/collections', '/usr/share/ansible/collections']' #4017

Closed
wookietreiber opened this issue Aug 16, 2023 · 17 comments · Fixed by #4022
Labels

Comments

@wookietreiber
Copy link

Issue Type

  • Bug report

Molecule and Ansible details

ansible [core 2.15.3]
  config file = None
  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/ansible
  ansible collection location = /home/runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/hostedtoolcache/Python/3.11.4/x64/bin/ansible
  python version = 3.11.4 (main, Jun  7 2023, 11:01:02) [GCC 11.3.0] (/opt/hostedtoolcache/Python/3.11.4/x64/bin/python)
  jinja version = 3.1.2
  libyaml = True
molecule 6.0.0 using python 3.11 
    ansible:2.15.3
    azure:23.5.0 from molecule_plugins
    containers:23.5.0 from molecule_plugins requiring collections: ansible.posix>=1.3.0 community.docker>=1.9.1 containers.podman>=1.8.1
    default:6.0.0 from molecule
    docker:23.5.0 from molecule_plugins requiring collections: community.docker>=3.0.2 ansible.posix>=1.4.0
    ec2:23.5.0 from molecule_plugins
    gce:23.5.0 from molecule_plugins requiring collections: google.cloud>=1.0.2 community.crypto>=1.8.0
    podman:23.5.0 from molecule_plugins requiring collections: containers.podman>=1.7.0 ansible.posix>=1.3.0
    vagrant:23.5.0 from molecule_plugins

Molecule installation method (one of):

  • pip

Ansible installation method (one of):

  • pip

Detail any linters or test runners used: I don't understand what is meant by this. I have made a branch of my role so you can look at the molecule and GitHub action configuration here.

Desired Behavior

It should just work.

Actual Behaviour

Molecule default > dependency
  INFO     Running from /home/runner/work/ansible-role-systemd-timesyncd/ansible-role-systemd-timesyncd : ansible-galaxy collection install -vvv community.docker:>=3.0.2
  CRITICAL Collection 'community.docker' not found in '['/home/runner/.ansible/collections', '/usr/share/ansible/collections']'
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.4/x64/bin/molecule", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/command/test.py", line 113, in test
    base.execute_cmdline_scenarios(scenario_name, args, command_args, ansible_args)
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/command/base.py", line 123, in execute_cmdline_scenarios
    execute_scenario(scenario)
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/command/base.py", line 166, in execute_scenario
    execute_subcommand(scenario.config, action)
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/command/base.py", line 156, in execute_subcommand
    return command(config).execute(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/logger.py", line 189, in wrapper
    rt = func(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/logger.py", line 100, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/command/dependency.py", line 40, in execute
    self._config.dependency.execute()
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/dependency/ansible_galaxy/__init__.py", line 95, in execute
    invoker.execute()
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/dependency/ansible_galaxy/base.py", line 115, in execute
    super().execute()
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/dependency/base.py", line 91, in execute
    self._config.runtime.require_collection(name, version)
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/ansible_compat/runtime.py", line 721, in require_collection
    self.require_collection(name=name, version=version, install=False)
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/ansible_compat/runtime.py", line 725, in require_collection
    raise InvalidPrerequisiteError(msg)
ansible_compat.errors.InvalidPrerequisiteError: Collection 'community.docker' not found in '['/home/runner/.ansible/collections', '/usr/share/ansible/collections']'

For the full output of the action/workflow, see here.

Notes

I've tried around a bit with (neither helped):

  • adding ansible-galaxy collection install community.docker to the action, output said it was already installed (probably via pip3 install ansible from the action)

  • adding a requirements.yml including the community.docker collection and

    dependency:
      name: galaxy
      options:
        requirements-file: requirements
    

    which didn't help because that would require molecule to start which it doesn't


I don't know if I'm doing something wrong here or if that's a bug with the recent 6.x release. I've had this setup with a few other roles before and it worked, likely with 5.x.

@wookietreiber
Copy link
Author

wookietreiber commented Aug 16, 2023

Also tried with explicitly staying at 5:

pip3 install ansible "molecule>=5,<6" molecule-plugins[docker] docker
ansible [core 2.15.3]
  config file = None
  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/ansible
  ansible collection location = /home/runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/hostedtoolcache/Python/3.11.4/x64/bin/ansible
  python version = 3.11.4 (main, Jun  7 2023, 11:01:02) [GCC 11.3.0] (/opt/hostedtoolcache/Python/3.11.4/x64/bin/python)
  jinja version = 3.1.2
  libyaml = True
molecule 5.1.0 using python 3.11 
    ansible:2.15.3
    azure:23.5.0 from molecule_plugins
    containers:23.5.0 from molecule_plugins requiring collections: ansible.posix>=1.3.0 community.docker>=1.9.1 containers.podman>=1.8.1
    delegated:5.1.0 from molecule
    docker:23.5.0 from molecule_plugins requiring collections: community.docker>=3.0.2 ansible.posix>=1.4.0
    ec2:23.5.0 from molecule_plugins
    gce:23.5.0 from molecule_plugins requiring collections: google.cloud>=1.0.2 community.crypto>=1.8.0
    podman:23.5.0 from molecule_plugins requiring collections: containers.podman>=1.7.0 ansible.posix>=1.3.0
    vagrant:23.5.0 from molecule_plugins

same issue, so maybe molecule-plugins is at fault?

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.4/x64/bin/molecule", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/command/test.py", line 113, in test
    base.execute_cmdline_scenarios(scenario_name, args, command_args, ansible_args)
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/command/base.py", line 119, in execute_cmdline_scenarios
    execute_scenario(scenario)
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/command/base.py", line 162, in execute_scenario
    execute_subcommand(scenario.config, action)
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/command/base.py", line 152, in execute_subcommand
    return command(config).execute(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/logger.py", line 189, in wrapper
    rt = func(*args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/logger.py", line 100, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/command/dependency.py", line 40, in execute
    self._config.dependency.execute()
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/dependency/ansible_galaxy/__init__.py", line 86, in execute
    invoker.execute()
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/dependency/ansible_galaxy/base.py", line 118, in execute
    super().execute()
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/molecule/dependency/base.py", line 89, in execute
    self._config.runtime.require_collection(name, version)
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/ansible_compat/runtime.py", line 710, in require_collection
    self.require_collection(name=name, version=version, install=False)
  File "/opt/hostedtoolcache/Python/3.11.4/x64/lib/python3.11/site-packages/ansible_compat/runtime.py", line 714, in require_collection
    raise InvalidPrerequisiteError(msg)
ansible_compat.errors.InvalidPrerequisiteError: Collection 'community.docker' not found in '['/home/runner/.cache/ansible-compat/8ed492/collections', '/home/runner/.cache/ansible-compat/8ed492/collections', '/home/runner/.ansible/collections', '/usr/share/ansible/collections']'

@jason-riddle
Copy link

jason-riddle commented Aug 16, 2023

I'm running into this as well.

For a workaround, I've started pinning to the following versions and molecule runs as expected

pip3 install ansible==8.2.0 ansible-core==2.15.2 ansible-compat==4.1.5 molecule==5.1.0 molecule-plugins[docker]==23.5.0 docker==6.1.3

jason-riddle added a commit to jason-riddle/ansible-role-tailscale that referenced this issue Aug 16, 2023
@geerlingguy
Copy link
Contributor

Also running into this issue starting a day or two ago.

jason-riddle added a commit to jason-riddle/ansible-role-tailscale that referenced this issue Aug 17, 2023
* Pin dependencies due to failing build.

See: ansible/molecule#4017

Workaround: ansible/molecule#4017 (comment)

* Add issue and workaround as comment
@wookietreiber
Copy link
Author

wookietreiber commented Aug 17, 2023

pip3 install ansible==8.2.0 ansible-core==2.15.2 ansible-compat==4.1.5 molecule==5.1.0 molecule-plugins[docker]==23.5.0 docker==6.1.3

tested a bit, miminal version of this is:

pip3 install ansible ansible-core==2.15.2 ansible-compat==4.1.5 'molecule<6' molecule-plugins[docker] docker

edit: confirmed that the min pin from @zerwes below works:

pip3 install ansible==8.2.0 ansible-core==2.15.2 ansible-compat molecule molecule-plugins[docker] docker

@giner
Copy link

giner commented Aug 17, 2023

Here is a minimal test case to reproduce this issue (while preparing this test case I've found out that the issue had been already submitted):

python3 -m venv /tmp/molecule
/tmp/molecule/bin/pip3 install ansible molecule molecule-docker

mkdir /tmp/role
cd /tmp/role

mkdir -p molecule/default
echo '{driver: {name: docker}, platforms: [{name: instance, image: 'ubuntu:22.04'}]}' > molecule/default/molecule.yml

/tmp/molecule/bin/molecule create

Possible temporary workaround:

export ANSIBLE_COLLECTIONS_PATH=/tmp/molecule/lib/python3.10/site-packages

/tmp/molecule/bin/molecule create

@cidrblock
Copy link
Contributor

cidrblock commented Aug 17, 2023

@giner Thanks for the minimal repro. There was a change in ansible-core 2.15.3 related to ansible-galaxy and it's default behaviour related to installing collections, I think this is related (ansible/ansible#81515)

If I switch to core 2.15.2, we can see the molecule has installed the collections in my ~/.ansible:

ansible-galaxy collection list                                

# /home/bthornto/.ansible/collections/ansible_collections
Collection                    Version
----------------------------- -------
ansible.posix                 1.5.4  
community.docker              3.4.8

Still digging...........

@cidrblock
Copy link
Contributor

Additional finding, if only ansible-core is installed and not ansible, the collections included in ansible are not installed into the venv site-packages directory and therefore not found by ansible-galaxy during the install. This results in the collections being installed in ~/.ansible and then found by molecule

@zhan9san
Copy link
Contributor

@cidrblock
Copy link
Contributor

@cidrblock

Do we need to add a force parameter in this function?

https://github.com/ansible/ansible-compat/blob/01f044866efab766e61d6a9b38d5bd510d479fe6/src/ansible_compat/runtime.py#L557-L565

Do you need to consider to bring isolated environment back?

Still looking..........getting to know the code. I'm trying to figure out why the ansible-galaxy install command finds the collections installed via ansible into the venv but the compat doesn't.

ansible_compat.errors.InvalidPrerequisiteError: Collection 'community.docker' not found in '['/home/bthornto/.ansible/collections', '/usr/share/ansible/collections']'

I suspect compat is not using the sys paths like core........

@cidrblock
Copy link
Contributor

cidrblock commented Aug 17, 2023

I think this is the logic compat needs to emulate: https://github.com/ansible/ansible/blob/9afaf2216b0bf38cfbd9a0c93409959f78d01820/lib/ansible/utils/collection_loader/_collection_finder.py#L254 Candidate collection paths include sys.paths unless configured otherwise. Looking deeper into compat to see where it should go

@zerwes
Copy link

zerwes commented Aug 21, 2023

pin ansible version ansible==8.2.0 ansible-core==2.15.2 is sufficient for me (so ist this really molecule related?)
see zerwes/git-remembrall#5

zerwes added a commit to Rosa-Luxemburgstiftung-Berlin/ansible-role-unbound that referenced this issue Aug 21, 2023
zerwes added a commit to Rosa-Luxemburgstiftung-Berlin/ansible-role-unbound that referenced this issue Aug 21, 2023
zerwes added a commit to zerwes/ansible-role-checkmkagent that referenced this issue Aug 21, 2023
zerwes added a commit to Rosa-Luxemburgstiftung-Berlin/ansible-role-unbound that referenced this issue Aug 21, 2023
* interim fix : pin latest ansible version due to ansible/molecule#4017

* interim fix : pin latest ansible version due to ansible/molecule#4017
zerwes added a commit to zerwes/ansible-role-checkmkagent that referenced this issue Aug 21, 2023
zerwes added a commit to Rosa-Luxemburgstiftung-Berlin/ansible-role-opera-dns-ui that referenced this issue Aug 21, 2023
zerwes added a commit to Rosa-Luxemburgstiftung-Berlin/ansible-role-gns3 that referenced this issue Aug 21, 2023
zerwes added a commit to Rosa-Luxemburgstiftung-Berlin/ansible-role-opera-dns-ui that referenced this issue Aug 21, 2023
zerwes added a commit to Rosa-Luxemburgstiftung-Berlin/ansible-role-gns3 that referenced this issue Aug 21, 2023
@milliams
Copy link

It seems that ansible-compat 4.1.7 was just released (https://github.com/ansible/ansible-compat/releases/tag/v4.1.7) which has fixed this issue for me.

@audgirka
Copy link
Contributor

@milliams thanks for confirming this :)

@cidrblock
Copy link
Contributor

@milliams TY

@ssbarnea
Copy link
Member

@ajinkyau Please add a change that requires the new version of compat for closing this bug in molecule.

zerwes added a commit to zerwes/ansible-role-checkmkagent that referenced this issue Aug 22, 2023
zerwes added a commit to Rosa-Luxemburgstiftung-Berlin/ansible-role-gns3 that referenced this issue Aug 22, 2023
zerwes added a commit to Rosa-Luxemburgstiftung-Berlin/ansible-role-opera-dns-ui that referenced this issue Aug 22, 2023
@giner
Copy link

giner commented Aug 23, 2023

Works for me as well. @wookietreiber I guess this issue can be closed?

@wookietreiber
Copy link
Author

Works for me as well. @wookietreiber I guess this issue can be closed?

Yeah, it works for me now as well. I'd close too but wanted to wait for @ssbarnea and @ajinkyau.

audgirka added a commit that referenced this issue Aug 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants