provisioner.options.extra-vars in molecule.yml no longer takes dict as input

[avictor0826]

Issue Type

• Bug report

Molecule and Ansible details

ansible 2.10.6
  config file = None
  configured module search path = ['/home/gitlab-runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.9.2 (default, Mar  3 2021, 09:23:36) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
molecule 3.2.3 using python 3.9
    ansible:2.10.6
    delegated:3.2.3 from molecule
    docker:0.3.3 from molecule_docker

Molecule installation method (one of):

• pip

Ansible installation method (one of):

• pip

Detail any linters or test runners used:
We use the yaml-lint and ansible-lint

lint: |
        yamllint . && ansible-lint .

Desired Behavior

The way we pass extra_vars is as given below in the molecule.yml

provisioner:
  name: ansible
  options:
    extra-vars:
      testing: 1234
  lint:
    name: ansible-lint
  env:
    ANSIBLE_ROLES_PATH: "${CI_PROJECT_DIR}/roles"

The expected behavior is to see the variable testing available in the converge step

This is breaking the converge step when the molecule > 3.2.0 tries to run the an

Actual Behaviour

The actual behaviour is an error from the shlex library

[root@04a7f4bae124 java]# molecule --debug converge
DEBUG    Validating schema /home/avictor/roles/java/molecule/default/molecule.yml.
INFO     default scenario test matrix: dependency, create, prepare, converge
INFO     Running default > dependency
WARNING  Skipping, dependency is disabled.
WARNING  Skipping, dependency is disabled.
INFO     Running default > create
WARNING  Skipping, instances already created.
INFO     Running default > prepare
WARNING  Skipping, prepare playbook not configured.
INFO     Running default > converge
INFO     Sanity checks: 'docker'
DEBUG: ANSIBLE ENVIRONMENT:
ANSIBLE_COLLECTIONS_PATH: /home/avictor/.cache/molecule/java/default/collections:/home/avictor/.ansible/collections:/usr/share/ansible/collections:/etc/ansible/collections
ANSIBLE_CONFIG: /home/avictor/.cache/molecule/java/default/ansible.cfg
ANSIBLE_FILTER_PLUGINS: /usr/local/lib/python3.9/site-packages/molecule/provisioner/ansible/plugins/filter:/home/avictor/.cache/molecule/java/default/plugins/filter:/home/avictor/roles/java/plugins/filter:/home/avictor/.ansible/plugins/filter:/usr/share/ansible/plugins/filter
ANSIBLE_FORCE_COLOR: '1'
ANSIBLE_LIBRARY: /usr/local/lib/python3.9/site-packages/molecule/provisioner/ansible/plugins/modules:/home/avictor/.cache/molecule/java/default/library:/home/avictor/roles/java/library:/home/avictor/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
ANSIBLE_ROLES_PATH: /home/avictor/.cache/molecule/java/default/roles:/home/avictor/roles:/home/avictor/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/roles

DEBUG: MOLECULE ENVIRONMENT:
MOLECULE_DEBUG: 'True'
MOLECULE_DEPENDENCY_NAME: galaxy
MOLECULE_DRIVER_NAME: docker
MOLECULE_ENV_FILE: /home/avictor/roles/java/.env.yml
MOLECULE_EPHEMERAL_DIRECTORY: /home/avictor/.cache/molecule/java/default
MOLECULE_FILE: /home/avictor/.cache/molecule/java/default/molecule.yml
MOLECULE_INSTANCE_CONFIG: /home/avictor/.cache/molecule/java/default/instance_config.yml
MOLECULE_INVENTORY_FILE: /home/avictor/.cache/molecule/java/default/inventory/ansible_inventory.yml
MOLECULE_PROJECT_DIRECTORY: /home/avictor/roles/java
MOLECULE_PROVISIONER_NAME: ansible
MOLECULE_SCENARIO_DIRECTORY: /home/avictor/roles/java/molecule/default
MOLECULE_SCENARIO_NAME: default
MOLECULE_STATE_FILE: /home/avictor/.cache/molecule/java/default/state.yml
MOLECULE_VERIFIER_NAME: testinfra
MOLECULE_VERIFIER_TEST_DIRECTORY: /home/avictor/roles/java/molecule/default/tests

DEBUG: SHELL REPLAY:
ANSIBLE_COLLECTIONS_PATH=/home/avictor/.cache/molecule/java/default/collections:/home/avictor/.ansible/collections:/usr/share/ansible/collections:/etc/ansible/collections ANSIBLE_CONFIG=/home/avictor/.cache/molecule/java/default/ansible.cfg ANSIBLE_FILTER_PLUGINS=/usr/local/lib/python3.9/site-packages/molecule/provisioner/ansible/plugins/filter:/home/avictor/.cache/molecule/java/default/plugins/filter:/home/avictor/roles/java/plugins/filter:/home/avictor/.ansible/plugins/filter:/usr/share/ansible/plugins/filter ANSIBLE_FORCE_COLOR=1 ANSIBLE_LIBRARY=/usr/local/lib/python3.9/site-packages/molecule/provisioner/ansible/plugins/modules:/home/avictor/.cache/molecule/java/default/library:/home/avictor/roles/java/library:/home/avictor/.ansible/plugins/modules:/usr/share/ansible/plugins/modules ANSIBLE_ROLES_PATH=/home/avictor/.cache/molecule/java/default/roles:/home/avictor/roles:/home/avictor/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/roles MOLECULE_DEBUG=True MOLECULE_DEPENDENCY_NAME=galaxy MOLECULE_DRIVER_NAME=docker MOLECULE_ENV_FILE=/home/avictor/roles/java/.env.yml MOLECULE_EPHEMERAL_DIRECTORY=/home/avictor/.cache/molecule/java/default MOLECULE_FILE=/home/avictor/.cache/molecule/java/default/molecule.yml MOLECULE_INSTANCE_CONFIG=/home/avictor/.cache/molecule/java/default/instance_config.yml MOLECULE_INVENTORY_FILE=/home/avictor/.cache/molecule/java/default/inventory/ansible_inventory.yml MOLECULE_PROJECT_DIRECTORY=/home/avictor/roles/java MOLECULE_PROVISIONER_NAME=ansible MOLECULE_SCENARIO_DIRECTORY=/home/avictor/roles/java/molecule/default MOLECULE_SCENARIO_NAME=default MOLECULE_STATE_FILE=/home/avictor/.cache/molecule/java/default/state.yml MOLECULE_VERIFIER_NAME=testinfra MOLECULE_VERIFIER_TEST_DIRECTORY=/home/avictor/roles/java/molecule/default/tests

Traceback (most recent call last):
  File "/usr/local/bin/molecule", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/molecule/command/converge.py", line 104, in converge
    base.execute_cmdline_scenarios(scenario_name, args, command_args, ansible_args)
  File "/usr/local/lib/python3.9/site-packages/molecule/command/base.py", line 113, in execute_cmdline_scenarios
    execute_scenario(scenario)
  File "/usr/local/lib/python3.9/site-packages/molecule/command/base.py", line 155, in execute_scenario
    execute_subcommand(scenario.config, action)
  File "/usr/local/lib/python3.9/site-packages/molecule/command/base.py", line 144, in execute_subcommand
    return command(config).execute()
  File "/usr/local/lib/python3.9/site-packages/molecule/logger.py", line 185, in wrapper
    rt = func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/molecule/command/converge.py", line 83, in execute
    self._config.provisioner.converge()
  File "/usr/local/lib/python3.9/site-packages/molecule/provisioner/ansible.py", line 693, in converge
    return pb.execute()
  File "/usr/local/lib/python3.9/site-packages/molecule/provisioner/ansible_playbook.py", line 107, in execute
    result = util.run_command(self._ansible_command, debug=self._config.debug)
  File "/usr/local/lib/python3.9/site-packages/molecule/util.py", line 144, in run_command
    result = run(
  File "/usr/local/lib/python3.9/site-packages/subprocess_tee/__init__.py", line 103, in run
    cmd = join(args)
  File "/usr/local/lib/python3.9/shlex.py", line 320, in join
    return ' '.join(quote(arg) for arg in split_command)
  File "/usr/local/lib/python3.9/shlex.py", line 320, in <genexpr>
    return ' '.join(quote(arg) for arg in split_command)
  File "/usr/local/lib/python3.9/shlex.py", line 329, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like **object**

The error is from the shlex library (used by subprocess_tee library) trying to escape the command strings -- pull #2917 and this file

Reproducing

This should be self explanatory

[root@04a7f4bae124 java]# python3
Python 3.9.2 (default, Mar  3 2021, 09:23:36)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess_tee import run
>>> args=['ansible-playbook', '--diff', '--extra-vars', {'testing': 1234}, '--inventory', '/home/gitlab-runner/.cache/molecule/java/default/inventory', '--skip-tags', 'molecule-notest,notest', '/home/gitlab-runner/roles/java/molecule/default/converge.yml']
>>> run(args)
run string=['ansible-playbook', '--diff', '--extra-vars', **{'testing': 1234}**, '--inventory', '/home/gitlab-runner/.cache/molecule/java/default/inventory', '--skip-tags', 'molecule-notest,notest', '/home/gitlab-runner/roles/java/molecule/default/converge.yml']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/subprocess_tee/__init__.py", line 104, in run
    cmd = join(args)
  File "/usr/local/lib/python3.9/shlex.py", line 320, in join
    return ' '.join(quote(arg) for arg in split_command)
  File "/usr/local/lib/python3.9/shlex.py", line 320, in <genexpr>
    return ' '.join(quote(arg) for arg in split_command)
  File "/usr/local/lib/python3.9/shlex.py", line 329, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object

[avictor0826]

Please note that extra-vars as a dict were working until molecule=3.1.x.

[hluaces]

In my case this error happens when I specify a forks option with a numeric value like this:

provisioner:
  name: ansible
  log: true
  options:
    vv: true
    forks: 20

If I change it to a string molecule works fine:

provisioner:
  name: ansible
  log: true
  options:
    vv: true
    forks: '20'

I see that your testing extra_option has a numeric value, could that be a problem as well?

[avictor0826]

@hluaces yup.. i have seen that molecule > 3.2 doesnt treat the integers in molecule.yml kindly. They have to be quoted to be passed on as args in the commandline.

However the issue is that extra-vars is not working with a dict as its value.

• extra-vars as a string -- 👍
• extra-vars as a int -- 👍 when quoted; 👎 when not quoted
• extra-vars as a dict -- 👎;but 👍 when passed as string; something like this

provisioner:
  name: ansible
  options:
    extra-vars: "{'testing': 1234}"
  lint:

and this works..

Commandline is

COMMAND: ansible-playbook --diff --extra-vars '{'"'"'testing'"'"': 1234}' --inventory /home/avictor/.cache/molecule/java/default/inventory --skip-tags molecule-notest,notest /home/avictor/roles/java/molecule/default/converge.yml

But this way, we loose the flexibility of dicts and ymls.

Thoughts?

[tadeboro]

Well, Ansible's -e switch expects key=value pairs or JSON, so technically, passing a dictionary as extra-vars is invalid. Molecule never did any special preprocessing of the parameters here, so my guess would be that in the past, things just happened to work in some cases. For example, this is what molecule did before we dropped the sh library for running commands:

>>> str(sh.ansible_playbook.bake({"extra-vars": dict(testing=1234)}))
"/usr/bin/ansible-playbook --extra-vars={'testing': 1234}"

In the new world order, we use the built-in subprocess python module, and that thing chokes on the dictionary.

All that being said, I do feel that treating --extra-vars specially and serializing it to JSON before passing it down to Ansible CLI (or even sticking it into a file and reference it using --extra-vars @tmp.json) would be beneficial. Retaining backward compatibility might be a bit painful here, but it should be doable.

Does special-casing the extra-vars option make sense to you all?

[zhan9san]

is this feature still required?
If yes, I'll try to fix it