Any way to have Molecule run all Ansible tests and report list of fail/pass like Testinfra?

[isuftin]

I am attempting to switch a playbook over from Testinfra to Ansible testing via asserts. With Testinfra, it's very convenient for Molecule to run through all the tests, whether they pass or fail, and collect the results and provide them at the end.

When using Ansible asserts via a verify playbook, I don't see a good clean way to replicate this behavior. The playbook converges for a long time and there are a lot of tests and I test multiple systems in parallel, so incrementally fixing tests one by one instead of knowing all the tests that will fail is a very painful workflow.

Does anyone have any thoughts?

[AbdouMechraoui]

Hi @isuftin,

did you manage to find any workarounds?
I am in the same position at the moment :/

[isuftin]

@AbdouMechraoui - Unfortunately, I haven't been able to and I don't think it's a problem specific to Molecule. This is more of an Ansible issue and Ansible failing fast on asserts :(

I've come up with a few hacky ways this could be done but haven't implemented them.

The best way I could think to do wrap each assert in a block with a rescue (https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_blocks.html#handling-errors-with-blocks) and within the rescue block, increment a fail counter via the set_fact or register module and also use the set_fact or register module to add to a list describing where things have failed. Then at the very end of all asserts, run a debug module with a failed_when parameter that fails if the increment variable is greater than 0 and spit out the list of failures.

Genius, I know. :D

But really, who wants to write their asserts in that way? Especially if you have a lot of tests.

There are probably other ways of doing it as well, but I'm guessing they're going to be equally as hacky.

[isuftin]

Note that I also asked it in the Ansible forums in May with no responses :(

https://forum.ansible.com/t/molecule-end-to-end-testing-via-ansible-verifier/5583