Testing Working Group Meeting Agenda 2

[gundalow]

The Testing Working Group will be the central point for all things related to testing Ansible, with some overlap with how to make it easier to develop and extend Ansible & modules

In particular this community have the following aims:

• Publicise the testing roadmap and correlate feedback from the community
• Fast track community PRs that are aligned with the Testing Strategy
• Fast track functional (new modules, bug fixes) PRs that have tests
• Make it easier to add/extend tests
• Be the centralised place for feedback for all things testing related

Please leave a comment regarding any agenda item you wish to discuss. If you don't show up for the meeting, your item will be skipped.

If your IRC nick is different from your Github username, leave that as well.

See https://github.com/ansible/community/blob/master/MEETINGS.md for the schedule

Community link https://github.com/ansible/community/tree/master/group-testing

[gundalow]

Mikedlr has been working on some Unit Test notes
Reviews and updates inline welcome https://public.etherpad-mozilla.org/p/ansible_module_unit_testing_guide

[gundalow]

from @willwagner602 ansible/ansible#30767 test_fortios_api needs requests_mock

How we we want to handle this?

[mikedlr]

I proposed adding hypothesis-python tests in the working group open floor section. These can cause some random failures so feeling was that they shouldn't go into normal CI but instead be part of the nightly run. There are a few of us interested in using it. I plan to set up a demo as part of the unit tests.

[mikedlr]

@gundalow please note that the unit test documentation mentioned above is now merged to ansible deve / 4.2.1 and should appear on the main documention site some time soon.

[gundalow]

No meeting on Thursday 2nd November.

[gundalow]

ansible/ansible#31456 Unit Tests @pilou-

[resmo]

I got in touch which 2 public cloud providers lately with proprietary API having or working on modules in Ansible: Vultr and Cloudscale. In both cases the question was, how would be the process to sponsor accounts (login credentials / API keys) the RedHat ( Ansible Project ) for having integration on PRs (via shippable).

[pilou-]

@resmo quoting mattclay:

if public cloud provider provided a service similar to ansible-core-ci for provisioning temporary credentials [1], that would work.

Otherwise, CI doesn't handle that :-( Same problem occurred with Packet cloud provider [2]. CI isn't able to store credentials.

There is something called Distributed CI [3] [4]. This project allows third parties to execute tests for their modules in their own testing environments. The tests are executed on scheduled time (for example twice a day) for the development branch, they don't prevent a wrong pull request to be merged.

As far as I know, this recent project is used by network vendors, there isn't any cloud vendors using it. Still, using DCI for cloud modules integration tests can be a possibility.

In the future (no date planned), Ansible CI might use Zuul [5], then third parties testing will be easier.

Even if not executed by Ansible CI, integration tests could be added in Ansible repository. These tests could be executed by related cloud provider.

In any case, unit tests could be added too (these would be executed by the Ansible CI).

[1] https://github.com/ansible/ansible/blob/devel/test/runner/lib/core_ci.py
[2] https://meetbot-raw.fedoraproject.org/ansible-meeting/2017-08-03/testing_working_group.2017-08-03-17.03.log.html#l-54
[3] https://github.com/ansible/dci-partner-ansible
[4] https://www.distributed-ci.io/
[5] https://docs.openstack.org/infra/zuul/

[gundalow]

No meeting today (Thursday 23rd November)

[gundalow]

Defending against missing unit tests dependencies ansible/ansible#33302

[gundalow]

Should test that we can't run, but will be invoked via DCI/Zuul be moved into the main Ansible Repo?
e.g. https://github.com/F5Networks/f5-ansible/tree/devel/test/integration/targets

[gundalow]

The 2.5 freeze dates (merge deadlines) have been pushed out by a week.

Please see updated dates
https://github.com/ansible/ansible/blob/devel/docs/docsite/rst/roadmap/ROADMAP_2_5.rst

To see whats change please see ansible/ansible@1826c27

[gundalow]

Validation of module's DOCUMENTATION (name & aliases) against argspec now live thanks to Sivel's ansible/ansible#34809

Examples of fixes:

• Module DOCUMENTATION should match argspec ansible#34895
• Remove deprecated params from module argspec nxos modules ansible#34911
• Add missing extends_documentation_fragment ansible#35013

[pilou-]

I would like to discuss (reviews required for both PR):

• integration tests for supervisorctl: supervisorctl: add integration tests ansible#36033
• last PR for broken import: webfaction modules: fix broken import ansible#35024

[gundalow]

Retrying package installs to work around external instabilities

We are often seeing "unstable" results due to external package repos being unavailable
evrardjp on IRC mentioned ansible/ansible-lint#324 could help us identify playbooks that need updating

- name: "Apt has retry test success"
  apt:
    pkg: foo
  register: apt_retry_workaround
  until: apt_retry_workaround|success
  retries: 5
  delay: 10

We have a number of different package modules, though not sure which are the worse offenders

http://docs.ansible.com/ansible/latest/playbooks_loops.html#do-until-loops

[gundalow]

ansible/ansible#18181 validate-modules: Shell out using run_command
ansible/ansible#26012

• Is this a sensible check
• If so, should it be in validate-modules or custom pylint rule?

[gundalow]

ansible/ansible#18183 validate-modules:
In stable-2.5 and later we ensure that names & aliases in argspec & DOCUMENTION match.

We are not checking:

• defaults
• choices
• type: bool #may not care about this

I'm sure the above would spot more issues. Though name & aliases catches documentation being missing which is the big win.

[gundalow]

https://github.com/ansible/ansible/pull/36468/files VCR Testing framework

[robinro]

ansible/ansible#35197 (reducing dependencies for tests, especially removing git from dockerfiles) broke the tests for the pip module. This wasn't detected in the PR or any other situation until the next PR for pip came along weeks later. For the future one should ensure that

• changes to the dockerfiles force a full CI run for all tests using them
• reguraly the full CI suite is run to find blind spots

[mattclay]

@robinro There were no Dockerfile changes in that PR, but rather to the setup script for VMs. The pip tests broke because they had a dependency on git, which was installed by another test. As long as the other test ran, the pip tests passed. If the pip tests were run in isolation they would fail.

Both of your suggestions are good. However, we're already doing both of those.

Unfortunately there isn't any practical way to detect this kind of issue, as it would require running every integration test in a separate container or VM.

[glitchcrab]

ansible/ansible#40071 Initial PR for integration with Memset's API. MERGED

IRC nick: bob_cheesey

[gundalow]

Notes from 2018-08-02

• sdoran is adding some tests for var_prompts, which needs passlib
• ACTION: sdoran 1 )So in your PR you should be able to update test/runner/setup/remote.sh to install passlib 2) and remove from runme.sh
• AGREED: that we should honor label:shipit for PRs that also touch test/integration/target/$modulename/ (+ units) allow automerge on module PRs with tests. ansibullbot#808

Minutes

• Minutes: https://meetbot.fedoraproject.org/ansible-meeting/2018-08-02/ansible_testing_working_group.2018-08-02-17.09.html
• Minutes (text): https://meetbot.fedoraproject.org/ansible-meeting/2018-08-02/ansible_testing_working_group.2018-08-02-17.09.txt
• Log: https://meetbot.fedoraproject.org/ansible-meeting/2018-08-02/ansible_testing_working_group.2018-08-02-17.09.log.html

[gundalow]

Ansible Contributor Summit 6 (Part of AnsibleFest 2018 Austin, Texas)

As part of http://www.ansible.com/ansiblefest

• AnsibleFest Austin will be on both Tuesday 2nd and Wednesday 3rd October 2018
• The Contributor Summit will also be over two days, on Monday 1st and Thursday 4th October, from 9:30am to 4pm (time subject to change).

This is your event

Ansible succeeds because of its community, and this contributor summit is for you. If you have any topics to discuss, large or small, please be sure to post those topics on the etherpad. For topics that other had added please vote by adding +1

https://etherpad.openstack.org/p/ansible-summit-october-2018

Please see https://groups.google.com/forum/#!topic/ansible-devel/ivrpVbLya8g for further info

Attending in person

We realize that not everybody can attend in person, which is why we’ll again be arranging a video conference and a meeting time on IRC in #ansible-meeting to encourage remote participation. This has worked very well for the past Contributor Summits. We’ll be sending out an email with the information for remote participation about a week before the Contributor Summit.

If you want to attend in-person please sign up at: https://www.eventbrite.com/e/ansible-contributor-conference-austin-2018-tickets-48557063484

After signing up you'll receive an email from Eventbrite with a discount code for AnsibleFest Austin.

Attending remotely

Please add your name at the bottom of this etherpad: https://etherpad.openstack.org/p/ansible-summit-october-2018

Call to action

1. Add ideas to https://etherpad.openstack.org/p/ansible-summit-october-2018
2. Attending in person: https://www.eventbrite.com/e/ansible-contributor-conference-austin-2018-tickets-48557063484 then use discount code to sign up to AnsibleFest
3. Attending in person OR virtually: Add your name to the end of https://etherpad.openstack.org/p/ansible-summit-october-2018
4. Happy automating!

Questions?

Thanks for all of your contributions to Ansible, and we hope to see you in Austin! And if you have any questions at all, please let us know.

Speak to gundalow on Freenode, or anyone else in #ansible-devel

[pilou-]

• hashi_vault lookup integration tests: hashi_vault lookup: add integration tests ansible#44814
• integration tests: should not ansible-playbook be called twice: a first playbook in order to setup dependencies, then a second playbook with the tests themselves (see and here) ?

[pilou-]

EOL distributions in the CI: why Fedora 24 and Fedora 25 are still used when Fedora 26 and Fedora 27 seem to work fine (shippable build of ansible/ansible#27795) ?

[mattclay]

Meetings cancelled for the holidays. The regular schedule will resume on January 10th.

[pabelanger]

I'd like to propose a slot in the weekly update for zuul related matters. I am also happy to give a little recap of the work with zuul control plane moving forward. Some of the highlights are:

• Create new Zuul control plane on public (openstack) cloud
• Zuul deployed using upstream playbooks / roles
• Migrate ansible-network tenant to new https://zuul.eng.ansible.com
• Migrate aws / molecule projects
• Work with ansible community and come up with plan to onboard community roles

[pilou-]

Meeting ended Thu May 23 17:49:40 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot .
Minutes: https://meetbot.fedoraproject.org/ansible-meeting/2019-05-23/ansible_testing_working_group.2019-05-23-17.14.html
Minutes (text): https://meetbot.fedoraproject.org/ansible-meeting/2019-05-23/ansible_testing_working_group.2019-05-23-17.14.txt
Log: https://meetbot.fedoraproject.org/ansible-meeting/2019-05-23/ansible_testing_working_group.2019-05-23-17.14.log.html

[pilou-]

ansible/ansible#56392: `consul_session` module: add integration tests

[pilou-]

ansible/ansible#58864: postgresql_user module: add integration tests (merged in the meantime)

[mattclay]

There will be no Ansible Testing Working Group meetings until January 9th.

[Andersson007]

Closed as no activity, thanks everyone!