Testing Working Group Meeting Agenda

[gundalow]

Moved to new agenda, please see #248

.

[alikins]

ansible/ansible#16864 - Add test apt,ppa,yum repos to httptester images for integration tests

[gundalow]

Hi, I'd like to propose https://duckduckgo.com/?q=1800+UTC&ia=answer Starting Thursday 18th August for the weekly meeting. We need to cater for East Coast & Europe, I hope that works for most people.

~~+1/-1 welcome :)~

[srgvg]

👍 svg on irc

[rcarrillocruz]

works great for me.

[pabelanger]

+1

[supertom]

+1

[gundalow]

Hi,

Due to other conflicts I need to move this one hour earlier so it will now be

https://duckduckgo.com/?q=1700+UTC
Hope most can still make that

Starting Thursday 18th August for the weekly meeting

[gundalow]

Agenda for meeting 1

• Welcome!
• Introductions from Ansible Core Team and Contributors
• Review of purpose
• What we already have planned see Infrastructure Buildout and Changes (Matt Clay) on ROADMAP_2_2.rst
• Poll on what people are interested in - Need more items https://public.etherpad-mozilla.org/p/ansible-testing-working-group
• What the community is interested in helping with
• Open Floor (& General Questions)

As with an of the Community Working Group meetings, please feel free to add your own thoughts and ideas to this ticket so I can add it to the agenda.

[dagwieers]

+1

[gundalow]

@linuxdynasty has put forward this module to be used as an example of a good module https://github.com/ansible/ansible-modules-extras/blob/devel/cloud/amazon/kinesis_stream.py

[caphrim007]

• Is there a way to test modules that use map() and expect a list to be returned? Python 3 changes map() to be an iterator
• ask @caphrim007 to show the functional test model he uses for F5 modules to stir up some testing ideas
  • caphrim007 provided a detail walk through of https://github.com/F5Networks/f5-ansible/tree/master/tests

[robinro]

PR by @cinerama70 to review/merge. Test for ansible_os_family and a related bugfix:
ansible/ansible#17620 and ansible/ansible#17575

In general the test coverage of basic facts could use some more attention.

PRs merged.

[mscherer]

Sorry I merged ansible/ansible#17620 before realizing you wanted to have it discussed in the meeting. In exchange of me removing a easy discussion during the meeting, I offer ansible/ansible#17628

[robinro]

Thanks @mscherer for the merge. I didn't really want to discuss it but only draw some attention to get it merged and maybe get more tests for the facts code.

[relaxdiego]

I'd like to submit my article on Writing Ansible Modules Complete With Tests for feedback by this group. I'll be speaking in AnsibleFest Brooklyn using this content and would like to make sure it aligns with community practices. Feedback will be collated in the next meeting.

The article is at https://www.relaxdiego.com/2016/06/writing-ansible-modules-with-tests.html

IRC nick: mmaglana

[caphrim007]

- I think @gundalow asked how I do coverage testing of the f5-ansible modules. I do it using playbooks. I can talk more about it at the next meeting

This is how I do it
F5Networks/f5-ansible@88fb0a4

[relaxdiego]

Submitting version 2.0.0 of the article for review in the next meeting. Changes are:

• Simplify main() and move business logic to separate function
• Take advantage of lessons from Boundaries talk by Gary Bernhardt
• Reference module checklist
• Give credit where credit is due :-)
• Other minor fixes

URL is still https://www.relaxdiego.com/2016/06/writing-ansible-modules-with-tests.html

[gundalow]

Do we want a test_pull_requests label so we can identify (and therefore fast track) PRs that improve testing? Would people find this useful and help review changes?

Agreed: 4 people said they would find it useful having a test_pull_requests label.

[alikins]

Wouldn't mind discussing #114 (comment) (adding httptester hosted apt/yum/etc repos to reduce test failures from repo connectivity issues)

We discussed in this Testing Working Group on 2016-09-29 - This is something that we've thought about before and believe would be a good thing - We do see tests failing from tom - There were concerns that we are making httptester a dependency of tests. Tests need to see if httptester is available before deciding to use that as a repo source.

Moving forward please pick just one package manager to port over to httptester then we can discuss and review.

It should be noted that not all package managers user http(s), so we may need to add support in for rsync, etc.

Update: Further discussions on this included the idea providing repos without involving httptester, so that the tests can take advantage of them without depending on docker. This is important for those running the tests without docker, as well as for platforms like FreeBSD and OS X.

[gundalow]

For discussion on Thursday 20th October

Prep for repo merge

• Integration tests moving -
  • https://github.com/ansible/ansible/tree/devel/test/integration/targets
  • Dropping of test_ prefix
• ansible-validate-modules is now https://github.com/ansible/ansible/tree/devel/test/sanity/validate-modules

httptester
After seeing the discussion during Contributor Summit regarding making httptester available in non-docker forms Sivel has raised:
ansible/ansible#18107

[gundalow]

ACTION: gundalow to find all old references to ansible-validate-modules in google and get them updates

[gundalow]

No Testing Working Group meeting today (27th October).

[gundalow]

No Testing Working Group meeting today (3rd November), instead there is a meeting about the repo merge, all welcome.

[frenzymadness]

Hi.

I really want to help with Python 3 compatibility for Ansible. I think that good point to start and learn more about Ansible is helping with tests coverage.

I want to ask (here or on the meeting) if there is any roadmap for improving tests coverage of Ansible.

[gundalow]

@frenzymadness Thanks for the offer of help, please join us in today's meeting

[gundalow]

Cleaning up test/

• test_win_group*.yml Do we still need these files, if not can we delete them?
• test/integration/Makefile Should this be deleted?

The legacy files have been deleted.

[gundalow]

Document codee Coverage https://codecov.io/gh/ansible/ansible/

[pilou-]

About

test_win_group*.yml Do we still need these files, if not can we delete them?

mattclay wrote:

The Makefile is the legacy method of invoking tests that predates ansible-test. Instead of updating the Windows related targets in the Makefile (and renaming their associated playbooks), they should be removed.

and:

The test_win_group*.yml files should be removed as well, as they're only used by the windows targets in the Makefile.

The legacy files have been deleted.

[dagwieers]

For testing the win_uri module (and eventually the uri module as well) we would like to look at using httptester. This was deemed the preferred way during some discussion (don't have a reference at hand).

For Windows that would mean that the VMs need to be able to connect the the httptester docker network. As a result, we will leave the idea to set up our own http listener process (#153 (comment)).

[dagwieers]

For the new xml module, we need lxml provisioned. For the docker images, this is already the case, but for the VMs (FreeBSD, MacOSX) lxml is still missing.

ansible/ansible#25323 (comment)

[gundalow]

https://meetbot.fedoraproject.org/ansible-meeting/2017-06-29/testing_working_group.2017-06-29-17.00.html
https://meetbot.fedoraproject.org/ansible-meeting/2017-06-29/testing_working_group.2017-06-29-17.00.log.html

[gundalow]

test_pull_requests label.
If we want to use this label to find PRs with tests should we stop test/sanity/pep8/*.txt from getting that label?

Action Item: If test/sanity/pep8/legacy-files.txt is the only test/ file modified by a PR, the bot should not apply the test_pull_requests label. This will avoid adding the test_pull_requests label when a PR only cleans up legacy PEP 8 issues without making any other test related changes.

See: ansible/ansibullbot#629

[willthames]

@gundalow maybe just exclude test/sanity completely if you want to find PRs that are testing themselves rather than testing everything, if that makes sense.

[gundalow]

Do we want to start testing fedora26?

[mattclay]

I'm updating our Fedora CI testing. My current plan is to:

• Remove testing of Fedora 24 with python 2.
• Remove testing of Fedora 25 with python 2.
• Add testing of Fedora 25 with python 3.
• Add testing of Fedora 26 with python 3.

Fedora 24+ default installations are python 3.5+ only. These changes update CI to match.

[pilou-]

Failure of `copy`module integration tests: [#25672](https://github.com/ansible/ansible/pull/25672).

PR needs to be rebased. Nothing to discuss today.

[pdellaert]

Can we add a short discussion on running ansible-test inside an environment that requires a proxy?

Specifically when running ansible-test ... --docker, the docker images are unable to run commands that require external access (for instance pip).

Suggestion: expose *_proxy environment variables to all containers.

Not sure if tox supports this, btw, i've seen some issues there (could be related to something else)

[pilou-]

@pdellaert, during Testing Working Group mattclay wrote:

I can understand why someone might need it to run tests. However, getting all the tests to work with a proxy might be difficult.

[pdellaert]

@pilou-, There's two aspects :

• Test infrastructure aspect: containers being able to install stuff through pip or apt/yum, or tox being able to use proxy
• Tests themselves: Tests needing to connect to the outside world through a proxy, for instance to connect to external APIs (mainly integration testing might be impacted here)

I'd suggest support a proxy within the test infrastructure.

The tests themselves is a different story, these are indeed a lot more complicated and will depend significantly on a test per test basis (some might require a proxy, while others might not because connections are local)

A rough approach could be (if enough support/interest, of course):

• Stage 1: Provide test infrastructure support for proxy, through a new flag in ansible-test (--proxy), this flag sets the http_proxy, https_proxy and the apt/yum proxy values within the docker containers (-e..., as far as i know both Ubuntu and CentOS base containers support an -e setting to configure apt or yum proxy). The flag also configures tox to use the proxy if tests are run through tox.

• Stage 2: For integration testing itself: Expand the test system to allow specifying a boolean in the test cases 'requires_proxy', which would push the proxy setting from Stage 1 into the facts used by the integration playbooks.

[pilou-]

To sum up the meeting:

• copy integration tests fails (PR) when managed and controller aren't the same (CI use one host for managed and controller).
  • that could be solved using different filename prefixes ("local" for local file/"remote" for local file)
  • Pilou will update the pull requests.
• test infrastructure support for proxy:
  • pdellaert will provide environment variables, then cc mattclay
  • mattclay will create a PR with implementation, then pdellaert will be able to test it
• Pylint tests:
  • abadger1999 has been doing quite a bit of recent code cleanup around Pylint failures: undefined-variable test is enabled
  • abadger1999 is working on cleaning up wildcard imports.
  • Volunteers for cleaning up existing pylint issues can reach out to abadger1999 anytime :-)

[mattclay]

@pilou- Thank you for the excellent summary.

[pilou-]

I would like to discuss cloud/packet integration tests (ansible/ansible#23127) at the next meeting.

Pilou to get more details on the Packet sandbox API for possible use with integration tests.

[pdellaert]

On my action point:
ansible/ansible#27697

mattclay to add experimental proxy support to ansible-test.
pdellaert has volunteered to test once a PR is ready.

[Spredzy]

I would like to discuss the addition of the latest macOS 10.12 image in the CI pipeline (current 10.11)

Spredzy wasn't around for the meeting, so we can discuss more next time.

A brief update from the meeting:

With our current CI resources, we can only test one version of macOS in CI.
So we'll either need to switch from 10.11 to 10.12 or allocate more resources for macOS testing.

[gundalow]

Ansible Contributor Summit 5 (Part of AnsibleFest 2017 San Francisco)

When: 2017-09-06:
Notes: https://public.etherpad-mozilla.org/p/ansible-summit-september-2017
Event will be in person + online (Video + IRC)

Agenda is open for topics for the next Contributors Summit

Please add (and vote +1) for any topics.

[Akasurde]

PR needs review:

• Add testcases for github_issue ansible#26309 - Add testcases for github_issue MERGED
• Refactor code and testcase for find_host_by_cluster_datacenter ansible#26859 - Refactor code and testcase for find_host_by_cluster_datacenter MERGED

[gundalow]

NOTE: meetings are canceled for the week Aug 21st to 25th due to 'all hands' meeting for the core team. We will resume normal schedule after that.

[mattclay]

What should qualify as a destructive test? Existing documentation covers this, but we may want to consider changing the criteria.

The documentation for destructive tests will be clarified. Destructive tests are those which make changes outside of files or directories belonging only to tests. This would include installing/upgrading/uninstalling packages, creating/deleting users, etc.

Additionally, tests which are destructive, but not yet marked as such, will be updated.

It was also requested that by default tests should be assumed destructive and must be marked non_destructive if they are not. This will be done separately, as it is a more invasive change.

[gundalow]

NOTE: meetings are canceled for the week September 4th through 8th due to AnsibleFest San Francisco. We will resume normal schedule after that.

[gundalow]

Now that 2.4.0 has been released, it seems like a sensible point to move to a fresh agenda
#248

NO MORE COMMENTS HERE PLEASE