Integrations tests : Use containers for more control and verify that versions match expectation #490

laurent-indermuehle · 2023-01-23T21:24:42Z

SUMMARY

Improve tests reliability:

Test more versions of MariaDB
Carefully verify every component of the tests in the new target 'setup_controller' to ensure expected versions are correct : Python, Ansible, connector and MySQL/MariaDB.
Add tools to test locally the same as on GHA by using same containers and virtualenv. Custom test containers are published in ghcr.io by this repo's workflows. MySQL/MariaDB are official Docker Hub images.
New name for many tasks to makes it easier to find failing tests: Rename duplicates. Add name for tasks which doesn't had one, refactor some tests files to better group tests by subject, ...

ISSUE TYPE

Integration tests

COMPONENT NAME

CI / Integration tests

ADDITIONAL INFORMATION

The first goal was to add all latest major versions of MariaDB. Mainly the Long Term Support 10.6 which I thought wasn't working (spoiler, it was because I installed PyMySQL 1.0.2). While I was at it, I added 10.5 too. I also inserted 10.4 but we will remove it in late 2023 when EoL.

Secondly, I sometimes run into situations were the tests title lies about versions tested. I recall one time using tmate to SSH into a GHA Ubuntu VM to find out a MySQL 8 instead of the MariaDB 10.3 in the title of the test.
I also lost so much time when working on this PR because I thought I was testing pymysql but was in fact running mysqlclient. So I say: Never again! and the file verify.yml in the target 'setup_controller' checks every component announced by the workflow. If the test say "I will test python 3.10", we go into the target container and do a "python3 -V" and check if the expected version is returned. In addition to Python, it also verify the version of Ansible, the connector and the database.

Finally, by using custom containers from ansible-test we have more control on what we install. I still use Ubuntu as base image and change the release to get the right mysqlclient and Python. For instance Ubuntu 18.04 provides mysql-client-5.7,mariadb-client-10.3 and Ubuntu 20.04 and 22.04 provides mysql-client-8,mariadb-client-10.6. Also availables Python vary between those versions. So this image include 3 things : 1) Python, 2) the Python connector and its dependencies and 3) mysql-client which provides the commands mysql and mysqldump. These custom containers have each a workflow to build and publish its images in ghcr.io. Those images can be found in the GitHub <> Code page of this repo in the "Packages" section.

~~To test properly the version of the connector I had to enhance mysql_info. It now always returns the version used for both PyMySQL and mysqlclient.~~ This is done in #497

In place of dbdeployer, I use official Docker Hub images for MySQL and MariaDB. The added value of this is that it is now easy to specify the version we want to test thanks to the docker images tags.

To communicate between containers, instead of "localhost" I lookup for the IP address of the podman/docker network and use that IP to connect with MySQL/MariaDB primary or replicas. Thus, many accounts have been renamed from "test_user@localhost" to "test_user@'%'" to allow remote connection.

Because we don't install MySQL/MariaDB anymore, 'setup_mysql' is renamed 'setup_controller'. Its purpose is to prepare the fake root folder (I'm still not sure why we do that), set variables for the tests and verify versions. Now I'm wondering if "setup_tests" wouldn't be even better. What do you think?

I tried to add quality of life when running tests locally. I'm not entirely happy with my Makefile because I'm not fluent with Make and think it's hard to modify. Also run_all_test.py is just a loop but I would prefer a menu telling me available tests and letting me choose which ones to fire. It's a start and we can improve later. I wrote a small documentation in the TESTING.md file. Quick note: I only tested Podman for local tests. If someone is willing to convert the Makefile and tests to work with Docker, I'll appreciate.

I also tried very hard to use the same code for GHA and local tests. Both now write the sames files, downloads the same containers and use the same venv to start ansible-test. If it pass on your machine, GHA will pass too! Thought, GHA runs into a VM and local tests in a Venv, so it's still not 1:1, but way closer than before.

By adding pymysql 1.0.2 I thought it was time we test this version. And boy I was right. I encountered many places were pymysql 1.0.2 behave differently from 0.9.3. I had to disable it until we fix the plugin. For instance with the query "CREATE TABLE IF NOT EXISTS existing_table" it reports a change instead of green.

Some of you may have noticed that I don't speak about speed gain. It's because, as of today, I can't see any difference between this repo and my fork. Maybe GHA run better lately?

This prevent the first test to fail because the db isn't ready yet.

Previously, everything was on localhost. Now ansible-test is in a venv and the db is in a container. The db see the IP address from the podman host (10.88.0.2)

It seams that that ansible-test-gh-action doesn't handle this option: https://github.com/ansible-community/ansible-test-gh-action/blob/main/action.yml#L483-L497

We are not logged in as 127.0.0.1 anymore, but 10.88... as I couldn't test this easily, I decided to simplify the test.

/root doesn't exist with --venv

https://docs.github.com/en/actions/using-containerized-services/about-service-containers I re-revert the workflow to use the new custom action. But I'm not sure it will works because I don't know how the container for ansible-test is started and if it will have access to the services containers.

I want to first prove that this setup is possible before adding safety

We use the Podman/Docker network gateway address to communicate between container. I haven't tested Docker. I would have preferred to use a pod but only Podman support it and ansible-test only support the --docker-network option.

This way we can split db_engine and db_version and simplify tests. Also this is mandatory to use the matrix.db_engine_version as the image name for our services containers.

Without this, the containers are not rebuilt when you modify the file built-docker-image.yml.

This time I think I understood. We publish in the github.repository_owner's namespace. In my case it's laurent-indermuehle and in case of upstream it's ansible-collection. A proof of that: https://github.com/orgs/ansible-collections/packages <- here there is one attempt I did in february to push my branch to the upstream. So, our tests containers will be visible to the whole community, not just community.mysql.

laurent-indermuehle · 2023-03-07T17:11:58Z

Oh dear... I discovered that we publish packages from GHA into the github.repository_owner's namespace, not in the repository project.
In my case it's laurent-indermuehle and here, it's ansible-collection.

A proof of that:
https://github.com/orgs/ansible-collections/packages
Here there is one attempt I did in february to push my branch to the upstream:

test-container-mariadb103-py38-mysqlclient201
Published on Feb 1 by Ansible Collections in ansible-collections/community.mysql

So, our tests containers will be visible to the whole community, not just community.mysql.
Is that ok? Or should I move this elsewhere (I don't want to use quay.io, it's so much pain from GHA)?

Andersson007 · 2023-03-08T13:50:48Z

@laurent-indermuehle

So, our tests containers will be visible to the whole community, not just community.mysql.
Is that ok?

I don't think there's any issue with visibility, should be fine

Or should I move this elsewhere (I don't want to use quay.io, it's so much pain from GHA)?

I don't know, feel free to ask in devel channel. I'm OK with any reliable option.

Also the PR is approved. I'll be on PTO next week, let's wait until next week and then feel free to merge the PR, thanks!

laurent-indermuehle · 2023-03-08T14:48:07Z

My... PR ... is approved 🥹!?? How do I missed that?
Ok, see you next week, have a nice PTO.
I hope the containers will build. Anyway I'll be here to fix any issue ;)

Andersson007

pardon, it had one approval, not it has two

Jorge-Rodriguez · 2023-03-10T19:46:51Z

We should add documentation on how to run the tests locally to the wiki, the README or both.

laurent-indermuehle · 2023-03-15T19:47:31Z

@Jorge-Rodriguez you're right! I didn't see there is a Wiki, sorry!

In the wiki, the setup guide is still relevant for unit and sanity tests. But won't work for integrations tests.

Should I keep the file TESTING.md and link to it from the Wiki or should I import it into the wiki?

Andersson007 · 2023-03-21T07:16:02Z

@laurent-indermuehle @Jorge-Rodriguez to me TESTING.md + links to it in README and Wiki sounds ok.
I'll merge the PR now. If any objections/blockers, please raise them ASAP after merge so that we can revert it.
The PR has been open for a long time, a lot of work has been done, and now it's time to merge it.

@laurent-indermuehle thanks for the great work! @Jorge-Rodriguez @betanummeric @rsicart thanks for reviewing!

…. (#514) * Fix explanation about containers images * Add definitive URI to the containers images * Document that new images must be set as public * Add makefile options possible values * Document that any mysql and mariadb tag can be use * Add computation of docker_image path * Refactor pre-command to separate commands for cleaner GHA output * Refactor to use GHA test matrix * Cut docker_image from documentation since it's now automatic * Document how to use run_all_test.py to display the test matrix * Temp: Add path to images in my fork to validate integrations tests

patchback · 2023-03-24T10:27:10Z

Backport to stable-2: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 6970aef on top of patchback/backports/stable-2/6970aef8f61373e9f85cb6f251b3b44decb7c496/pr-490

Backporting merged PR #490 into main

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/ansible-collections/community.mysql.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/stable-2/6970aef8f61373e9f85cb6f251b3b44decb7c496/pr-490 upstream/stable-2

Now, cherry-pick PR Integrations tests : Use containers for more control and verify that versions match expectation #490 contents into that branch:
```
$ git cherry-pick -x 6970aef8f61373e9f85cb6f251b3b44decb7c496
```
If it'll yell at you with something like fatal: Commit 6970aef8f61373e9f85cb6f251b3b44decb7c496 is a merge but no -m option was given., add -m 1 as follows instead:
```
$ git cherry-pick -m1 -x 6970aef8f61373e9f85cb6f251b3b44decb7c496
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Integrations tests : Use containers for more control and verify that versions match expectation #490 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/stable-2/6970aef8f61373e9f85cb6f251b3b44decb7c496/pr-490

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback · 2023-03-24T10:27:10Z

Backport to stable-1: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 6970aef on top of patchback/backports/stable-1/6970aef8f61373e9f85cb6f251b3b44decb7c496/pr-490

Backporting merged PR #490 into main

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/ansible-collections/community.mysql.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/stable-1/6970aef8f61373e9f85cb6f251b3b44decb7c496/pr-490 upstream/stable-1

Now, cherry-pick PR Integrations tests : Use containers for more control and verify that versions match expectation #490 contents into that branch:
```
$ git cherry-pick -x 6970aef8f61373e9f85cb6f251b3b44decb7c496
```
If it'll yell at you with something like fatal: Commit 6970aef8f61373e9f85cb6f251b3b44decb7c496 is a merge but no -m option was given., add -m 1 as follows instead:
```
$ git cherry-pick -m1 -x 6970aef8f61373e9f85cb6f251b3b44decb7c496
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Integrations tests : Use containers for more control and verify that versions match expectation #490 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/stable-1/6970aef8f61373e9f85cb6f251b3b44decb7c496/pr-490

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

…versions match expectation (ansible-collections#490) * Draft: Add a mariadb container * Add playbook to test connection to the server * Add healthcheck to MariaDB before starting the tests This prevent the first test to fail because the db isn't ready yet. * Add default file for root necessary since using venv instead of docker * Add % instead of the default 'localhost' since we use remote connection Previously, everything was on localhost. Now ansible-test is in a venv and the db is in a container. The db see the IP address from the podman host (10.88.0.2) * Add ansible-test integration inventory to .gitignore * Revert to old workflow to use ansible-test --venv It seams that that ansible-test-gh-action doesn't handle this option: https://github.com/ansible-community/ansible-test-gh-action/blob/main/action.yml#L483-L497 * Cut target filtering * Fix comparison We are not logged in as 127.0.0.1 anymore, but 10.88... as I couldn't test this easily, I decided to simplify the test. * Add path to default-file /root doesn't exist with --venv * Fix workflow unknown option container_name * Attempt GHA communication between container using "docker host network" https://docs.github.com/en/actions/using-containerized-services/about-service-containers I re-revert the workflow to use the new custom action. But I'm not sure it will works because I don't know how the container for ansible-test is started and if it will have access to the services containers. * Cut anchors currently unsupported by GHA * Disable healthcheck I want to first prove that this setup is possible before adding safety * Disable sanity, units and matrix to speed up tests in GHA * Further disable tests to speed up * Add mysql_client to the controller * Install mysql_client the correct way * Fix package name and missing apt cache * Prepare controller with Podman/Docker Network We use the Podman/Docker network gateway address to communicate between container. I haven't tested Docker. I would have preferred to use a pod but only Podman support it and ansible-test only support the --docker-network option. * Swap MariaDB with MySQL * De-duplicate the mysql_command alias * Generalize mysql and mariadb version based on container name This way we can split db_engine and db_version and simplify tests. Also this is mandatory to use the matrix.db_engine_version as the image name for our services containers. * Cut docker healthcheck unsupported by GHA * Fix replication server_id already in use * Add static test with replication containers * Fix database not selected * Fix replication due to usage of gateway_addr instead of localhost * Simplify version computation * Linting * Refactor setup_mysql into setup_controller * Fix test_mysql_role * Fix server_id in GHA GHA lack a way to pass option to docker's command. Also server_id is not read as a environment variable. So I'm forced to use a config file. * Add back a package to connect to MySQL 8+ * Linting * Refactor test_mysql_user to work with other host than localhost * Refactor way tests info are passed from sed to file with lookup The idea is to avoid modifying test targets from the workflow to prevent ansible-test to think every tests needs to be run. * Fix missing var * Refactor test to use the db_version from setup_controller * Add temporary files to .gitignore * Fix volume path * Fix volume path by adding a final / * Fix volume path using $(pwd) * Fix volume path using github.workspace var * Cut files from gitignore because it prevents ansible-test to copy them * Fix pre-test-cmd missing separators * Cut the newline added by lookup 'file' * Fix tailing newline by not created it in the first place * Disable tests to concentrate on the \n and quote issue with my files * Fix trailing newline and quote in db_engine_version * Re-enable integration tests to validate db_engine_version is fixed * lint * Cut unused file * Fix pre-test-cmd paste in wrong context * Re-enable service containers * Add back docker healthcheck on services I saw in the GHA logs that it perform an healtcheck ! So I hope this will work. * Add tmate to debug the server_id in replicas * Attempt to fix "invalid syntax" * Enclose command in quotes * Refactor the way server_id is set for replicas The simple way is to add '--server-id 2' after the name of the image of the container. But GHA doesn't let us do that. The idea of mount a file from our repo doesn't work because the repo is check out later in the workflow and I failed to find a pre-job hook. Then I realized that this MySQL option is dynamic! So we will set that in the test target! * Re-activate all tests * Cut useless task * Use same variable as other target for consistency * Linting * Update version tested * Add options to the makefile * Add same variables as other target for consistency * Add IF NOT EXISTS to prevent misleading error on retry * Cut python 3.11 not supported by ansible-test yet * Attempt to set log-bin into docker * Reformat for readability * Document that full version is mandatory * Fix newline * Github complain it doesn't find python 3.1 !!! * Add option to run only a single target * Fix mysqlclient not supporting Python 3.9 * Enhance installation of mysql_client Initially I wanted to install mysql-client-5.7 to test mysql server 5.7 but this package is not available for Ubuntu 18+. I keep those changes because it allow us to specify the name of the package based on the Ubuntu version. * Linting * Add unique name to simplify debugging * Fix mysql_dump for MySQL 5.7 and MariaDB when using mysqldump 8 * Add unique name to simplify debugging * Deduplicate tasks * Lining * Add python script to recreate the test matrix from github workflow file * Fix dump with mysqldump 8 against mysql 5.7 * Disable test for replication with chanel for mysql 5.7 * Add better task name * Fix exclusion function * Disable replication with channel tests entirely for MySQL 5.7 * Activate Mysql 8 and Mariadb into GitHub Action Workflow * Cut Ansible since we can't change what the user have on his computer * Add running make command for all tests of the matrix * Add unique test names * Document run_all_tests.py * Add unique test names * Add tmate to experiment with docker healthcheck * Fix replication settings sh don't know 'echo -e', so we use bash instead. Also, we need to wait for the container to be healthy before trying to restart it. Otherwise that could corrupt it. * Add TODO verify that the version of mysql/mariadb is correct * Add more descriptive tests names * Use mysql_host var name instead of gateway_addr in tests * Refactor user@<gateway ip addr> into user@% * Fix healthcheck in GHA * Disable tests that fails only on MariaDB * Refactor to remove useless variables * Workaround for plugin role that fails with any MariaDB versions * Fix Python 3.10 beein run as 3.1 * Ensure replicas are healthy before rebooting them * Enable all tests * Add a virtualenv for ansible-test used locally * Simplify connector_name variables * Add PoC using custom ansible-test containers * Fix docker_container variable name * Cut forgotten comment * Fix error when using local registry by using quay.io * Change tag of test-containers to latest * Fix ansible-test unknown option I copied blindly https://github.com/ansible-collections/community.sops/blob/main/.github/workflows/ansible-test.yml#L195 and forgot what ansible-test was expecting * Cut column-statistics disabling Thanks to our test-container, we now use the correspond mysql-client. So to test mysql 5.7 we use mysql-client-5.7 and to test mysql 8 we use mysql-client-8. * Add manual test matrix (MariaDB 10.6, 10.7 and 10.8 missing) * Fix test matrix Python version should be quoted, otherwise 3.10 become 3.1 We can skip 2.14 and devel with Python3.8 We can skip devel with Python 3.9 We can skip MariaDB 10.4 with mysql-client-10.6 Add tests for MariaDB 10.6, 10.7 and 10.8 * Reduce number of tests and adapt containers images * Fix queries for roles * Add filter for issues resolved in newer version of mysqlclient * Add names to tests * Fix assertion for mariadb * Linting * Cut tests for incompatible MySQL 8 and pymysql 0.7.11 * Fix assertion for older mysqlclient than 2.0.1 with mysql (mariadb ok) * Cut playbook that are now handled by the test-containers * Change timeout from 10 to 30 seconds to let mysql/mariadb restart * Add connector information to the returned values I need to know what python library was used. I had a container with both mysqlclient and pymysql installed and tests used a different connector that what is advertised by the title of integration tests. We need to prevent that otherwise our tests are worth nothing. * Add a verify stage at setup of test to assert all version are correct * Attempt to build and publish an image on ghcr.io * Add latest release of actions and with a context * Add trigger on workflow file edit * Fix env not recognized in the 'on' clause * Add latest tag * Fix insufficient context * Add missing slash * Cut addition of tag 'latest' as GHA does it automatically * Add ghcr.io image for mariadb10.3 python3.8 mysqlclient2.0.1 * Change docker-image workflow to work on all images using matrix * Fix workflow title * Add support for version of mysqlclient * Fix context path * Workaround failed to push ghcr.io Error was: failed to copy: io: read/write on closed pipe * Add back all tests using ghcr.io images * Cut unused images * Fix verify database version Sometimes, version_full contains trailing information (-log). To prevent issues it's best to concatenate major and minor version. * Fix verify for mysqlclient second name MySQLdb * Rename variable for consistency * Fix container name * Add tag 'latest' to images * Cut filter for tests now that the right connector is used * Fix test of mysql/mariadb version in use * Fix python version lookup * Add clean up in "always" phase of the block Because our tests use --retry-on-error, and the first thing the test does is to try to create the database. We must cleanup otherwise if there is a retry, it will throw a misleading "database already exists" error. * Document TODO * Disable tests using pymysql 1.0.2 Many tests are failing but this must be fixed in the plugins in a future PR. * Cut test MySQL 8 with incompatible pymysql 0.7.11 It fails to connect with error about cryptography unsupported * Fix dict key lookup * Fix indentation * Cut tests that was excluded in previous matrix * Enable back sanity and unit tests * Refactor get_driver_version to display name while passing sanity tests * Fix variable name * Fix missing cffi package to connect to MySQL 8 using Python 3.9 * Fix image not found * Split Docker image workflow to rebuild only changed Dockerfile My goal is not to save the planet but to make it work. Currently docker/setup-buildx-action@v2 often fails. You have to rerun the workflow multiple times until it succeed. When you do that with the matrix with 15 containers, you never get to the point where they all built successfully. Having separate workflows makes rerun the failing build easier. * Fix verify ansible 'devel' for which the version is unknown Today 'devel' means 2.15, but in the future it will be something else. * Fix ansible version extraction for "devel" * Cut matrix from when build was done in a single workflow * Document fix container name * Add bold * Add option to let containers alive at end of testing * Enhance error handling and doc of get_driver_name and get_driver_version * Migrate tests documentations in their own file * Skip retry-on-error by default and add option to activate it on demand * Rename folder to better purpose * Enable back push and schedule workflow * Rename registry from fork to upstream * Cut Docker Image workflow's filter for branch from my fork * Add changelog fragment * Update supported versions * Rename file for clarity * Cut mariadb non long term releases * Add '-client' to the block title to better explain what it is * Update readme for tested versions of long term release of MariaDB * Attempt to add the workflow to the Action tab * Second attempt to add the workflow to the Action tab * Cut folder re-created by merge from main * Cut filter by branch GHA will build the image using the branch name as tag. So we can safely remove this filter. * Cut changelog item done in ansible-collections#497 * Attempt to fix upload of image under c.mysql instead of my fork * Add debug to buildkitd * Bump setup-buildy-action to latest * Cut dot in image name in attempt to fix buildx bad request 400 error * Sanitize the repository name using metadata-action https://github.com/docker/build-push-action/blob/master/TROUBLESHOOTING.md#repository-name-must-be-lowercase * Document why we use optional checkout action * Cut debugging from setup-buildx-action * Fix workflow to work both on fork and c.mysql repository * Use apt-get instead of apt that not have a stable CLI interface * Use apt-get instead of apt * update docker image path to my personal repo I'm unable to publish under community.mysql. Either it's the dot in the name or I do something wrong with the GITHUB_TOKEN, but we need to test my PR, so I'll use docker images from my fork for now. * Fix test after merge of PR497 * Enhance testing documentation header * Fix installation of ansible venv ansible-test is included in ansible package. Also, on Fedora 37 with python 3.11, pip is missing. By using ensurepip we solve that issue. * Document usage of continue_on_errors * Fix versions used in examples * Add support for systems with unsupported python set as default * Fix cleanup task * Fix variable assignation to the include task * Add forgotten variable to handle unsupported python version * Fix user site-packages not visible in virtualenv * Fix test connection to the database and tasks names * Add create podman network for system missing it. We saw that on a Fedora 33 with Podman 3.3.1, an old system. I didn't find in which release the default network changed and maybe it's defined in the Linux distribution. So in doubt I always attempt to create the network. * Add full path to image to prevent podman asking which registry to use * Add options to enforce recreate containers even if already exists * Reformat command multiline to oneline * Add deletion of anonymous volumes associated with the container * Comment unused variable * Change shebang from python to python3 to avoid confusion with python2 This script is a python3 script. * Add disk and RAM requirements * Cut the 3 from python command to follow shebang recommendations https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/shebang.html * Reformat spelling Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> * Reformat file path Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> * Fix link URI Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> * Fix link URI Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> * Lint Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> * Lint Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> * Add better task name Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> * Move utility task files in their own folder * Refactor using reusable GHA workflows * Fix path to called workflow file * Fix path to use local workflow * Fix cannot specify version when calling local workflows * Attempt to use a fixed repo name in the image name My last attempts produced duplicates images under my name + repo name: laurent-indermuehle/community.mysql. Previously I had only my name. And none of the above are what we want. We want only community.mysql in the image name... * Add called workflow file in the GHA hooks Without this, the containers are not rebuilt when you modify the file built-docker-image.yml. * Rollback to github.repository in container image name This time I think I understood. We publish in the github.repository_owner's namespace. In my case it's laurent-indermuehle and in case of upstream it's ansible-collection. A proof of that: https://github.com/orgs/ansible-collections/packages <- here there is one attempt I did in february to push my branch to the upstream. So, our tests containers will be visible to the whole community, not just community.mysql. --------- Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> (cherry picked from commit 6970aef)

…ers (#517) * Integrations tests : Use containers for more control and verify that versions match expectation (#490) * Add healthcheck to MariaDB before starting the tests This prevent the first test to fail because the db isn't ready yet. * Add % instead of the default 'localhost' since we use remote connection Previously, everything was on localhost. Now ansible-test is in a venv and the db is in a container. The db see the IP address from the podman host (10.88.0.2) * Add mysql_client to the controller * Prepare controller with Podman/Docker Network We use the Podman/Docker network gateway address to communicate between container. I haven't tested Docker. I would have preferred to use a pod but only Podman support it and ansible-test only support the --docker-network option. * Generalize mysql and mariadb version based on container name This way we can split db_engine and db_version and simplify tests. Also this is mandatory to use the matrix.db_engine_version as the image name for our services containers. * Fix replication due to usage of gateway_addr instead of localhost * Refactor setup_mysql into setup_controller * Fix server_id in GHA GHA lack a way to pass option to docker's command. Also server_id is not read as a environment variable. So I'm forced to use a config file. * Refactor test_mysql_user to work with other host than localhost * Refactor way tests info are passed from sed to file with lookup The idea is to avoid modifying test targets from the workflow to prevent ansible-test to think every tests needs to be run. * Refactor test to use the db_version from setup_controller * Add temporary files to .gitignore * Add back docker healthcheck on services I saw in the GHA logs that it perform an healtcheck ! So I hope this will work. * Refactor the way server_id is set for replicas The simple way is to add '--server-id 2' after the name of the image of the container. But GHA doesn't let us do that. The idea of mount a file from our repo doesn't work because the repo is check out later in the workflow and I failed to find a pre-job hook. Then I realized that this MySQL option is dynamic! So we will set that in the test target! * Add IF NOT EXISTS to prevent misleading error on retry * Cut python 3.11 not supported by ansible-test yet * Add option to run only a single target * Disable replication with channel tests entirely for MySQL 5.7 * Activate Mysql 8 and Mariadb into GitHub Action Workflow * Document run_all_tests.py * Fix replication settings sh don't know 'echo -e', so we use bash instead. Also, we need to wait for the container to be healthy before trying to restart it. Otherwise that could corrupt it. * Add more descriptive tests names * Use mysql_host var name instead of gateway_addr in tests * Refactor user@<gateway ip addr> into user@% * Workaround for plugin role that fails with any MariaDB versions * Ensure replicas are healthy before rebooting them * Add a virtualenv for ansible-test used locally * Cut column-statistics disabling Thanks to our test-container, we now use the correspond mysql-client. So to test mysql 5.7 we use mysql-client-5.7 and to test mysql 8 we use mysql-client-8. * Fix test matrix Python version should be quoted, otherwise 3.10 become 3.1 We can skip 2.14 and devel with Python3.8 We can skip devel with Python 3.9 We can skip MariaDB 10.4 with mysql-client-10.6 Add tests for MariaDB 10.6, 10.7 and 10.8 * Fix queries for roles * Add filter for issues resolved in newer version of mysqlclient * Add names to tests * Cut tests for incompatible MySQL 8 and pymysql 0.7.11 * Fix assertion for older mysqlclient than 2.0.1 with mysql (mariadb ok) * Change docker-image workflow to work on all images using matrix * Add support for version of mysqlclient * Fix verify database version Sometimes, version_full contains trailing information (-log). To prevent issues it's best to concatenate major and minor version. * Cut filter for tests now that the right connector is used * Add clean up in "always" phase of the block Because our tests use --retry-on-error, and the first thing the test does is to try to create the database. We must cleanup otherwise if there is a retry, it will throw a misleading "database already exists" error. * Disable tests using pymysql 1.0.2 Many tests are failing but this must be fixed in the plugins in a future PR. * Cut test MySQL 8 with incompatible pymysql 0.7.11 It fails to connect with error about cryptography unsupported * Fix missing cffi package to connect to MySQL 8 using Python 3.9 * Split Docker image workflow to rebuild only changed Dockerfile My goal is not to save the planet but to make it work. Currently docker/setup-buildx-action@v2 often fails. You have to rerun the workflow multiple times until it succeed. When you do that with the matrix with 15 containers, you never get to the point where they all built successfully. Having separate workflows makes rerun the failing build easier. * Add option to let containers alive at end of testing * Migrate tests documentations in their own file * Document usage of continue_on_errors * Add support for systems with unsupported python set as default * Add create podman network for system missing it. We saw that on a Fedora 33 with Podman 3.3.1, an old system. I didn't find in which release the default network changed and maybe it's defined in the Linux distribution. So in doubt I always attempt to create the network. * Add full path to image to prevent podman asking which registry to use * Add options to enforce recreate containers even if already exists * Add deletion of anonymous volumes associated with the container * Change shebang from python to python3 to avoid confusion with python2 This script is a python3 script. * Add disk and RAM requirements * Cut the 3 from python command to follow shebang recommendations https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/shebang.html * Reformat file path Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> * Move utility task files in their own folder * Add called workflow file in the GHA hooks Without this, the containers are not rebuilt when you modify the file built-docker-image.yml. * Rollback to github.repository in container image name This time I think I understood. We publish in the github.repository_owner's namespace. In my case it's laurent-indermuehle and in case of upstream it's ansible-collections. A proof of that: https://github.com/orgs/ansible-collections/packages <- here there is one attempt I did in february to push my branch to the upstream. So, our tests containers will be visible to the whole community, not just community.mysql. --------- Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> (cherry picked from commit 6970aef) * Add changelog fragment * Disable tests that doesn't work on stable-1 It's shameful to disable tests, but they didn't makes much sense anyway. * Fix error message being different on stable-1 than v3 * Disable tests against MariaDB 10.5 and 10.6 Does version have never been tested on stable-1 and many tests fails.

…ers (#519) * Integrations tests : Use containers for more control and verify that versions match expectation (#490) * Add healthcheck to MariaDB before starting the tests This prevent the first test to fail because the db isn't ready yet. * Add % instead of the default 'localhost' since we use remote connection Previously, everything was on localhost. Now ansible-test is in a venv and the db is in a container. The db see the IP address from the podman host (10.88.0.2) * Add mysql_client to the controller * Prepare controller with Podman/Docker Network We use the Podman/Docker network gateway address to communicate between container. I haven't tested Docker. I would have preferred to use a pod but only Podman support it and ansible-test only support the --docker-network option. * Generalize mysql and mariadb version based on container name This way we can split db_engine and db_version and simplify tests. Also this is mandatory to use the matrix.db_engine_version as the image name for our services containers. * Fix replication due to usage of gateway_addr instead of localhost * Refactor setup_mysql into setup_controller * Fix server_id in GHA GHA lack a way to pass option to docker's command. Also server_id is not read as a environment variable. So I'm forced to use a config file. * Refactor test_mysql_user to work with other host than localhost * Refactor way tests info are passed from sed to file with lookup The idea is to avoid modifying test targets from the workflow to prevent ansible-test to think every tests needs to be run. * Refactor test to use the db_version from setup_controller * Add temporary files to .gitignore * Add back docker healthcheck on services I saw in the GHA logs that it perform an healtcheck ! So I hope this will work. * Refactor the way server_id is set for replicas The simple way is to add '--server-id 2' after the name of the image of the container. But GHA doesn't let us do that. The idea of mount a file from our repo doesn't work because the repo is check out later in the workflow and I failed to find a pre-job hook. Then I realized that this MySQL option is dynamic! So we will set that in the test target! * Add IF NOT EXISTS to prevent misleading error on retry * Cut python 3.11 not supported by ansible-test yet * Add option to run only a single target * Disable replication with channel tests entirely for MySQL 5.7 * Activate Mysql 8 and Mariadb into GitHub Action Workflow * Document run_all_tests.py * Fix replication settings sh don't know 'echo -e', so we use bash instead. Also, we need to wait for the container to be healthy before trying to restart it. Otherwise that could corrupt it. * Add more descriptive tests names * Use mysql_host var name instead of gateway_addr in tests * Refactor user@<gateway ip addr> into user@% * Workaround for plugin role that fails with any MariaDB versions * Ensure replicas are healthy before rebooting them * Add a virtualenv for ansible-test used locally * Cut column-statistics disabling Thanks to our test-container, we now use the correspond mysql-client. So to test mysql 5.7 we use mysql-client-5.7 and to test mysql 8 we use mysql-client-8. * Fix test matrix Python version should be quoted, otherwise 3.10 become 3.1 We can skip 2.14 and devel with Python3.8 We can skip devel with Python 3.9 We can skip MariaDB 10.4 with mysql-client-10.6 Add tests for MariaDB 10.6, 10.7 and 10.8 * Fix queries for roles * Add filter for issues resolved in newer version of mysqlclient * Add names to tests * Cut tests for incompatible MySQL 8 and pymysql 0.7.11 * Fix assertion for older mysqlclient than 2.0.1 with mysql (mariadb ok) * Change docker-image workflow to work on all images using matrix * Add support for version of mysqlclient * Fix verify database version Sometimes, version_full contains trailing information (-log). To prevent issues it's best to concatenate major and minor version. * Cut filter for tests now that the right connector is used * Add clean up in "always" phase of the block Because our tests use --retry-on-error, and the first thing the test does is to try to create the database. We must cleanup otherwise if there is a retry, it will throw a misleading "database already exists" error. * Disable tests using pymysql 1.0.2 Many tests are failing but this must be fixed in the plugins in a future PR. * Cut test MySQL 8 with incompatible pymysql 0.7.11 It fails to connect with error about cryptography unsupported * Fix missing cffi package to connect to MySQL 8 using Python 3.9 * Split Docker image workflow to rebuild only changed Dockerfile My goal is not to save the planet but to make it work. Currently docker/setup-buildx-action@v2 often fails. You have to rerun the workflow multiple times until it succeed. When you do that with the matrix with 15 containers, you never get to the point where they all built successfully. Having separate workflows makes rerun the failing build easier. * Add option to let containers alive at end of testing * Migrate tests documentations in their own file * Document usage of continue_on_errors * Add support for systems with unsupported python set as default * Add create podman network for system missing it. We saw that on a Fedora 33 with Podman 3.3.1, an old system. I didn't find in which release the default network changed and maybe it's defined in the Linux distribution. So in doubt I always attempt to create the network. * Add full path to image to prevent podman asking which registry to use * Add options to enforce recreate containers even if already exists * Add deletion of anonymous volumes associated with the container * Change shebang from python to python3 to avoid confusion with python2 This script is a python3 script. * Add disk and RAM requirements * Cut the 3 from python command to follow shebang recommendations https://docs.ansible.com/ansible-core/devel/dev_guide/testing/sanity/shebang.html * Reformat file path Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> * Move utility task files in their own folder * Add called workflow file in the GHA hooks Without this, the containers are not rebuilt when you modify the file built-docker-image.yml. * Rollback to github.repository in container image name This time I think I understood. We publish in the github.repository_owner's namespace. In my case it's laurent-indermuehle and in case of upstream it's ansible-collections. A proof of that: https://github.com/orgs/ansible-collections/packages <- here there is one attempt I did in february to push my branch to the upstream. So, our tests containers will be visible to the whole community, not just community.mysql. --------- Co-authored-by: Jorge Rodriguez (A.K.A. Tiriel) <jorge.rodriguez@futurice.com> (cherry picked from commit 6970aef) * Add changelog fragment * Disable tests that doesn't work on stable-1 It's shameful to disable tests, but they didn't makes much sense anyway. * Fix error message being different on stable-1 than v3 * Disable tests against MariaDB 10.5 and 10.6 Does version have never been tested on stable-1 and many tests fails.

…sible-collections#490. (ansible-collections#514) * Fix explanation about containers images * Add definitive URI to the containers images * Document that new images must be set as public * Add makefile options possible values * Document that any mysql and mariadb tag can be use * Add computation of docker_image path * Refactor pre-command to separate commands for cleaner GHA output * Refactor to use GHA test matrix * Cut docker_image from documentation since it's now automatic * Document how to use run_all_test.py to display the test matrix * Temp: Add path to images in my fork to validate integrations tests (cherry picked from commit e2aa655)

…h integration tests after merge of #490 (#520) * Fix issues and documentation with integration tests after merge of #490. (#514) * Fix explanation about containers images * Add definitive URI to the containers images * Document that new images must be set as public * Document makefile options possible values * Document that any mysql and mariadb tag can be used * Add computation of docker_image path * Refactor pre-command to separate commands for cleaner GHA output * Refactor to use GHA test matrix * Cut docker_image from documentation since it's now automatic * Document how to use run_all_test.py to display the test matrix (cherry picked from commit e2aa655) * Add changelog fragment * Cut tests for MariaDB 10.5 and 10.6 that never worked with stable-1

…h integration tests after merge of #490 (#521) * Fix issues and documentation with integration tests after merge of #490. (#514) * Fix explanation about containers images * Add definitive URI to the containers images * Document that new images must be set as public * Document makefile options possible values * Document that any mysql and mariadb tag can be use * Add computation of docker_image path * Refactor pre-command to separate commands for cleaner GHA output * Refactor to use GHA test matrix * Cut docker_image from documentation since it's now automatic * Document how to use run_all_test.py to display the test matrix (cherry picked from commit e2aa655) * Add changelog fragment

webknjaz · 2023-07-27T11:56:13Z

.github/workflows/docker-image-mariadb103-py38-mysqlclient201.yml

+
+jobs:
+
+  call-workflow-passing-data:


@laurent-indermuehle why did you go for multiple reusable workflow invocations? This causes a lot of duplication, which is prone to human error.

Besides, multiple separate workflows may fight for the resources more, aren't consolidated visually and make it hard to have unified merge gates…

Have you tried wrapping this into a matrix? I'm pretty sure this would work combined with reusable workflows.

@webknjaz you raise a good question. I think my main goal was to rebuild the containers only if needed. Do you think it would be possible to combine a matrix and an "on: push: paths + branch_ignore: stable-*" ?

Yeah, it's 💯 possible with an extra small decision-making job that calculates conditions or even better — generates the matrix. I've been meaning to make an example PR but it fell off my radar. Maybe, later...

@webknjaz I'm finally attempting to simplify the build of the test containers as you suggested. I've done a first attempt in #643. I would love to have your input on this if you have the time.

laurent-indermuehle added 30 commits December 9, 2022 15:22

Draft: Add a mariadb container

86188a2

Add playbook to test connection to the server

5bd358e

Add healthcheck to MariaDB before starting the tests

d781636

This prevent the first test to fail because the db isn't ready yet.

Add default file for root necessary since using venv instead of docker

8c58068

Add % instead of the default 'localhost' since we use remote connection

c895a86

Previously, everything was on localhost. Now ansible-test is in a venv and the db is in a container. The db see the IP address from the podman host (10.88.0.2)

Add ansible-test integration inventory to .gitignore

f2c2173

Revert to old workflow to use ansible-test --venv

d06cefa

It seams that that ansible-test-gh-action doesn't handle this option: https://github.com/ansible-community/ansible-test-gh-action/blob/main/action.yml#L483-L497

Cut target filtering

d98aa89

Fix comparison

01e664f

We are not logged in as 127.0.0.1 anymore, but 10.88... as I couldn't test this easily, I decided to simplify the test.

Add path to default-file

eec9f97

/root doesn't exist with --venv

Merge branch 'main' into lie_tests_using_containers

574fef5

Fix workflow unknown option container_name

634d1fb

Cut anchors currently unsupported by GHA

589a0de

Disable healthcheck

60c87ab

I want to first prove that this setup is possible before adding safety

Disable sanity, units and matrix to speed up tests in GHA

8652a9a

Further disable tests to speed up

cd9f216

Add mysql_client to the controller

0cd57ad

Install mysql_client the correct way

676509e

Fix package name and missing apt cache

0666669

Prepare controller with Podman/Docker Network

b72ca0d

We use the Podman/Docker network gateway address to communicate between container. I haven't tested Docker. I would have preferred to use a pod but only Podman support it and ansible-test only support the --docker-network option.

Swap MariaDB with MySQL

4b6617a

De-duplicate the mysql_command alias

90266f1

Generalize mysql and mariadb version based on container name

b821db9

This way we can split db_engine and db_version and simplify tests. Also this is mandatory to use the matrix.db_engine_version as the image name for our services containers.

Cut docker healthcheck unsupported by GHA

c3458f4

Fix replication server_id already in use

1e03a25

Add static test with replication containers

87acd3d

Fix database not selected

72b3ecb

Fix replication due to usage of gateway_addr instead of localhost

17f8ece

Simplify version computation

1c863da

laurent-indermuehle added 2 commits March 7, 2023 17:44

Add called workflow file in the GHA hooks

c0e8077

Without this, the containers are not rebuilt when you modify the file built-docker-image.yml.

Andersson007 approved these changes Mar 10, 2023

View reviewed changes

Andersson007 merged commit 6970aef into ansible-collections:main Mar 21, 2023

laurent-indermuehle mentioned this pull request Mar 21, 2023

Fix issues and documentation with integration tests after merge of #490. #514

Merged

laurent-indermuehle added backport-2 labels Mar 24, 2023

laurent-indermuehle mentioned this pull request Mar 27, 2023

[PR #490/6970aef backport][stable-1] Integrations tests : Use containers #517

Merged

This was referenced Mar 27, 2023

[PR #490/6970aef backport][stable-2] Integrations tests : Use containers #518

Closed

[PR #490/6970aef backport][stable-2] Integrations tests : Use containers #519

Merged

laurent-indermuehle deleted the lie_tests_using_containers branch March 29, 2023 07:00

webknjaz reviewed Jul 27, 2023

View reviewed changes

laurent-indermuehle mentioned this pull request Jun 20, 2024

[CI] Remove ansible-test custom containers #650

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrations tests : Use containers for more control and verify that versions match expectation #490

Integrations tests : Use containers for more control and verify that versions match expectation #490

laurent-indermuehle commented Jan 23, 2023 •

edited

Loading

laurent-indermuehle commented Mar 7, 2023

Andersson007 commented Mar 8, 2023

laurent-indermuehle commented Mar 8, 2023

Andersson007 left a comment

Jorge-Rodriguez commented Mar 10, 2023

laurent-indermuehle commented Mar 15, 2023

Andersson007 commented Mar 21, 2023

patchback bot commented Mar 24, 2023 •

edited

Loading

patchback bot commented Mar 24, 2023 •

edited

Loading

webknjaz Jul 27, 2023

laurent-indermuehle Jul 31, 2023

webknjaz Aug 4, 2023

laurent-indermuehle Jun 6, 2024


		jobs:

		call-workflow-passing-data:

Integrations tests : Use containers for more control and verify that versions match expectation #490

Integrations tests : Use containers for more control and verify that versions match expectation #490

Conversation

laurent-indermuehle commented Jan 23, 2023 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

laurent-indermuehle commented Mar 7, 2023

Andersson007 commented Mar 8, 2023

laurent-indermuehle commented Mar 8, 2023

Andersson007 left a comment

Choose a reason for hiding this comment

Jorge-Rodriguez commented Mar 10, 2023

laurent-indermuehle commented Mar 15, 2023

Andersson007 commented Mar 21, 2023

patchback bot commented Mar 24, 2023 • edited Loading

Backport to stable-2: 💔 cherry-picking failed — conflicts found

Backporting merged PR #490 into main

patchback bot commented Mar 24, 2023 • edited Loading

Backport to stable-1: 💔 cherry-picking failed — conflicts found

Backporting merged PR #490 into main

webknjaz Jul 27, 2023

Choose a reason for hiding this comment

laurent-indermuehle Jul 31, 2023

Choose a reason for hiding this comment

webknjaz Aug 4, 2023

Choose a reason for hiding this comment

laurent-indermuehle Jun 6, 2024

Choose a reason for hiding this comment

laurent-indermuehle commented Jan 23, 2023 •

edited

Loading

patchback bot commented Mar 24, 2023 •

edited

Loading

patchback bot commented Mar 24, 2023 •

edited

Loading