about the integration test method

[sky-joker]

PURPOSE

Here we discuss the integration test method for the Zabbix module.
It provisioning for test environment uses docker-compose? or ansible-test integration command? (and so on)

[D3DeFi]

I believe that ansible-test is not an option. If I understood it right, it creates isolated containers with a base OS in a non routable network. The other thing is that the CI will be unnecessary slow if we are to set up MySQL and Zabbix services ourselves.

My opinion here is that we will benefit greatly from a pre-built Zabbix docker containers, but just a minimal setup possible with zabbix-server + mysql containers and maybe zabbix-web if that is required for a working API.

The final goal is to have a single script (or rather playbook) that will be run from a Github Actions and can be run locally by anyone without installing a ton of dependencies. Pseudo code can go like so:

docker-compose
for plugin in plugins/modules/; test plugin
cleanup

We can probably reuse your container provisioner for this purpose. What do you think?

[gundalow]

Have a look at how Grafana integration tests work
They use GitHub actions to spin up the Docker container for the three currently supported Grafana versions: https://github.com/ansible-collections/grafana/blob/master/.github/workflows/ansible-test.yml#L55-L59

Note that when using this you can't use the --docker option to ansible-test as you will end up with the Zabbix container and ansible-test container in isolated networks: https://github.com/ansible-collections/grafana/blob/master/.github/workflows/ansible-test.yml#L80

[sky-joker]

Thank you @gundalow for information!
To run Zabbix containers, use official Zabbix images and SQL images.
So I think it would be better to use docker-compose to solve container dependencies easily.
If it does CI like Grafana, I think we need to create a single container image.

@D3DeFi

My opinion here is that we will benefit greatly from a pre-built Zabbix docker containers, but just a minimal setup possible with zabbix-server + mysql containers and maybe zabbix-web if that is required for a working API.

I think so too.
I tried making a sample for CI.

https://github.com/sky-joker/community.zabbix-sample1-ci

Sample docker-compose

https://github.com/sky-joker/community.zabbix-sample1-ci/blob/master/docker-compose.yml

Sample Workflow

https://github.com/sky-joker/community.zabbix-sample1-ci/blob/master/.github/workflows/ansible-test.yml

CI result

https://github.com/sky-joker/community.zabbix-sample1-ci/runs/511571372

I used zabbix_user for sample testing.
In order to make the content of the test easier to understand, we only conduct the minimum test.
Does it make sense?

[D3DeFi]

Thanks for the example and initial setup @sky-joker!

Looking at the actions documentation, I am still not sure whether they can be used to specify container dependencies as compose can.

Even if it would be possible, we would probably still be using something to test local changes before pushing them. Might as well be the same thing and actions cannot (true?) be run locally.

So, lets continue with docker compose? Unless @gundalow says that everything I said is wrong.

[gundalow]

My docker knowledge is very basic.

Could you please expand on

I am still not sure whether they can be used to specify container dependencies as compose can.

https://github.com/sky-joker/community.zabbix-sample1-ci/blob/master/.github/workflows/ansible-test.yml looks good to me.

How were you thinking that the test would run against Zabbix version 3.0,4.0,4.4?

[D3DeFi]

My docker knowledge is very basic.

So there are two of us :)

I am still not sure whether they can be used to specify container dependencies as compose can.

What I meant was linking containers. In order for zabbix to work it needs to link these three:

zabbix-web -> zabbix-server -> zabbix-mysql

In a docker command form:

docker run --name zabbix-web --link mysql-server:mysql --link zabbix-server-mysql:zabbix-server...

It seems that documentation of actions always refers to a stand-alone containers and nothing about linking 2 containers together, except that all containers expose all ports to each other. To be honest, I've only scrolled through it very briefly and haven;t had a chance to play with it.

How were you thinking that the test would run against Zabbix version 3.0,4.0,4.4?

@sky-joker used a loop to run a test against each container (tasks file). Maybe define a custom inventory:

zbx3.0-container ansible_host=IP
zbx4.0-container ansible_host=IP
zbx4.4-container ansible_host=IP

and use server_url: "http://{{ ansible_host }}" instead?

[sky-joker]

if when integration test uses ansible-test command, I look like it can't specify inventory file.
The following file is default created.

# Do not put test specific entries in this inventory file.
# For script based test targets (using runme.sh) put the inventory file in the test's directory instead.

[testgroup]
# ansible_python_interpreter must be set to avoid interpreter discovery
testhost ansible_connection=local ansible_python_interpreter="{{ ansible_playbook_python }}"

I didn't find an option to specify the inventory file for ansible-test command.

(venv) [root@zbxcont zabbix]# ansible-test integration -h | grep inventory

[D3DeFi]

oh well, so in order to use ansible-test, the testing needs to run directly from zabbix-web-nginx-mysql container for example.

I've been trying to simulate our docker compose in Github actions with no luck so far. The problem is, that zabbix-server-mysql needs an ENV variable DB_SERVER_HOST set to the name of the mysql container, which would be zabbix30-db in our case. This name needs to simultaneously exist in /etc/hosts of the zabbix-server-mysql. That is the theory at least.

What happens in practice is that Github actions reserve --name for randomly generated string and set --alias zabbix30-db --alias ${{ job.services.zabbix30-db.id }} in this order. The newly created zabbix-server-mysql container then ends with /etc/hosts configured like so:

127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.18.0.2	7d34e663ca66

You can see it in this build: https://github.com/D3DeFi/community.zabbix/runs/524346301?check_suite_focus=true

My attempts to:

• set env DB_SERVER_HOST: ${{ job.services.zabbix30-db.id }}
• add options --add-host: zabbix30-db:${{ job.services.zabbix30-db.ip }} not sure if something like this even exists. The documentation is horrible when it comes to services of Github actions
• override name via options as --name zabbix30-db

all failed. In order to make this work via Github actions, we somehow need to fill DB_SERVER_HOST with a container id of mysql container.

As for the CI definition, this was one of my most promising ones:

jobs:
  integration:
    runs-on: ubuntu-latest
    container:
      image: zabbix/zabbix-web-nginx-mysql:ubuntu-4.4.0
      env:
        DB_SERVER_HOST: "zabbix44-db"  # how to get container ID here?
        MYSQL_USER: "zabbix"
        MYSQL_PASSWORD: "zabbix"
        MYSQL_DATABASE: "zabbix"
        MYSQL_ROOT_PASSWORD: "zabbix"
        ZBX_SERVER_HOST: "zabbix44-server"
      ports:
        - 8030:80

    services:
      zabbix44-db:
        image: mysql:5.7
        env:
          MYSQL_DATABASE: "zabbix"
          MYSQL_USER: "zabbix"
          MYSQL_PASSWORD: "zabbix"
          MYSQL_ROOT_PASSWORD: "zabbix"
        options: --name zabbix44-db

      zabbix44-server:
        image: zabbix/zabbix-server-mysql:ubuntu-4.4.0
        env:
          DB_SERVER_HOST: "zabbix44-db"
          MYSQL_USER: "zabbix"
          MYSQL_PASSWORD: "zabbix"
          MYSQL_DATABASE: "zabbix"
          MYSQL_ROOT_PASSWORD: "zabbix"

[sky-joker]

hmm...
I think there is a limit to preparing a container environment with GitHub Actions alone.
If you prepare a container environment only with GitHub Actions, I think that it is easier to create one package container(web+SQL+server) and use it.
However, in that case, we need to maintain a test container.

[D3DeFi]

I think there is a limit to preparing a container environment with GitHub Actions alone.

Yeah, it is probably like this. Maybe it is possible to link multiple containers, but documentation is really unclear about that. Only thing stated there is that all containers for Github action are on the same network and all open ports are exposed to each container. Seems to me that something like that should be possible.

However, in that case, we need to maintain a test container.

What is your opinion on this @sky-joker? Judging by how active we both are, I would say it is bigger overhead than we need right now. Currently, docker-compose looks to me as the best solution, even tho we need to work around the localhost problem that ansible-test introduces.

Maybe crazy idea, but there is still a possibility to omit ansible-test for integration testing at all.

[dj-wasabi]

oh well, so in order to use ansible-test, the testing needs to run directly from zabbix-web-nginx-mysql container for example.

I've been trying to simulate our docker compose in Github actions with no luck so far. The problem is, that zabbix-server-mysql needs an ENV variable DB_SERVER_HOST set to the name of the mysql container, which would be zabbix30-db in our case. This name needs to simultaneously exist in /etc/hosts of the zabbix-server-mysql. That is the theory at least.

What happens in practice is that Github actions reserve --name for randomly generated string and set --alias zabbix30-db --alias ${{ job.services.zabbix30-db.id }} in this order. The newly created zabbix-server-mysql container then ends with /etc/hosts configured like so:

127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.18.0.2	7d34e663ca66

You can see it in this build: https://github.com/D3DeFi/community.zabbix/runs/524346301?check_suite_focus=true

My attempts to:

• set env DB_SERVER_HOST: ${{ job.services.zabbix30-db.id }}
• add options --add-host: zabbix30-db:${{ job.services.zabbix30-db.ip }} not sure if something like this even exists. The documentation is horrible when it comes to services of Github actions
• override name via options as --name zabbix30-db

all failed. In order to make this work via Github actions, we somehow need to fill DB_SERVER_HOST with a container id of mysql container.

As for the CI definition, this was one of my most promising ones:

jobs:
  integration:
    runs-on: ubuntu-latest
    container:
      image: zabbix/zabbix-web-nginx-mysql:ubuntu-4.4.0
      env:
        DB_SERVER_HOST: "zabbix44-db"  # how to get container ID here?
        MYSQL_USER: "zabbix"
        MYSQL_PASSWORD: "zabbix"
        MYSQL_DATABASE: "zabbix"
        MYSQL_ROOT_PASSWORD: "zabbix"
        ZBX_SERVER_HOST: "zabbix44-server"
      ports:
        - 8030:80

    services:
      zabbix44-db:
        image: mysql:5.7
        env:
          MYSQL_DATABASE: "zabbix"
          MYSQL_USER: "zabbix"
          MYSQL_PASSWORD: "zabbix"
          MYSQL_ROOT_PASSWORD: "zabbix"
        options: --name zabbix44-db

      zabbix44-server:
        image: zabbix/zabbix-server-mysql:ubuntu-4.4.0
        env:
          DB_SERVER_HOST: "zabbix44-db"
          MYSQL_USER: "zabbix"
          MYSQL_PASSWORD: "zabbix"
          MYSQL_DATABASE: "zabbix"
          MYSQL_ROOT_PASSWORD: "zabbix"

If you have a docker-compose configuration file, you can also create a docker network. See example: https://github.com/dj-wasabi/consul-acl-examples/blob/master/multiple-dc/docker-compose.yml

When all of the images are in the same docker network, you can use the name of the container to access them, because the docker network handles the DNS requests and thus no entry in the hosts file of the container needs to exist.

But I have no idea if this docker compose you referring to with Github actions is the same as the docker-compose you can run locally on any host.

My 2 cents. 😄

[D3DeFi]

If you have a docker-compose configuration file, you can also create a docker network. See example: https://github.com/dj-wasabi/consul-acl-examples/blob/master/multiple-dc/docker-compose.yml

When all of the images are in the same docker network, you can use the name of the container to access them, because the docker network handles the DNS requests and thus no entry in the hosts file of the container needs to exist.

But I have no idea if this docker compose you referring to with Github actions is the same as the docker-compose you can run locally on any host.

My 2 cents. 😄

Welcome to the debate @dj-wasabi ! :) We can use docker-compose just fine, that is not a problem. Only workaround needed in this setup is to run ansible-test command from "localhost" (that needs to be the zabbix-web-mysql-nginx for example) as the ansible-test --docker would break the networking. In this configuration, Github actions will run docker-compose and prepare docker network for us.

What I was trying to prove with that example was to mirror behavior of docker-compose in Github actions without using docker-compose at all. Just to test that we can use native features - Github actions container + services alone. Documentation states that docker network is created for you automatically and all containers can see each other.

Only problem is with resolving as Github actions name the containers with IDs. Maybe there is a way to make this work, but I haven't had time to dive into it again since then. Only missing puzzle is to configure ENV variables like DB_SERVER_HOST and ZABBIX_SERVER_HOST to something resolvable.

[sky-joker]

Judging by how active we both are, I would say it is bigger overhead than we need right now.

yes.
I think the best idea right now is to use docker-compose.
Using docker-compose, it easily build the same integration test environment in local environment :)
(However, it is assumed that docker and docker-compose are installed localhost.)

For example, how about specifying the version of the container image in the environment variable?

CI Results

https://github.com/sky-joker/community.zabbix-sample1-ci/runs/541636476

docker-compose file

https://github.com/sky-joker/community.zabbix-sample1-ci/blob/sample02/docker-compose.yml

GitHub Flow

https://github.com/sky-joker/community.zabbix-sample1-ci/blob/sample02/.github/workflows/ansible-test.yml

This way, integration testing can be done in parallel in GitHub Actions :)

[D3DeFi]

Looks good to me. If @gundalow agrees with such setup I would say it is already the time to implement something for real.

[gundalow]

Judging by how active we both are, I would say it is bigger overhead than we need right now.

Start simple, and try not to set yourself up for too much work.

Looks good to me. If @gundalow agrees with such setup I would say it is already the time to implement something for real.

If you are happy, and it's an improvement over what we have today, then go for it :)

[D3DeFi]

It definitely is :) @sky-joker can you open an official PR for the integration testing with the docker compose you've built? I think the use of ENV variables is fine

[sky-joker]

okay :)
I will prepare to PR.