[BUG] kitchen tests don't work

[NutipA]

---

Bug details

Describe the bug

I've tried to test docker-formula using kitchen, but nothing from my runs was successful. I've tried not all platforms but around half of them. For my tries, there was a problem with installing the docker package, but the exact reasons were different in each try.

Steps to reproduce the bug

kitchen test

Expected behaviour

All tests for all platforms in .kitchen.yaml should pass

[myii]

@NutipA Thanks for the report. The tests are being run via. Travis so the instances listed there are all definitely working:

• https://travis-ci.com/saltstack-formulas/docker-formula/builds/133237981

As a first step, please confirm if you can get these working or not locally at your end.

[NutipA]

@myii thanks for the reply. Unfortunately, I don't know much about travis, but I'll try to inspect its output and rerun locally tests for those platforms.

[NutipA]

@myii I've run locally all instances from your travis link. Results:

• 172.2 default-debian-10-develop-py3 is not defined in kitchen.yml thus cannot be tested locally
• 172.3 default-debian-9-2019-2-py3 test passed
• 172.4 default-ubuntu-1804-2019-2-py3 test passed
• 172.5 default-arch-base-latest-2019-2-py2 failed from docker-package-dependencies state
• 172.6 default-debian-9-2018-3-py2 test passed
• 172.7 default-ubuntu-1604-2017-7-py2 failed from docker-service state

As for me, right now I need to use docker-formula for centos. I've tested all centos items, none of them could finish kitchen converge stage. For example for default-centos-7-2019-2-py2 it failed from the docker-package state.

[myii]

• 172.2 default-debian-10-develop-py3 is not defined in kitchen.yml thus cannot be tested locally

@NutipA Yes, unfortunately the last Travis run was from before the last update to kitchen.yml, when we upgraded a number of the pre-salted images. It would have been better if I pointed you to my fork instead:

• https://travis-ci.org/myii/docker-formula/builds/604703407

I.e. It will now be default-debian-10-master-py3 instead of default-debian-10-develop-py3, etc. You can look in the latest .travis.yml for the exact invocations.

As for me, right now I need to use docker-formula for centos. I've tested all centos items, none of them could finish kitchen converge stage. For example for default-centos-7-2019-2-py2 it failed from the docker-package state.

Kitchen-based testing was only introduced to this repo very recently, so there's work to be done. For example, all of the centos instances are currently failing:

• https://travis-ci.org/myii/docker-formula/builds/607579648

I've gone through and "quick-fixed" centos-7 (probably needs further reviewing before it can be merged into this repo):

• https://travis-ci.org/myii/docker-formula/jobs/607598143
• The commit used to "fix" things: myii@fd56aac

The issue is effort; if someone can make the time to go through cleaning things up, we can incorporate them to the formula. Ideally, we can get all of the instances passing in Travis, like we do for some of the formulas out there.

[NutipA]

@myii yeah, I understand. After my tests there is one question bothering me: for both 172.5 and 172.7 my local runs failed, but on travis they passed.

[myii]

@NutipA We have that from time to time, where things work locally and not in Travis and vice versa. Ultimately, there's going to be differences between the host environments for the containers. From a formula perspective, it's more important to have things passing in Travis, to ensure a consistent result for all. I prefer to push a lot of testing to my own GitHub forks, which then run in my own Travis account. Once they are passing there, they always work here.

[noelmcloughlin]

Fixed in merged #256
However, Kitchen test is using Docker driver so salts docker_container and dockercompose states cannot be tested in Travis CI. To support those tests we may need vagrant driver (instead?).