Improve consistency/best practices of docker files

[chuckha]

We put stuff in / and run as root. We probably shouldn't do either of those things if we can help it.

Make sure our sonobuoy & related dockerfiles behave and follow some conventions. Super bonus to this ticket, document our conventions.

[johnSchnake]

Check the sonobuoy image and https://github.com/heptio/sonobuoy-plugin-systemd-logs/blob/master/Dockerfile

Make sure the base image makes sense and that we follow best practices in general.

[zubron]

I spent some time looking at our current Dockerfiles and possible changes we could make. The following is a summary of what I learned:

• We are currently using two separate OS images, debian:stretch-slim for AMD64 and ubuntu:16.04 for ARM64. Looking through the PR that added ARM support, and some related issues (Update base images  #609), it's not clear why these two are different. It's been questioned a few times but I couldn't find a definitive answer. I would like to try and use the same OS for both to avoid possible differences in behaviour.

• Looking at whether we can move the Sonobuoy executable and scripts out of /, I think that is a move that we could make but it would require some care due to how these files are used. If we were only concerned about the images, moving the files would be safe to do. However, the CLI generates a manifest that expects the files to be in a particular location when running the images. If we move the files, that contract is broken. This would be fine in the case where the CLI version matches the image version for new manifests, however it's entirely possible that we could have users with existing manifests, or using a newer image with an older CLI version. We could find a way to transition these locations gradually over the course of a few releases, but I don't know if it's a priority at this stage.

• With regard to running as root, I think this is something we should change. I'm not aware of anything within Sonobuoy requiring root permissions so running as the root user seems unnecessary.

My concern with making the above changes is that we don't have any integration testing in place, particularly with ARM. We have nothing that ensures that those builds are good. Before making significant changes that would rely on manual testing to verify, I would like to get some basic integration testing in place. Ideally something for both architectures, but at least something that could verify that any change in user or location of the files is safe. This is being worked on as part of #668.

I'm going to make some follow up issues to deal with each individual point above and will close this in favour of those issues.