Discussion Dev Setup Docker

[fidoriel]

There are several vagrant issues on my setup.

1. Every Monday I have to reprovision the container. Vagrant throws an OCI Shim error.
2. There is this permission issue.
3. Starting from today, I even cannot rebuild the setup:

+ MOUNTPOINT=/evap
+ USER=evap
+ REPO_FOLDER=/opt/evap
+ ENV_FOLDER=/home/evap/venv
+ NODE_MODULES_FOLDER=/home/evap/node_modules
+ EVAP_PYTHON=python3.10
+ export DEBIAN_FRONTEND=noninteractive
+ DEBIAN_FRONTEND=noninteractive
+ apt-get -q update
Hit:1 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease
Hit:3 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease
Get:4 http://ports.ubuntu.com/ubuntu-ports jammy InRelease [270 kB]
Err:4 http://ports.ubuntu.com/ubuntu-ports jammy InRelease
  The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
Reading package lists...
W: GPG error: http://ports.ubuntu.com/ubuntu-ports jammy InRelease: The following signatures were invalid: BADSIG 871920D1991BC93C Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
E: The repository 'http://ports.ubuntu.com/ubuntu-ports jammy InRelease' is not signed.

VBox even discontinued the arm version for macOS: https://www.virtualbox.org/ticket/21771
So we can assume, arm will not be supported for Arm Guest/Arm Host for Win and Linux hosts.
Also, using docker with vagrant abuses the container idea.

We also had some discussion about testing setups. If we would fully change to a container setup we would reduce the dev setup complexity. We are currently using containers in CI. All provisioning could go into an 15 line Dockerfile. We could have an idiot-proof, reproducible docker-compose file. For new members, we could provide a prebuild development docker container to speedup setup. If we do not want to be responsible for a container. We do not need this option. Docker benefits from docker-layer-caching, to speedup rebuilds and save storage.

I am very frustrated from a lot of breaking changes in my setup. It takes all the energy I would like to put into EvaP. Despite, I have to find a hack every to have a working setup to get around a broken setup. This week is the third in a row. The third is probably an error on Canonicals site. docker-compose/podman-compose is the de facto standard in this area. Vagrant is niche, so troubleshooting is way more difficult and annoying. Maybe we can try to create a docker compose setup for testing and developer feedback?

[niklasmohrin]

Alright, here are my opinions on the various points you brought up:

Vagrant throws an OCI Shim error.

I don't know what this is, can you post some output?

There is this permission issue.

In your issue, I asked what this problem even is, but you haven't responded yet, so I don't know what to think about this point. Is it connected to your first problem?

Starting from today, I even cannot rebuild the setup:

Tried it today, the error does not occur anymore (also in Docker). I think this has nothing to do with Vagrant or VirtualBox, this is just some ubuntu stuff.

VBox even discontinued the arm version for macOS

As far as I can tell, this was only a beta so far anyways and they are just not pushing any more beta builds for 7.0.x. From the thread, I would expect the builds to continue with versions 7.1.0 and 8. In any case, if Docker would still work on M1, I would not see this as an urgent motivation to move away from Vagrant.

Also, using docker with vagrant abuses the container idea.

No, it does not. I don't want to have files from apache, postgres etc. spread across my host filesystem, so I mock up a file system for these tools by using a container - exactly what containers are meant to do.

If we would fully change to a container setup we would reduce the dev setup complexity.

I think we would have to see an actual setup before judging this. My intuition would be that there isn't really a way to reduce complexity of the setup, afterall, we are just setting everything up as we need it. The configuration could possible be more robust with a declarative configuration instead of our imperative bash script though - again, would have to see a working setup.

All provisioning could go into an 15 line Dockerfile

How? Where does all the code we currently have go?

reproducible docker-compose file

Tempting, but I don't think plain docker is the way for reproducibility. If we run apt update in the container (and we will), reproducibility goes out the window.

For new members, we could provide a prebuild development docker container

Interesting idea, I think that would be very cool, if it is turns out faster to setup than building it yourself.

Docker benefits from docker-layer-caching, to speedup rebuilds and save storage.

I don't see how it would be better than our single-layer docker image that is used by vagrant where we only have exactly the stuff that we need and is only rarely rebuilt.

---

We have talked about this before. There are some requirements for the development setup that vagrant has solved really well where other tools do not work too great:

• replicating the setup of the production server (not sure how we stand on running containers in production)
• has a way of running a sandbox in the development vm (for example for frontend tests; does not work in docker)
• few tools needed on the host to get it running (having small containers might lead to us installing more container-related tools in the host)

That said, I think you are well aware that I am also not too happy with our bash script that occasionally swallows errors from commands either. I have been looking into packaging evap with nix for some time now and I like to think that once I could get it to work, it would solve all problems. However, I can't really argue that it is better than what we currently have before I really have something to play around with and there are also some tradeoffs that I haven't made up my mind about yet.

The way I see it, is that there is no reason to abandon vagrant currently before we are sure that the next solution would really improve the aspects we are trying to fix. In that sense, I gladly take a look at any suggested changes to the setup, but can't promise that I will agree with them and even less that we get a consensus to merge it into evap. If I recall correctly, you did start to write a docker-compose file for evap some time back? What happened to that?

[richardebeling]

Additional stuff I'd like to point out:

• You shouldn't have to reprovision the container every week, we intend stopping and restarting to work.
• The OCI Shim stuff seems to be a containerd problem. Keep in mind that Vagrant is basically just a CLI user of docker. If there is something wrong with the docker setup, removing Vagrant will not help.
• As you already said, the apt error is a ubuntu thing, possibly due to a corrupted apt cache. Instructions on how to fix this are available on the internet. I can see how this might happen if you interrupt the provisioning while apt is running. In any way, it has nothing to do with the provider that we use to make a containerized/virtualized ubuntu system available. Removing Vagrant will not help here.
• The file permission error happens because docker doesn't handle file permissions of mounted files consistently across implementations/platforms. This is a problem with docker, we will not fix it by removing Vagrant.
• The "ports keep being used after container is stopped" issue is also fundamentally a docker problem in my opinion. This also will not be fixed by removing Vagrant.

Overall, I think you blame Vagrant for a lot of problems that it is not responsible for.

Our current provisioning script is runnable documentation of how to setup an instance of evap. In the deployment folder, we currently have ~230 lines of setup-logic and configuration templates that are actively used and thus tested during each development-box-setup-procedure. In the developer box, we have a correctly named user with the correct permissions, and we provide python with a venv git, bash with autocompletion and colored output, access to the management script, apache with a working configuration and mod-wsgi, postgres, redis, the possibility to test production updates and rollbacks, the correct version of node.js, and the option to run frontend tests. We have all common linux tools available for debugging. (note: list non-exhaustive). In short: We provide executable setup instructions for a developer box for people wanting to develop evap. Vagrant allows us to turn these instructions into a ready-to-go environment with a 10-letter command (vagrant up).

The developer-box in some way mismatches docker principles because we want to use it as an interactive environment while docker wants a container to non-interactively run one service while "removing" the environment. However, dropping Vagrant and VirtualBox doesn't change anything about this mismatch. We could remove Vagrant from the equation by just copy&pasting the handful of docker commands that it currently issues for us and it would not improve the situation.

That said: For most work on EvaP, you are completely free to build your own development setup the way you would like it to be. If you can replace everything with a 15 line Dockerfile, feel free to do that and use it. Just be aware that we will not easily merge a PR into our codebase if it lacks main features that the current development box has.