Refine docker network configuration

[christophertubbs]

The current process for deploying currently requires some complicated steps revolving around setting up networks. Things in docker are more streamlined now compared to when this project started, so some of the more complicated steps can be smoothed out. For example, the following in one of the deployment scripts can be pushed out to the docker-compose file:

    docker_dev_init_swarm_network ${DOCKER_REQUESTS_NET_NAME:=requests-net} \
            ${DOCKER_REQUESTS_NET_SUBNET:?Cannot init ${DOCKER_REQUESTS_NET_NAME} network without configured subnet} \
            ${DOCKER_REQUESTS_NET_GATEWAY:?Cannot init ${DOCKER_REQUESTS_NET_NAME} network without configured gateway} \
            ${DOCKER_REQUESTS_NET_DRIVER:-overlay}

to:

network:
    requests-net:
        attachable: true
        ipam:
            driver: overlay
            config:
                subnet: ${DOCKER_REQUESTS_NET_SUBNET}
                gateway: ${DOCKER_REQUESTS_NET_GATEWAY}

(just an example - the 'real' solution would still need to be worked out)

The subnet and gateway definitions may no longer be needed. This change will cut down on some extra environment variables and scripts, making the project easier to maintain.

[aaraney]

I'm afk, so I'll check when I get home, but I'm fairly sure that's only applicable to non-docker swarm containers. There is and has been a divide between docker compose and docker swarm and docker swarm does not work seamlessly with docker compose. I'm fairly certain you need to create the networks via the docker swarm api. Fingers crossed I'm wrong.

[aaraney]

Revisiting this, there are several issues as to why this does not work in our context. The main and most important being that the docker compose spec does not provide a way to specify the scope of a given network. See here the struct that a network is marshaled into. Why is this an issue to begin with? For mpi related reasons, we use a macvlan network driver for the mpi-net. @robertbartel can elaborate on this if we need to, I don't know all the details off the top of my head without doing some more digging. To use a macvlan driver in a docker swarm context, you must tell the docker network api that you want the network to be swarm scoped. However, using just docker compose you cant do that. Ive found a few github issues related to this and the general consensus is to just use the overlay network driver which is swarm scoped by default, however as mentioned, we use the macvlan driver for the mpi-net which is not swarm scoped by default.

Another potential issue is network persistence. There is not a straightforward way to just create the networks needed for our deployment and have them live longer than the services themselves. Meaning, you would need to recreate the docker networks each time you redeploy or at least be very careful that you don't destroy the networks when you are doing a service upgrade for example.

[christophertubbs]

Still looking through details, but:

The main and most important being that the docker compose spec does not provide a way to specify the scope of a given network.

I believe overlay networks (which we have to use) are swarm scoped by default, so that saves our asses in this circumstance.

I'm looking through some of the macvlan stuff. Luckily, only one network is set up for that and we're using three or four of them.

[aaraney]

Yeah, ideally we could have a separate docker compose file that just creates the networks. Im not sure if that is possible though. I tried to do that with deployx and it did not work. I need to see if I can do the same in a non-docker swarm context just to compare behavior. That should be pretty easy for me to add into deployx.

[aaraney]

Moby, the open source go library for docker (once was the official api and still pretty much is before docker went more corporate), has some good documentation on the macvlan driver that we might look to. At a quick glance, it seems like macvlan would let you integrate with the underlying network and is likely much faster provided you have a fast network switch and appropriate network topology than the overlay driver.

[christophertubbs]

Thanks for the link - wasn't easy to find.