Extend from multiple services as mixins

[bfirsh]

It has been requested that extending services should work with multiple services, where each one may add additional bits of configuration.

Something along these lines:

services:
  base-has-term:
    environment:
      TERM: xterm-256color
  base-has-code:
    volumes:
     - .:/code
  web:
    extends:
     - base-has-term
     - base-has-code
    ...


This is related to #1988, because the idea of an "abstract" or "base" service will make this much neater (in fact, the example above assumes there is this sort of concept).

[dunk]

I requested this. Here is why.

Containers share configuration; usually when there is a one-to-many relationship going on. If you want to define a database and have several containers access it then all of those containers are going to need to know the DSN, along with some other settings that form a little collection. I think that using env_file declarations is a non-solution to this, as it only address environment variables and not other declaration types. I might want this little collection of settings to include a depends_on, or if I have put this database on a separate network I would want to include that network in my networks. Perhaps this database lives externally and requires an external_links. You get the point.

I think that a mixin-style approach would be a very neat solution. Through it we would be able to cleanly group all configuration relating to the access of a particular container, right there in the same docker-compose.yaml. And, if you think about it, we are half-way there already. We have the extends declaration. The demand for extends proves that people need to share configuration between containers - I'm just suggesting that we take it one logical step further and allow people to extend from many. Inheritance has majorly fallen out of favour recently in the OO space, and for good reason - it only allows you to choose once. As soon as you want to chop things up differently you have to hack around your existing structure. "prefer composition over inheritance", right? Should we not follow the same rubric here? I know that some will raise concerns over the need for simplicity, but I think that is wrong-headed. While I agree that at the Dockerfile level we don't want to have, for example, a bind-mount because then the image is inherently non-shareable, I think that in the case of docker-compose we are already talking about a collection of interconnected containers which make up a whole system - the parts of which are implicitly not extractable in the way that an isolated Dockerfile is. Remember, for every complex problem there is an answer that is clear, simple, and wrong. I think that mixins would walk the correct line between (marginal) additional complexity and beneficial utility.

[dunk]

@dnephin I saw that you downvoted the original issue. I have added some reasoning and would be interested in your opinion.

[dnephin]

Thanks for adding more information.

There is always a trade-off between being more explicit (adding the options to every service) and trying to reduce duplication (supporting ways to share options).

It almost sounds like what is desired here is something like "roles" (as it would be called in puppet), were you can assign a block of options because the node has some role (db client).

So far in my experience I've always opted to try and keep the options specified in the compose file to a minimum. Once it starts to grow, I'll move it to a file and use volumes_from, or an env_file. As you've mentioned the issue is that it's not always application config, sometimes it's actually container configuration that is shared.

I would agree with "prefer composition over inheritance", however I don't see mixins as composition, mixins are multiple-inheritance. If we wanted to use composition, maybe one way to do that would be to support an idea of a "role".

[dunk]

@dnephin Indeed, I also think this is related to config management. I imagine chef, salt, ansible, et al. have an equivalent to puppet's roles. I think that multiple inheritance, mixins, and this concept are in a similar space, especially because we are limiting ourselves to data representation (there's no polymorphism, method overloading, etc. when you're talking about a config file). However, I think that this is a bit different to how roles operate, because with roles you are composing modules together and I don't think that they interact all that much (?). Whereas here we want to be able to interleave the definitions.

Stripping it down, the concrete behaviour needed is to define the shared "atoms" once, usually as part of aggregate sets, and be able to create aggregates of those aggregate sets. I can think of three classes of atoms present in docker-compose.yml files:

• single-item atoms (dockerfile, image, depends_on, ...)
• set-type atoms (volumes, ports, networks, ...)
• dict-type atoms (environment, logging options, driver_opts, ...)

For each some form of aggregating behaviour must be defined:

• single-item: Your only options are to replace or do nothing, so you must define what order you will process the collections of atoms in, and whether you will adopt a replace-with-last or a never-replace policy. There may be other subtleties.
• set-type: This one is easy. You always add; duplicates are implicitly removed.
• dict-type: Here the behaviour will likely be similar to the single-item atoms case; service definitions are basically dicts themselves, so this is just the same problem in a different guise.

I think that there are two main use-cases. Firstly where the service "is-a" type of a common base service, and secondly where the service "has-a" common collection of settings (OO terms only used here for the purpose of analogy).

Applications (or some subset of services) which look like:

• is-a: differ little from a base service (which should probably be abstract, ala Extend a base service from the same file #1988)
• has-a: differ greatly from one another, probably have separate image/dockerfile declarations, but share some config subset (most commonly, in my experience, because of a common dependency on another service)

It seems that there have been a number of efforts to solve this problem that have each provided a part-solution. The extends definition, multiple files passed on the command line, env_file declarations, and variable substitution are all different ways to switch out parts of the docker-compose.yml file under different circumstances. I think that the approach outlined here could obviate much of the need for the first three, and potentially provide a better way to switch out settings that differ between dev/test/prod (which is I think the main utility of variable substitution).

I think that ultimately the most intuitive way to represent this is all in one place - in the docker-compose.yml file - rather than being woven through an assemblage of discordant files which are awkwardly, and confusingly, tangled up together.

I'm currently using the following code to achieve this, though I'd much _much_ rather use something native https://gist.github.com/dunk/8d258be750c7da048f52

[aanand]

The extends definition, multiple files passed on the command line, env_file declarations, and variable substitution are all different ways to switch out parts of the docker-compose.yml file under different circumstances. I think that the approach outlined here could obviate much of the need for the first three...

I can understand how mixin-like behaviour would replace/enhance extends (essentially by providing a superset of its functionality), but I don't see how it would do the job of multiple -f arguments or the env_file option. In both of those cases, the presence of a separate file seems key, as it allows the core configuration to remain a static, version-controlled part of the app, while the file containing environment-specific configuration is supplied at runtime.

Could you clarify how you see mixins working to serve the same use cases?

[dunk]

@aanand Sure thing. On closer examination I was wrong about multiple -f arguments, but I think I am right about env_file. By including a mixin declaration containing the env vars in the file passed to -f, we can apply these env vars to multiple services but still have them be dependent on which environment we are in. No external file needed.

There are two aspects at play - env-dependant composition and reusable grouping.

feature	env-dependant composition	reusable grouping
-f	yes	no
env_file	yes	yes
mixins	no	yes

Of course, -f and mixins are applicable much more generally than env_file. I think it feels analogous to factoring depends_on and networks out from links.

[dunk]

Is there any chance of this issue being on the roadmap at the moment?

[colinross]

As for shared configuration inside the docker-compose.yml -- I would look into yaml anchors ('variables' for sharing structure/values).

As for true mix-ins (what you are asking for)-- I believe (and please correct me if I'm wrong), but the role of docker-compose, specifically, is to handle already built images and putting them together in a cohesive set of inter-working containers. Mix-ins seem to me hinting towards the image creation process sharing functionality, not containers.

On the image building side-- while it may be a bit hackish-- I am currently working on a project that involves packaging up functionality (such as loading zsh, oh-my-zsh, vim, etc.) into a shell script that are executed (and therefore added as an image layer) given a specific --build-arg such as
docker build -f shell.Dockerfile --build-arg MIXINS='vim zsh'.

[dunk]

@colinross Anchors don't solve this problem - only a subset. You cannot, for instance, interleave items from dictionaries or lists - you can only reference a static predefined lump of stuff.

I think that the mix-ins idea is actually inevitable. There are many things that are going to apply to multiple containers - a database username / password combo, for instance, for every container which accesses that container. While this particular case can be solved by an env var file, other cases can't (particularly when groups of directives cluster - say a volumes_from+depends_on or particular environment+volumes). Without something isomorphic to mix-ins you must repeat these declarations in every container's definition in the yaml.

Indeed that is a hack. Wouldn't it be nicer if there was a way to have it built in?

[soufianetomase]

Hello,
Any news for this function ?

[dunk]

I would still love it too, fwiw.

[beephotography]

Almost 1 year passed - any news? I would also love that feature

[pi0]

Almost 1 year passed - any news? I would also love that feature

True love never dies :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.

[SeanMXD]

I would also like to vote on this issue being resolved. I am working with a memory constrained environment and would like to create a file called constraints.yml for each of the containers on this server which would be excluded from Git repositories and controlled by the server admin to restrict or release system resources based on real-time availability without occupying the single slot that's available for extending services as a result, because there are still some containers on this server which have extended other services for their own respective needs, and resolving these conflicts wouldn't be necessary if one service could extend from multiple files/services. Kind of like this (which I actually tried before researching led me here):

# snippet of docker-compose.yml
extends:
      - file: hostnames.yml
        service: lab
      - file: constraints.yml
        service: lab

I'm about to start figuring out a way around this in the meantime, but the workaround is gonna be hacky and the code will be spaghetti.

[Wingqvist]

I would find this feature very useful

My scenario:

I have a system where we have multiple microservices and we use docker compose for integration testing locally. We have 2 main compose files, 1 for normal test and 1 with debug capabilities, as well as 1 with common services.

There are 3 compose files:

1. common.yml
2. docker-compose.yml
3. docker-compose-debug.yml

where in common.yml there are common 3pp services used in both scenarios and a base definition with common config shared among our services.

in docker-compose.yml each service is defined and extends from the common service definiton

in debug.yml each service extends from it's docker-compose.yml counterpart.
It is here it would be nice if we could share all debug config from a single place, since it's a lot of services and editing the same thing in multiple places during the testing process is error prone.