Proposal: Docker Compose

[aanand]

This proposal, deprecating #9459, is for a command-line tool for orchestration of multi-container apps, suitable for development environments, automated CI setups and staging and production deployments, and working fluidly with Docker Machine and Docker Swarm.

Based in large part on feedback to #9459 from the Docker community, we propose the following:

• Compose is a standalone command-line tool, rather than built into the Docker client binary.
• The initial release of Compose is directly based on the Fig codebase, rather than written from scratch, and focused around cleanup and any backwards-incompatible changes from Fig.
• A roadmap of features for Compose and (if deemed appropriate) the Docker engine, so as to achieve production-suitability and seamless integration with Swarm, is planned.

Separate tool

The arguments in favour of building Compose directly into the Docker client are based on usability issues we’ve observed with Fig:

• Newcomers to Docker find Fig more immediately easy to understand and relevant to their needs, so it makes sense for them to have that benefit out-of-the-box.
• People are often confused about the distinction between Fig and Docker, so it would be beneficial from a UX perspective if there were no distinction.
• Fig is “one more thing to install” that prevents users from getting up and running.
• Problems arise when versions of Fig and Docker are incompatible.

However, there are also usability arguments against building it in:

• The various Docker commands become context-sensitive (i.e. they behave differently based on the contents of the current directory) and overloaded with meaning (arguments can either designate literal container names or names in the configuration file).
• Docker users who aren’t interested in Compose’s features will nonetheless have to live with a tool that’s both physically and conceptually larger.

Furthermore, technical and social arguments strongly favour separation: the responsibilities of Compose and those of the current Docker client don’t overlap, and it’s better to avoid conflating non-coupled concerns in one codebase, because:

• It’ll make the codebase even larger.
• It’ll force both projects to share a release cycle, making it harder to experiment on Compose and to make changes to either one.
• It’ll put extra burden on the core maintainers.
• It’ll be harder to find issues and PRs if they all live in the same GitHub repository.

Furthermore, a separate tool will help guide the definition and maintenance of the Docker API(s), keeping the door open for a healthy ecosystem of alternatives.

Finally, it’s still possible to go carefully from a separate tool to an integrated one in the future; extracting a separate tool back out again, however, is much harder.

Based on Fig

Fig in its current state is a great orchestration tool for development/CI, and a usable one for simple production deployments. Its codebase is mature and well-understood, and doesn’t carry a lot of technical debt (though there are improvements which can and should be made!)

It would make an excellent starting point - and, given Compose is a new tool, we’d be free to make breaking changes from Fig’s logic for the first release.

Features roadmap

As well as a number of smaller features that have been requested for Fig (which will still be triaged and implemented where appropriate), several high-level features are needed to get Compose to a state where it’s suitable for multi-host production deploys. Compose’s priorities after the initial release will be identifying, designing and implementing these in collaboration with the community and users. Discussion for many of these is already ongoing on the Fig GitHub project.

[jokeyrhyme]

As mentioned in #9459:

• I'd like the option of using a less ambiguous definition format: TOML or JSON are less "magical" than YAML
• it'd be terrific if every docker CLI option was able to be handled in the definition: there are loads of things I end up needing external scripts for with fig, where crane covers everything I can think of

[pkieltyka]

I like the idea of a Stackfile in TOML. Anyone else?

[dnephin]

I personally really prefer YAML, but I don't see a need to restrict it to just one format. All of these formats should be able to map to a common representation internally.

[aanand]

I like the reduced ambiguity/size of TOML. My main complaint is that nesting is more verbose. Translating the group.yml example from #9459:

name = "rails_example"

[containers]

  [containers.db]
  image = "postgres:latest"

  [containers.web]
  build = "."
  command = "bundle exec rackup -p 3000"
  volumes = [".:/myapp"]
  ports = ["3000:3000"]
  links = ["db"]

I'm not a huge fan of how the containers key is repeated for every sub-dictionary. If another level of nesting were introduced for some reason, the whole file would get yet more verbose.

This might not end up being an issue, but it warrants consideration.

[aanand]

@jokeyrhyme: yes, Compose should aim to cover every Docker CLI option.

[pkieltyka]

@aanand you can make this work too:

name = "rails_example"

[containers.db]
image = "postgres:latest"

[containers.web]
build = "."
command = "bundle exec rackup -p 3000"
volumes = [".:/myapp"]
ports = ["3000:3000"]
links = ["db"]

I do something similar in a project conf.

I think YAML is also fine though, I just would really prefer not to see the .yml extension on the filename.. even .conf would make me happier. Just my 2cents.

[tianon]

One issue I've seen with YAML in the wild (specifically in fig.yml) is that things like an environment: value of true have to be quoted or they become True (thanks to true -> True -> True.to_s).

[ewindisch]

Not suggesting we solve every use case, but one major limitation of most of the proposals I've seen is the lack of piping. I want a compose framework which pipes containers together. Currently, I've been using shell scripts wrapping the client...

[softprops]

One + for json is that it's extremely easy to consume due to ubiquity in json parsers for every flavor of programming language. It's designed for both humans and machines to read. Yaml is easy for humans to read ( sometimes... if you can eye up your whitespace ) but deceptively hard for machines. Compare it's spec with json's if you disagree :).

I would prefer a format that makes it easy to build new tooling on top of rather than encouraging a single project to have different descriptions of the same information for different tools.

[xiaods]

just like boot2docker, docker is already the owner of fig, why not enhance the fig to implement the docker compose feature?

[thaJeztah]

+1 on supporting multiple formats. Depending on your case, JSON, TOML or YAML could be prefered. Since all formats should be easy to map to the same "internal" format, I don't see a reason to not support them.

Should there be a "default" format? (Thinking of what format to use for examples in the documentation)

[xiaods]

share the fig multi host patch
Enable services to use different Docker Hosts
docker/compose#607

[mkawalec]

If we are requesting features, docker exec is immensly important to me for debugging (sometimes you just want to tinker with a raw source file), I would love to see it in an exposed place.

[thaJeztah]

share the fig multi host patch
Enable services to use different Docker Hosts

I don't think that is nescessary to implement in Compose; Compose should be able to talk to Swarm and have Swarm take care of which host a service will be deployed to.

The only thing that compose has to take care of, is passing the constraints (ie, ports, memory etc) to Swarm to act on.