Define services which are not started by default

[bfirsh]

Users quite often define maintenance scripts, test suites, debugging systems in their Compose files which they don't want to run when they do docker-compose up.

There should be some way of defining what services are started by default, but can be still be run manually by doing docker-compose up servicename or docker-compose run servicename ....

Possible solutions

1. Recommend users to use a separate Compose file
2. Add an option to services to make them not start by default
3. Add a top-level configuration option to define the default services
4. Add a concept of a thing like a service, but is just for one-off commands ("scripts", "tasks", etc...)

(Please suggest others if you have ideas.)

Data points:

• Proposal: Allow fig up (no other arguments) to run a subset of containers #697
• Add optional flag, so container is not starting with all "fig up", but only on manual demand #912
• Data Only Containers/No Run Containers #942
• Mark container not to start by default #1439
• Don't always start a service defined in yml when docker-compose up #1547
• Possibility to define in fig.yml if container should be created or not #542
• "test" service in https://github.com/heroku/logplex/blob/master/docker-compose.yml
• @cpuguy83: "I've got a little helper service in my compose yaml that injects a bunch of stuff into redis for hipache, which of course exits when it is done. Can't use compose up w/o the -d" this is fixed now that we don't exit from compose up until all services have exited.

[dnephin]

+1 for option 1

I think adding services that aren't actually part of a composition, but just happen to need to link/attach to it, is a bad design choice. There are a few other use cases that would be solved by allowing some form of includes syntax. Solving multiple problems with a single feature is always nice.

Some of these issues (the ones that deal with data-only containers, #942, the last comment from @cpuguy83) are actually already fixed by #1754, I don't think we need to consider them a problem any more (after 1.5).

[kojiromike]

We develop Magento extensions. To do that, we need a simple way to run a webstore on a LAMP stack. Compose makes that easy. But we also want to run phpunit, various static analysis tools, documentation builders, etc. In fact, the vast majority of our "services" are just commands, like docker-compose run --rm phplint. Ideally, an unqualified command like

docker-compose up -d

would only start long-running service (actual services as it were), and would not trigger other effects like phpunit, unnecessarily. (It's also important that it not trigger things out-of-order, like running Selenium tests before the web service is up.)

All we are really trying to do is take the pain of environment management away from developers, not run production services. I don't know if @dnephin 's comment is representative of the direction Compose is headed or not, but it seems ambiguous to me whether Compose is still planning to serve the niche Fig started out in. (Because, to be clear, I think option 1 does not support the ease-of-use that developers need to use this as Fig apparently intended.) I totally understand if Compose doesn't want to have multiple responsibilities, but I hope that someone can let us know, clearly, if those of us using this product for fast, isolated development environments should head in another direction.

[ryneeverett]

@kojiromike, what is it about (1) that is unsuitable for your use case? Is it just about the command semantics (docker-compose run --rm phplint vs. docker-compose --file phplint.yml up)?

[ppg]

Its that docker-compose up -d tries to 'start' phplint and docker-compose ps reports that that the phplint 'service' is down. In reality that's not a service container its a tool container, it should have no concept of up/down. It was my understanding that tool containers are embraced by docker (its how you would run something like redis-cli, clearly not a 'service') and while I use them more in development, I think they have a place for production too; like why have say awk installed on production machines or in containers, why not run it through a container with linking to get predictable behavior out of it.

[chrisguidry]

👍 I routinely find myself wanting to create a tests container alongside my other services to encapsulate running unit tests. My "workaround" is to set the command to "/bin/true" and run the container with the unit test command specified on the CLI. Being able to specify which containers should start at up time would be great!

(btw, nice work all around, folks)

cc @jameydeorio

[kojiromike]

@ryneeverett The semantics are one part of it. It's a problem of self-documenting and findability. Currently, we tell developers docker-compose run --rm foo bar. We invite them to create a shell function or alias, but we don't maintain a standard alias/rcfile for projects. (We don't want to standardize things outside of containers; we want to use Docker to standardize.) Adding a new file for some commands creates a hierarchy of importance: The docker-compose.yml file becomes the "default" file for important things, and the "other" file becomes less important.

The other thing is just maintaining relationships between services becomes more onerous. Just because we want something not to run by default doesn't mean it doesn't use a link or volume from a service that is long-running. extends doesn't actually provide all the facilities we would need to link services to "commands" (one-run services). Even if it did, if we have to use multiple yaml files, we will be forced to use extends where we otherwise wouldn't need to.

[ryneeverett]

extends doesn't actually provide all the facilities we would need to link services to "commands" (one-run services). Even if it did, if we have to use multiple yaml files, we will be forced to use extends where we otherwise wouldn't need to.

@kojiromike That's what I suspected. I wonder if you would be satisfied with improved extends support + some way to do subcommands (which would be functionally identical to extends) within a single docker-compose.yml. But perhaps the latter is the same as (4).

[tomfotherby]

2. Add an option to services to make them not start by default

I vote for Option 2. e.g. something like a start: false directive. The advantages are that we avoid the need for multiple compose.yml files or extra config files, you can just read one compose.yml to get the feel of the whole app stack.

[dnephin]

I think our proposed solution for #1987 (comment) will handle this. "admin" services can be defined into a separate config, and added with -f when an admin command needs to be run against a composition.

[sherter]

@dnephin The solution in #1987(comment) does handle "admin services", but it doesn't handle "data only containers", right? You would still have to use the command: /bin/true workaround.

[cpuguy83]

You shouldn't need data-only containers with compose as compose will handle swapping the volumes to re-created containers for you.

[sherter]

@cpuguy83 In the following example the data container is not really necessary, but if I want to change the volume, I only need to look at one designated place.

nginx:
  image: nginx:1.9
  volumes_from:
  - data

php:
  image: php:5.6-fpm
  volumes_from:
  - data

data:
  image: debian:jessie
  volumes:
  - ./:/app

However, I agree that this is a corner case and adding additional syntax is probably not worth it. I'm just saying that the proposed solution doesn't handle this case.

[dnephin]

True, some command is still required for data volumes, so you would have to use a minimal image to support that command.

I'm not completely up-to-date with the new volumes API that's coming out, but my hope is that we will be able to add a volumes: section to compose which will handle the data volume in a better way (instead of requiring a container for them).

[CWSpear]

This doesn't necessarily make it the right choice, but I think learning curve/ease of understanding should be considered.

I feel like (2) is the easiest to comprehend. I don't really have a way to confirm this, but my gut says most people who aren't intimately familiar with all of docker-compose's options that run into the problem we're trying to solve here say, "I wish there was some way to get that container to not start when I run docker-compose up," and they see start: false and bam, we're done and happy.

They don't say, "If only there was a way I could create a 2nd file with an awkward linking story to solve this..." (tho I realize that #1987 (comment) helps with the "awkward linking story," ya?).

(4) was kind of vague, but a dedicated avenue for the way for scripts and one-offs like this fits this "makes sense" bill.

[kalbasit]

Today I was looking for exactly (2) but ended up with my least favorite solution, a second YML file. My use case:

I have couple of container and all of them link to the same mongo container. I'd like to offer myself and the team the ability to load fixtures into the mongo database and I figured the easiest way to do that is to load the mongo container under a different name fixtures that itself link to mongo and then run mongorestore --host mongo --port 27017 --drop /dump. Since we don't want to load the fixtures at all time, it felt natural to have it with a start: false but ended up having a separate YML file for both containers fixtures and mongo.

Works well but start: false is much cleaner IMO. If I would have 10 or more permutations of this so-called fixtures container then yes start: false would be a bad idea and I'd go with (1).

[tonivdv]

Just had to make a compose file where some services should not be executed with the docker-compose up -d command. For me option (2) would be the best solution to my problem.

[mgriego]

I've run across this as well. Not sure which I think is the best solution, but it what it comes down to for me is that I need the ability to build images that are utilized as part of a composition. They're for ephemeral containers, but I need the image to be ready for use so I can run the container on demand.

[qcho]

Hi! same issue here, glad there is a nice grouping issue, great work!

In my case I have a python stack working with a couple of containers with services. Everything is working fine, but I've just found a case out of the stack. I'm managing static dependencies with bower, and want to run the bower command inside a container, it's just a script that we run once a while.

Would be great to run this script within the compose structure we have. I imagen something like:
docker-compose run node bower install

So I like the 2th of 4th option. Just need the service not to start, it's not needed :P

If there is some consensus I could try to send a pull-req for something like "restart"
start: always
maybe... dunno.

[jmreicha]

Another vote for option 2.

[gittycat]

I would prefer option 2 as well. The new Docker Data Containers can be specified in a docker-compose file but they don't need to be running to be used.

[ekkis]

@tberne I agree with that. docker-compose is simple and simple is a good thing. we should vote to keep it

[OnkelTem]

Heh, what an extensive discussion! Don't pay much attention to this comment, just my two cents.

At first I agreed with the majority to just add a sort of enabled flag.
Then I saw a proposition to branch definitions by groups and considered it as a good idea.
Then I read about Dobi, tried it, ensured that it can't be used in my project right now and went on reading.
Then I read arguments of @shin- and deemed them good enough, despite I also agreed with some counter-arguments.

My current solution is to create some docker-compose yamls and write some bash wrapping scripts to not run docker-compose by hand.

E.g. this is how I run containers which run a website (./docker/scripts/up script):

#!/usr/bin/env bash

def_opts="-d"

cd "$(dirname "$0")"/../

# A separate bash file with functions like: get_args, add_hosts...
. ./functions.sh

docker-compose -p ${PROJECT} up ${def_opts} $(get_args "options services" "$@") && \

# Adding host to /etc/hosts
add_hosts "$HOSTS" web && \
add_hosts "$HOSTS_MAIL" mail

[CpuID]

Big +1 to either Option 2 or 3.

[kitingChris]

+1 for Option 2 or 3.

Also the proposed "group" concept sounds interesting!

[gregunderwood]

+1 for option 2

[luckydonald]

There are so many +1 on option x, that I can't even find the original statement. Please use the reaction buttons under posts instead.

To add a new idea:

Can't we just set a default scale to 0?

web:
  image: busybox:latest
  command: echo 'scaled'
  scale: 0

[joeyhub]

You should just do option 2 (but maybe nostart instead or something) because people have been waiting years now, then wrapping docker-compose with their own configuration managers because the tool is too limited, which defeats the point as it's doing what docker compose should be doing. In two years someone could create their own more powerful docker-compose as well. That's a lot of time this issue has been open for.

In the long run there should be some consideration of true options grouping. Some are relevant only to build, some only to run and some are mutual (you really want a layer of nesting at least for run or just separate out and have a build section, problem solved). Currently though a lot of things are problematic in the configuration, I'm all for just remake it all in version 5 but getting it right that time (primarily by representing the real on a technical level domain rather than trying to make it map a usage based domain or overly simple, instead create a config layer for those goals on top of the right one, you can't do it in reverse). I think because of that state of affairs the quick fix is justified.

[pjxiao]

+1 for option 2 or 4

If there aren't any plans to add this feature currently, I want the way to run such services (or tasks) in a composition is written in docker-compose's docs.

[paulodiovani]

This is an old thread, and I agree that docker-compose shall not care about services that won't run from start.

But I also commonly add a test service, so here is what I did:

1. Override the entrypoint of the service.

2. In the new entrypoint, check if there is a tty available (docker-compose run uses --tty by default while up doesn't).

   # Check if there is a tty
   if [ ! -t 1 ] ; then
     echo "No tty available."
     exit 0
   fi

   This will always make the service exit when run without a tty (with docker-compose up, for example) while still runs as expected with docker-compose run

[z3ntu]

@paulodiovani That looks more like a hack than a solution.

[bijayrungta]

+1 for a feature request.
hacks like having commands that do nothing means even if the container doesn't run, it has to still be built, initialized etc..

[neumannjan]

I solved this in my project by providing a simple bash script that calls the appropriate docker-compose command based on provided arguments.

Now if I do ./docker-compose.sh -e testing up, the result is the same as if I did docker-compose -f docker-compose.yml -f docker-compose.testing.yml up.
If I just do ./docker-compose.sh up, I get, as you'd expect, regular docker-compose up.

Therefore, my -e ${environment} option (a short for --env) is basically an alias to -f docker-compose.yml -f docker.compose.${environment}.yml. This results in docker-compose.yml being the base/default compose file and files like docker-compose.testing.yml being its extensions.

This somewhat solves @acran's problem of the user not knowing which files to include (and in which order) to the docker-compose command by enforcing sane defaults. It is not a very robust solution as some may use more complex compose file combinations, but I think docker-compose should be able to do something similar. Most docker-compose files are likely to start with docker-compose. and end with .yml anyway, so why do I always have to type out such a long command (docker-compose -f docker-compose.yml -f docker-compose.testing.yml up) if I just need a single additional service for my testing environment?

[oppianmatt]

No need for envsubst, docker-compose config does the environment variables and then docker stack takes it from stdin:

docker-compose config | docker stack deploy --prune --compose-file -

Let's you use the docker-compose .env substitutions as well.

[frnco]

@kogli I think the problem here is to pick an option. All have Pros and Cons. As I said before:

Every solution will be better for some use cases, and worse (Or even useless) for others. If the something is implemented, the mantainers it forever, so I believe they're wise in taking their time with it.

@shin- I guess you're the one to ask: Does it even make sense to consider this? Considering the current position of the team, how hard it is to design it, and the many tools out there, wouldn't it be easier to simply tell people this won't happen? Most people if not everyone who commented here found a way to deal with it.

Closing this seems pretty good, plus you can focus on the stuff that matters, and no one wonders if they should make a pull request and how to meet all those needs at the same time. They can just go build something else, and maybe even publish it to help others.

Just for the record: My solution was not perfect, but way better than the 2 days it took me to install the dependencies and fix a shitload of stuff and make the project run (and it already used Docker). Just 3 commands, with Docker and Docker Compose as the only dependencies. Yes there was more code than I wanted. Yes there's a shitty order to run commands. And yes there's a tiny bit more to keep in mind. A Wiki page covered those. Preety good, if I do say so myself, which means Docker Compose did it's job, I just also needed somethig else.

Point is: If it's out of the scope, it's a waste of time to keep discussing it. Many people just stop here to ask about this and give their opinion, before trying to decide what to do. They will still find or build something else if this is closed, just faster. And you get one less thread to babysit.

Edit: Just thought of a good way to put it: If there's no clear criteria to accept PR and there is no one in the core team either involved with it or planning to take a look at it in the foreseeable future, it's not an "issue", just an idea.

I see no reason to clutter the issue list, I believe many ideas here are good, but I think they belong somewhere else, for good or for worse. It's almost 3 years, and on the off chance that one day the situation changes, you can always reopen this, or open a new issue about it.

[rubycut]

This feature takes like one day to implement because it's really simple. However, after 2 years of discussions, still nothing. Sad.

[piit79]

@rubycut Yeah, that's what happens when so called "higher principles" take over from usability and user-friendliness. I completely understand the devs don't want to clog the product with illogical features. However, this particular highly requested feature actually seem to make sense. It's sad when the developers think they know better than their users. (Again, they usually do. But not always. Not this time.)

[9nix00]

@rubycut Talk is cheap, show me the code. If you can,send your PR, if you can't, just report it and wait .

[rubycut]

@wangwenpei Are you kidding me? First of all, you are not maintainer of this project. If maintainers would accept this idea, and put "help wanted" tag, I am sure somebody could contribute pull request. But they said that they are not interested. So, why bother will pull request if it will not be accepted.

[alecmev]

Just FYI, "the code" is already there: #3047

[acran]

@wangwenpei well, #3047, there's the pull request...
waiting to get at least some feedback on it for over two years now. It was closed recently

for the reasons highlighted in #1896 (comment)

which quite angers me to be honest because the arguments given in #1896 (comment) do not relate in any way with the pull request:

1. What is a service? By design, a service is a clearly defined, long-running process that interacts with other services to form an application. That is the basic assumption that Compose operates under and it guides our design for most, if not all features we implement.
2. By that definition, a "service that is not started by default" is not really a service [1] - it's more of a pre-defined task, running in a container and acting on the application, rather than being part of it.

I don't see the assertion By that definition, a "service that is not started by default" is not really a service to be true. A service that is not started by default is just a service that is not started by default, nothing more, nothing less at this point. Being started by default or not doesn't say anything about the service being long-running or not, being part of the application or not, it only says that is not to be started by default.

And that's also exactly what the change in the pull request does: add the ability to mark a service to be started by default or not.
There is no need to introduce a new concept such as tasks to do this. In fact all the other arguments in #1896 (comment) follow from the false premise of 2. and are thus not applicable to the pull request which does not try to do anything else.

Just for completeness:

[1] Some people do mention wanting to have "optional services", but that's not the most common use case mentioned in this thread. Most people want the "task" feature, as far as I can tell.

Yes, there were examples of one-off tasks, but there also were examples for real services according to your definition, i.e. legitimate use cases for this feature.
Being able to use this to also run one-off tasks I can not really accept as a valid argument against this. It is already possible to define a service which is not long-running, not exposing any ports and is not an essential part of the application in a totally valid docker-compose.yml.

In fact this is also one of the suggested workarounds! Just split the tasks into a separate docker-compose.yml.
The other workarounds mentioned are to use your own home-made (platform-dependent) scripts, which totally contradicts the self-stated purpose of docker-compose, see my comment above, or to simply switch to another tool, i.e. throw away your perfectly well working toolchain just to get this one tiny feature in convenience which could easily be implemented in your current toolchain (docker-compose).

The only real argument brought up against the most basic implementation of option 2 - i.e. having some way to tell docker-compose to not start a particular service by default, and nothing more, no new concepts or anything - is that this could be_somehow_ be misused by some users.
But this argument is quite weak compared to the demand for it and the legitimate use cases brought up here...

@shin-

but also because we [3] simply don't have the time or resources to implement or maintain what this feature would need to become to be truly useful to people.
[3] And at this point I mean "I", as I'm the only maintainer actively working on docker-compose at this time.

On the one hand highlighting that there is only one active maintainer while on the other hand just ignoring pull requests and therefore wasting any opportunity to maybe get other developers involed in the project isn't something I could comprehend.
The code changes provided in #3047 already do implement a feature which would be truly useful to [many] people - maybe not all the people with all of their use cases, but most of the people here.

All the additional requirements - having to introduce completely new concepts, new dependency management, up to a cmake-like complexity - simply follow from a false premise!

[Songworks]

I came upon this issue just a couple days ago because I was looking for a feature like this.

Since it's unfortunately not implemented, and from what I could gather, the officially supported alternative is to use multiple yaml files.

Well, I find the idea of using multiple yaml files for services not run by default a poor alternative in comparison to using one yaml file for all services containing one additional setting for each service not run by default.

So, here's my +1.