How to install docker-compose v2 on Linux

[rfay]

Description

I'm trying to allow DDEV to use docker-compose v2 and don't have the resources for a whole set of tests on every macOS variant... but I thought it would be nice to run all the tests using docker-compose v2 on Linux (where it hasn't been introduced yet to my knowledge?)

I followed the linux instructions in https://github.com/docker/compose-cli/blob/main/INSTALL.md but that doesn't do anything I can detect (curl -L https://raw.githubusercontent.com/docker/compose-cli/main/scripts/install/install_linux.sh | sh) (and doesn't enable the docker compose command either).

How can I use docker-compose v2 on Linux?

[ibnesayeed]

As per https://docs.docker.com/compose/cli-command/#install-on-linux

$ mkdir -p ~/.docker/cli-plugins/
$ curl -SL https://github.com/docker/compose-cli/releases/download/v2.0.0-rc.1/docker-compose-linux-amd64 -o ~/.docker/cli-plugins/docker-compose
$ chmod +x ~/.docker/cli-plugins/docker-compose

Note: The version number may change, obviously!

[rfay]

Hmm, after taking those actions on Ubuntu 20.04 I see:

docker-compose --version
docker-compose version 1.28.5, build c4eb3a1f

I do see that docker compose then becomes a thing, but docker-compose is unaffected AFAICT.

Also, would be good to maintain the INSTALL.md

[ibnesayeed]

... but docker-compose is unaffected AFAICT.

Yes, that's a separate tool and can co-exist (for now).

[rfay]

The question here is "How to install docker-compose v2 on Linux"

Hundreds or thousands of tools wrap docker-compose. An installation technique to make compose-cli be "docker-compose" is essential for those tools.

[thaJeztah]

Let me copy my comment from docker/docker-ce-packaging#553 (comment)

On linux, we could document creating an alias;

alias docker-compose="docker compose"

or to set it automatically for new sessions (for Bash)

echo 'alias docker-compose="docker compose"' >> ~/.bashrc

Or zsh:

echo 'alias docker-compose="docker compose"' >> ~/.zshrc

If you take the .bashrc or zshrc approach, that file needs to be "reloaded" to take effect for the current shell, by sourcing it;

source ~/.bashrc

source ~/.zshrc

[rfay]

So there are a number of problems with this approach.

1. It requires changing the full environment of an unknown user.
2. It assumes that we know that a user uses bash
3. It assumes a logged in user.

I really recommend a more robust wrapper that will let tools in the Linux world move forward. All it takes is a simple script or binary that replaces the original docker-compose.

[thaJeztah]

A separate "compatibility package" could be possible, but if docker-compose v1 is not deprecated, it should not come by default.

For Docker Desktop, the installation is somewhat more "opinionated", but for a Linux server install, things are more involved. Docker itself has not shipped docker-compose .deb or .rpm packages, but some Linux distributions have. While we (once stable) would like to see users migrate to v2, the v2 and v1 versions do not have to conflict (both can be used, and it's possible users have a reason to use the python version).

Adding the wrapper to the default package means we will have to add a Conflicts: or Deprecates: annotation to the .rpm or .deb packages, which means the package cannot be installed without uninstalling any existing docker-compose (which could be a distro package).

Things become more involved if the user has docker-compose installed from a binary or through pip, because the .deb or .rpm package managers will not be aware of that binary to be present.

So yes, possibly a "compatibility" package could be created (although it would be just a script doing the same as an alias). That package would be an optional install (user to opt-in), and that package could Conflict with distro packages (that we know of), but would probably not resolve the "manual" and "pip" installs. For those, "document how to migrate from v1 to v2" would (I think) be the only solution.

So even with the above, I don't think we can resolve the "It requires changing the full environment of an unknown user." situation. It will be difficult / impossible to make any assumptions about a user's configuration.

[rfay]

So even with the above, I don't think we can resolve the "It requires changing the full environment of an unknown user." situation. It will be difficult / impossible to make any assumptions about a user's configuration.

What I'm saying is you can't use shell workarounds to solve this problem. You have to use a binary wrapper or something. Shell workarounds are fine for individual users trying to solve particular problems, but they don't solve a migration problem for an important tool that is deployed in thousands of different contexts.

[thaJeztah]

I don't see why a shell script wouldn't work

$ cat /usr/bin/wait

#!/bin/sh
builtin wait "$@"

$ cat /usr/bin/gunzip | grep exec
exec gzip -d "$@"

[rfay]

Of course I'm talking about a docker-compose equivalent distributed in a standard way by the Docker organization.

[ndeloof]

Compose v2 is a CLI plugin, just download it from the release page under ~/.docker/cli-plugins/docker-compose. Install script is for the Compose CLI ("Cloud Integrations") which require a more complex setup.

see https://github.com/docker/compose-cli#compose-v2-aka-local-docker-compose for more details. We plan to move compose v2 into a separate repository ASAP to avoid confusion with this installation process

[ndeloof]

Also see https://docs.docker.com/compose/cli-command/#install-on-linux

[mat007]

@ndeloof yes but then you can’t use docker-compose anymore, you need to type docker compose .
I think what @rfay is asking for is a linux version of the switch/shim that Desktop ships.

[ndeloof]

@mat007 makes sense. Unfortunately this codebase has been designed to support canary testing, but we could extract the core redirection logic into a public component as part of compose.v2 codebase

[rfay]

The shortest-term view is I'd like to be able to run all ddev tests against compose v2 on Linux... because github actions is there and free. That's just to see if we can finally get compose v2 compatible and not have to embargo it.

But the long term view is... surely you don't want Linux users of docker-compose to stay on v1 because you've never provided a migration path? It seems like docker-compose on linux deserves first-class status. And yes, I know that supposedly everybody can just start using docker compose IF you package that, which you're not yet if I understand. But since every user of docker-compose has usages with docker-compose wired into it, that migration may never take place and all linux will stay on v1.

[ndeloof]

the point is that we can't bundle compose CLI plugin in docker Linux package until it get released as G.A.
As "installing" is just about donwloading adequate binary under $HOME/.docker/cli-plugins, I don't consider this a blocker for Linux users...

[thaJeztah]

the point is that we can't bundle compose CLI plugin in docker Linux package until it get released as G.A.

I think it'd be ok to publish a package (before GA); docker/docker-ce-packaging#553 implements it as a separate package (so users will be able to "opt-in" by installing the package).

For the transition docker-compose > docker compose, perhaps a transitional "compatibility" package (https://github.com/docker/compose-cli/issues/2012#issuecomment-906402612) that provides the alias/redirect could be an option to ease the migration.

[ndeloof]

We have published https://github.com/docker/compose-switch
This tiny tool handles docker-compose .. command being rewritten for docker compose.
I'll update documentation accordingly

[mozgbrasil]

Good morning my friends

I am running the process to use the updated version of docker-compose but the error is returned

$ curl -SL https://github.com/docker/compose/releases/download/v2.0.1/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose

$ docker-compose version
bash: /usr/bin/docker-compose: File or directory does not exist

Thank you in advance to everyone who can contribute.

[mat007]

@mozgbrasil try docker compose version with a space.

[mozgbrasil]

It worked as follows

Install latest docker-compose

sudo apt install jq

DOCKER_COMPOSE_VERSION=$(curl --silent https://api.github.com/repos/docker/compose/releases/latest | jq .name -r)
echo $DOCKER_COMPOSE_VERSION
sudo curl -L "https://github.com/docker/compose/releases/download/$DOCKER_COMPOSE_VERSION/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

sudo chmod +x /usr/local/bin/docker-compose

[zipzit]

The installation for Docker Compose was pretty confusing. I'm trying to follow along via https://docs.docker.com/compose/install/ and / or
https://docs.docker.com/compose/cli-command/#install-on-linux

I believe I should be using the command line instruction version, correct?

(insert about an hour of different trials here...) The answer is to follow the cli-commands version.

Note: between the different instruction links above, I did a test on $ echo $(uname -s)-$(uname -m) And that test returns Linux-x86_64 The instructions for cli version already had the flavor decoded, but note the L in Linux. In the $(uname -m) gets returned as a capital L, but the download from the releases version uses a small L. That could certainly mess folks up in the future.

Again, for anybody else who gets here, I was totally able to get a successful build using the Command Line Instructions from https://docs.docker.com/compose/cli-command/#install-on-linux

Docker version 20.10.11, build dea9396, Linux Ubuntu 20.04.3 LTS

[johnthagen]

Is there any rough timeline for when Compose v2 will be G.A? It would simplify deployments if Compose was bundled into the docker-ce-cli Ubuntu/CentOS packages, so we are trying to prepare for our transition to Compose v2.

[sbward]

Is there any reason Compose v2 cannot be published as a plugin? It would be great to be able to just write docker plugin install compose to get up and running before G.A.

[vjanssens]

Would it be possible to install docker compose v2 plugin globally instead of per user (as the docs describe)?

[ndeloof]

yes, you can install under any of the default system plugin dirs:
https://github.com/docker/cli/blob/a32cd16160f1b41c1c4ae7bee4dac929d1484e59/cli-plugins/manager/manager_unix.go
https://github.com/docker/cli/blob/a32cd16160f1b41c1c4ae7bee4dac929d1484e59/cli-plugins/manager/manager_windows.go

[tobiasmuehl]

I made and plan to maintain a gist for an unattended Docker Compose v2 install on Debian based distros under https://gist.github.com/tobiasmuehl/4de1f1c294f96441bec63c5bb0a97b21

[johnthagen]

3 days ago this was merged, which likely is of interest to this thread:

• [master] add docker-compose-plugin package (deb, rpm) docker-ce-packaging#553

[johnthagen]

I see that now both Ubuntu and CentOS have docker-compose-plugin-2.3.3 .deb's and .rpm's respectively.

• https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/amd64/
• https://download.docker.com/linux/centos/8/x86_64/stable/Packages/

Curious if Compose V2 is at or near GA now? Should/will the docs be updated to make these the official way to install Compose V2 on Linux?

[johnthagen]

Also, related:

• publish images for compose v2 to dockerhub #8871

[ndeloof]

docs will be updated soon then G.A can be announced

[rgl]

Can you please confirm why the latest versions (it only contains docker-compose-plugin_2.3.3~ubuntu-focal_amd64.deb) of compose are not in https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/amd64/?

If compose is not yet G.A., can the GitHub releases be marked as pre-release?

[ndeloof]

those are not pre-releases, just we didn't claimed compose v2 to be G.A. before we got it fully integrated in the distribution pipeline (which took much time than expected). Many users relied on github release for manual installation on Linux.

[rgl]

From a github release viewpoint, not making one as pre-release means it's a final release (which for me, it's the same as GA). IOW, my definition is different than yours, and I don't known what you mean by GA. Can you please point me to the documentation that explains its meaning in this project context?

[ndeloof]

I don't think we have a documented definition.
I agree with you, from my point of view compose is GA for months.
IIUC the missing requirement from a product perspective is that compose was not available by 'apt install docker' using our official packaging, + doc isn't fully up to date (which is a chicken/egg problem: we can't update docs for a not GA product so need some sync with docs team)

[johnthagen]

This thread mentions that GA should be within a month:

• [Docker Compose] V2 GA roadmap#256 (comment)

[chriswininger]

A big issue with an alias approach is that the interface of the api has changed a lot. For example docker-compose --version will return an error if docker-compose is just an alias to docker compose.

This is perhaps not a problem it's a new API we would expect breaking changes, but interestingly on mac this does not seem to be the case.

Mac user can run docker-compose --version and other v1 style docker-compose commands but they seem to translate to a correct v2 command.

In a company with a mix of mac and linux being used this is going to get old and confusing quite fast.

Is there a more powerful shim available for linux that does something similar to what is being done on mac through docker-desktop?

[ndeloof]

@chriswininger don't use an alias, just invoke cli plugin binary as docker-compose and it will automatically translate flags

[johnthagen]

Docker Compose V2 was announced as GA:

• https://www.docker.com/blog/announcing-compose-v2-general-availability/

[johnthagen]

I believe this issue is now addressed in the docs: https://docs.docker.com/compose/install/

[glours]

We're still have a glitch in the linux install documentation ATM, I'll close this issue when everything will be ok
It's already fixed, but not deployed in production for now

[glours]

Official doc is fixed now, we can close this issue

[RafalSkolasinski]

I just don't understand why it cannot be simply bundled into main docker binary and need to be installed separately...

[glours]

@RafalSkolasinski some people want only the Docker CLI, some others want only Compose, some want the both, that's why each component is shipped separately 🤷‍♂️
But you can use the convenience script to install everything at once.

[ilude]

since it appears that the maintainers want to make this as painful a process as possible I will just leave this here in a vain attempt that people will find it instead of the innumerable number of broken and unmaintainable ways to get docker compose to work consistently

sudo apt install jq
DOCKER_COMPOSE_VERSION=$(curl --silent https://api.github.com/repos/docker/compose/releases/latest | jq .name -r)
DOCKER_CLI_PLUGIN_PATH=/usr/local/lib/docker/cli-plugins
sudo mkdir -p $DOCKER_CLI_PLUGIN_PATH
sudo curl -L "https://github.com/docker/compose/releases/download/$DOCKER_COMPOSE_VERSION/docker-compose-$(uname -s)-$(uname -m)" -o $DOCKER_CLI_PLUGIN_PATH/docker-compose
sudo chmod +x $DOCKER_CLI_PLUGIN_PATH/docker-compose

If you want docker-compose to still work you can then link it with

sudo ln -s  $DOCKER_CLI_PLUGIN_PATH/docker-compose /usr/local/bin/docker-compose

[ilude]

@RafalSkolasinski some people want only the Docker CLI, some others want only Compose, some want the both, that's why each component is shipped separately 🤷‍♂️ But you can use the convenience script to install everything at once.

@glours if the convenience script can be used to install both docker and docker compose at once then someone should document that and how to use the script to do that, cause it does not install the docker compose plugin by default. And if you want people to use the convenience script you may want to consider not burying it at the bottom of the page three links into the install documentation. Put it at near the top of the https://docs.docker.com/engine/install/ page and then provide links to how to do the manual install per distro from there.

Imagine you are a new user and want to install docker on a new machine, your install page should provide links to the Mac/Windows installers and then instructions to run the convenience script on supported distros followed by links to the distro specific manual installs. not to mention the convenience script link on https://docs.docker.com/compose/install/ which links to the docker engine install page where there is NO MENTION OF THE CONVENIENCE SCRIPT AT ALL!

Please feel free to take the above script I provided and integrate it into the docs and your convenience script if needed. Provide it to the user as the default method to install and then as a secondary section show installing a specific version if needed. And when you do that provide a link to a list of version numbers that are available, preferably with changelogs so that users can determine what version they need should they require locking their install to a specific version.

The other issue your org has created is that the above way of providing the plugin (and its completely unclear if the convenience script does anything different cause its not documented) is that its outside the standard package management workflow for distros. In the past the manual steps, at least for ubuntu, would set things up in a fashion that meant you got docker updates when you ran your normal apt package upgrade commands. doing things in this half way fashion and not clearly stating how to update and maintain your software means that you end up with differing versions of docker and docker compose across machines if your not careful.

I'm sorry that some of my comments come across as aggressive, but the documentation for docker has been getting more and more frustrating over the past few years. There is zero mention in anywhere that I have found about the change from docker-compose to docker compose which should probably be something you want to put right up front so returning users can know and understand the change!

And while I'm on my soapbox, please for the love of all that's : call a major version rev of the docker compose file syntax and make the standard syntax have complete feature parity with the docker command line, and then add any other docker swarm stuff as options under the deploy section syntax. Removing cpu, memory and other command line features from version 3 and trying to play this game where version 2 and 3 exist for different reasons while you only push version 3 forward is not fooling anyone.