volumes_from doesn't respect container-based volume attachment in docker-compose v2.

[fubarhouse]

Description

I'm looking to attach a volume to a docker-compose service in the form of a container - which is not part of the docker-compose declaration.

Steps to reproduce the issue:

docker-compose.yaml

version: "3"

services:
  test1:
    container_name: test3
    image: alpine
    command: sleep 100000
    volumes:
      - ./testvolume:/testvolume

  test2:
    image: alpine
    volumes_from:
    - container:test3

1. docker-compose up -d
2. Observe the project coming up.

Describe the results you received:

no such service: test3

the volume was attempted to attach from the service test3 - not the container

Describe the results you expected:

Project should come up with the attached volume from the container named test3 (in this case it's the test1 service.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker compose version:

Docker Compose version v2.0.0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.3)
  compose: Docker Compose (Docker Inc., v2.0.0)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 12
  Running: 11
  Paused: 0
  Stopped: 1
 Images: 15
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.47-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 1.939GiB
 Name: docker-desktop
 ID: MJM6:MMLE:X3IT:JVUH:N45Q:54KE:EF26:FCUH:KW7L:WYJE:VTMF:5XBX
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 133
  Goroutines: 111
  System Time: 2021-11-03T10:42:33.974343305Z
  EventsListeners: 5
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:
n/a

[ndeloof]

Please give v2.1.1 a try, IIRC we already fixed this

[fubarhouse]

Just gave this a try on version 2.1.1 and I'm not seeing any difference...
I've tried the example docker-compose.yml above and another one with the same result...
Feel free to point out if I'm doing something wrong.

$ docker-compose up -d
[+] Running 3/3
 ⠿ test2 Pulled                                                                                                                                         2.8s
   ⠿ a0d0a0d46f8b Already exists                                                                                                                        0.0s
 ⠿ test1 Pulled                                                                                                                                         2.8s
[+] Running 1/0
 ⠿ Network test_default  Created                                                                                                                        0.0s
no such service: test3

$ docker-compose version
Docker Compose version 2.1.1

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  compose: Docker Compose (Docker Inc., 2.1.1)

Server:
 Containers: 4
  Running: 2
  Paused: 0
  Stopped: 2
 Images: 36
 Server Version: 20.10.10
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8686ededfc90076914c5238eb96c883ea093a8ba.m
 runc version: v1.0.2-0-g52b36a2d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.14.15-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.7GiB
 Name: karl
 ID: ONMH:J2DL:OF7K:CFRI:P5D2:YVFX:LK2O:QBJQ:4IGI:SMUL:XF2A:JF3O
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: fubarhouse
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  http://localhost:6000/
 Live Restore Enabled: false

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[fubarhouse]

Tested this again on 2.6.0, please indicate if the test is not doing what you'd expect.

$ docker-compose version
Docker Compose version 2.6.0

$ cat docker-compose.yml 
version: "3"

services:
  test1:
    container_name: test3
    image: alpine
    command: sleep 100000
    volumes:
      - ./testvolume:/testvolume

  test2:
    image: alpine
    volumes_from:
    - container:test3

$ docker-compose up -d
[+] Running 3/3
 ⠿ test1 Pulled                                                                 7.2s
   ⠿ 2408cc74d12b Pull complete                                                 1.6s
 ⠿ test2 Pulled                                                                 7.5s
[+] Running 1/0
 ⠿ Network karltest_default  Created                                            0.0s
no such service: test3

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., 2.6.0)

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 2
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1.m
 runc version: 
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.18.3-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.68GiB
 Name: karl
 ID: ZRHQ:OYNW:KCWI:EGGY:QQKV:N2UG:JNIF:KIZ7:EBPK:6NHD:LCES:5DOJ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[deepto98]

If this is still open, can I pick it up? @glours, can you point me to the files I need to look at?

[ajmoorexyz]

This is definitely still an issue.

[ajmoorexyz]

I'm looking to attach a volume to a docker-compose service in the form of a container - which is not part of the docker-compose declaration.

volumes_from only works for services that are part of the docker-compose declaration -- this is a breaking change.

[ajmoorexyz]

@deepto98 are you picking this up?

[deepto98]

@deepto98 are you picking this up?

Yes, I'll work on it this weekend (busy with some office work). Can you point me to the files I'd need to look at?

[deepto98]

@truckfondue I'm trying to work on this, but can't find any docs to set up docker-compose locally. There is a link here for setting up a Docker development environment and the contribution process, which redirects to this. But there are no steps to build locally, only a guide on contributing to docs.

[glours]

@deepto98 you can take a look at the BUILDING.md file.
Basically you just need to do run make compose-plugin which will create a ./bin directory with a docker-compose binary

[deepto98]

Okay, thanks @glours

[ajmoorexyz]

@glours as I mentioned above, this bug is preventing my team from upgrading to docker-compose v2.x. What's the process of getting this bug prioritized and or seeking official support from Docker? My company is paying for licenses to use Docker Desktop for Mac if that helps.

We run into this issue in CI where we use the open source version of docker and docker-compose

The scenario:

• An EC2 instance with a jenkins agent running in a container
• Inside the jenkins agent we call on docker-compose to spin up Cypress which in turn uses a volumes_from to mount the jenkins WORKSPACE into the compose project.

The jenkins agent is a container managed outside of docker-compose by way of a systemd service definition.

Lastly, we're still using the 2.x variant of the docker-compose file spec -- support for volumes_from is removed in v3.x of the spec.

[ajmoorexyz]

@deepto98 any luck making progress?

[ajmoorexyz]

@fubarhouse did you find a workaround?

[deepto98]

@truckfondue I was able to reproduce the issue following @fubarhouse's initial comment, but I need some guidance in understanding where exactly to look to fix it. I found the string no such service in 3 files - cmd/compose/ps.go, pkg/compose/compose.go and pkg/e2e/cascade_stop_test.go. I replaced them with dummy text and ran make compose-plugin. Then, on running ./bin/docker-compose up, I still get an error with the original string no such service. Need some help with this.

[fubarhouse]

@fubarhouse did you find a workaround?

Unfortunately not. Some users that use some software that uses this are still using Docker-compose v1 as a result.

[glours]

@fubarhouse @truckfondue @deepto98
Can you build a version of compose based on this branch and let me know if it fixes your issue

[fubarhouse]

@glours that is working nicely!

$ ./build/cmd -f docker-compose-fubarhouse.yml up
[+] Running 1/0
 ⠿ Container compose-test2-1  Created                                                                                                                             0.0s
Attaching to compose-test2-1, test3
compose-test2-1 exited with code 0

$ ./build/cmd version                                       
Docker Compose version dev

[sa-engineer]

Hi @glours,

So i've installed your patch of the docker compose binary and i've still getting an error.
no such service: container:amazeeio-ssh-agent
Docker Compose version v2.9.0-29-g12ad0fdd
Server: Docker Desktop 4.11.1 (84025) Engine: Version: 20.10.17

docker-compose.yml:
https://github.com/amazeeio/drupal-example-simple/blob/9.x/docker-compose.yml

The Container referenced does not created a named volume.
Container Referenced (container:amazeeio-ssh-agent) https://github.com/pygmystack/ssh-agent/blob/main/Dockerfile

[drugan]

@sa-engineer

I used the same patch for docker binary and had the same error;

What I found is the pygmy should put amazeeio_ssh-agent volume into /tmp folder but it puts this somewhere else.
So nobody sees this volume except the pygmy itself which reports like everything os okay in the cli output.