Since docker 1.3.0 / fig 1.0.0 volumes are not mounted when fig up'ing existing containers

[morvans]

Since docker 1.3.0 / fig 1.0.0 (I upgraded both at the same time), I cannot fig up my projects without fig rm'ing before.

If I do fig up on already existing (stopper) containers, volumes specified in fig.yml aren't remounted (default docker mounts instead, so empty dirs).

fig rm && fig up recreates everything correctly but for locally built containers this is pretty time consuming.

[mindo]

This also happens with docker 1.3.1 and fig 1.0.1.
Even more weird is that if a container is ran using fig:

fig run mycontainer bash

The volumes are mounted just fine.

[dnephin]

Are you using volumes or volumes_from? A sample fig.yml may help in debugging.

[mindo]

My fig.yml:

haproxy:
    build: haproxy
    ports:
        - "80:80"
        - "443:443"
    links:
        - ngx

ngx:
    build: ngx-php
    volumes:
        - var/logs/:/var/log
        - ngx-php/etc/:/etc/nginx
        - myapp/:/var/www
    links:
        - percona
        - memcached

memcached:
    build: memcached

percona:
    build: percona
    ports: 
        - "3306:3306"
    volumes:
        - var/mysql/:/var/lib/mysql

All folders defined as volumes are on the same level as fig.yml.

[dnephin]

This seems correct to me. Some things you could try to debug it:

• docker inspect the containers when they're up to see what it's doing
• docker will silently ignore a volume if the path doesn't exist, it's possible it's not doing these relative paths correctly. Maybe try adding a ./ prefix
• paths in the fig.yml are relative to the current working directory, so if you do something like fig -f somepath/fig.yml it might not work (because the paths will be wrong

[nireak]

Perhaps it's a silly question.... but are you using a Virtual Machine to run docker and fig ? If so, do the paths referenced in your "volumes" directives exists inside the VM?

[mindo]

cd /to/folder/with/fig.yml

• fig up

    "Volumes": {
        "/etc/nginx": "/var/lib/docker/vfs/dir/6b04e7ee535de76bfd359ad1c08fbe35db3e435c2301c464a7390ba67cfefe63",
        "/var/log": "/var/lib/docker/vfs/dir/17b86c34458c046e15709731c3e3c760e2c766ce113c5b2fd0f7de9aafdcc17b",
        "/var/www": "/var/lib/docker/vfs/dir/11a480315b57b48c6b7f5abaada5741a5656e5febdd3365e40e62f04e94f3be2"
    },

• fig run ngx bash

    "Volumes": {
        "/etc/nginx": "/mnt/HG/docker/ngx-php/etc",
        "/var/log": "/mnt/HG/docker/var/logs",
        "/var/www": "/mnt/HG/docker/myapp"
    },

I am running this directly on my notebook (ubuntu 14.10, no virtual machine).
I also tried:

ngx:
    build: ngx-php
    volumes:
        - ./var/logs/:/var/log
        - ./ngx-php/etc/:/etc/nginx
        - ./myapp/:/var/www

and

ngx:
    build: ngx-php
    volumes:
        - /mnt/HG/docker/var/logs/:/var/log
        - /mnt/HG/docker/ngx-php/etc/:/etc/nginx
        - /mnt/HG/docker/myapp/:/var/www

and neither worked.

[dnephin]

@mindo this looks like a similar issue to #613. fig up will copy volumes from the previous container when it recreates them. I think if you do: fig stop && fig rm --force -v ngx it should remove those volumes and the next time you create it, it should mount the correct host volumes.

[mindo]

@dnephin yes, doing

fig stop && fig rm && fig build

solves it but only until the next reboot/service docker restart.

Test run:

fig rm
fig build
fig up

Volumes are mounted just fine.

fig stop
fig up

Volumes still work.

fig stop
service docker stop
service docker start
fig up

Volumes fail to mount! But running:

fig run ngx

The Volumes work without problems.

[schickling]

Same here. Any progress on this?

[morvans]

FWIW, the issue seems fixed for me with docker 1.3.2 / fig 1.0.1

[schickling]

I'm using docker 1.3.2 / fig 1.0.1. Still doesn't work for me.

[morvans]

@schickling : yeah, spoke too soon, it still doesn't work for me either eventually.

[bercab]

Hello, I have the same issue. Can't bind-mount volumes with fig run nor fig up.

• fig versions tested: 0.3.2, 1.0.0, 1.0.1 and master
• docker version: 1.4.0
• docker-py versions tested: 0.5.3, 0.6.0 (with fig master)

Running with fig --verbose, I see volume is correctly parsed when passing to client.start():

 binds={u'/test': {u'bind': u'/test', u'ro': False}}

But docker inspect shows volume is not bind-mounted:

    "Volumes": {
        "/test": "/var/lib/docker/vfs/dir/fa2cbf2c4b3a560b9e99dd07b68968b99260a717e4ab4f6371dc88e7aa805414"
    },

I think is not a problem of fig, but a problem of docker-py with newer versions of docker.
According to docs, binds dict seems correct: http://docker-py.readthedocs.org/en/latest/volumes/
Maybe docker-py is breaking its API?

Looking at docs link above, seems that client.create_volume() spects a list of volumes (ej: ['test', ]), but fig is passing a dict:

 volumes={u'/test': {}}

Also seems other projects are running the same issue:
saltstack/salt#14089

[shusugmt]

I've faced the exact same issue using

• Ubuntu 14.04
• docker 1.4.0, installed from ppa (lxc-docker-1.4.0)
• fig 1.0.1

Reverting docker version back to lxc-docker-1.3.3 solved the problem for now.

[dnephin]

@s2ugimot for docker 1.4.0 see #723

[anthonylouisburns]

has anyone fixed this

[dnephin]

I believe #711 will resolve the original issue

[anthonylouisburns]

Awesome - I'm new to github - how do I know what version #711 is in or whether it is in at all

[ihrwein]

I have the same problem:

• Ubuntu 14.04
• docker 1.4.0, installed from ppa (lxc-docker-1.4.0)
• fig 1.0.1

[loa]

Same issue

$ fig --version
fig 1.0.1
$ docker version
Client version: 1.3.3
Client API version: 1.15
Go version (client): go1.4
Git commit (client): d344625
OS/Arch (client): darwin/amd64
Server version: 1.4.0
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 4595d4f

[jaycyb]

+1 also reported here #443

docker 1.3.1 + fig 1.0.1 works fine
docker 1.4.0 + fig 1.0.1 does not

[gquemener]

From my experience, docker 1.4.1 + fig 1.0.1 does not work also: #443 (comment)

[cpuguy83]

@jaycyb @loa Bug is in Docker 1.4.0, please use Docker 1.4.1 if you want to use 1.4.x.

[gquemener]

This does not solve the issue for me :-/

[dreamcat4]

Same experience as @gquemener here: docker 1.4.1 + fig 1.0.1. adding host volumes does not work unless i do a 'fig rm' first. Not so-sure what happens on restarts. #723

[dmp1ce]

I also experienced the same thing. For one of my containers the 'fig up' would not mount host volumes. However it would mount if I used 'fig run --rm mycontainer bash'. Running 'fig rm mycontainer' fixed it.

[vh]

Same issue: docker 1.4.1 + docker-compose 1.1.0-rc1. Fixed it to run 'docker rm container' before 'docker-compose up'

[jgillich]

+1, I just wasted a lot of time on this.

Docker version 1.3.2, build 39fa2fa/1.3.2
fig 1.0.1
CentOS Linux release 7.0.1406 (Core)

[nomadster]

+1 This "feature" actually makes docker+fig a tool to make a deploy in just two days :)

Docker version 1.3.2, build 39fa2fa/1.3.2
fig 1.0.1
CentOS Linux release 7.0.1406 (Core)

[ddliu]

+1

fig 1.0.1
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

[aanand]

Can anyone produce a minimal failing case? I'm currently unable to reproduce. Here's my test case:

$ cat docker-compose.yml
test:
  image: busybox
  command: cat /data/hello.txt
  volumes:
    - ./data:/data

$ docker-compose --version
docker-compose --version

$ docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): darwin/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef

$ docker-compose rm --force
$ mkdir -p data
$ echo "hello world" > data/hello.txt

$ docker-compose up
Creating 622_test_1...
Attaching to 622_test_1
test_1 | hello world
622_test_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)

$ docker inspect -f '{{.Volumes}}' 622_test_1
map[/data:/Users/aanand/work/docker/compose-bugs/622/data]

$ docker-compose up
Recreating 622_test_1...
Attaching to 622_test_1
test_1 | hello world
622_test_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)

$ docker inspect -f '{{.Volumes}}' 622_test_1
map[/data:/Users/aanand/work/docker/compose-bugs/622/data]

[cpuguy83]

@aanand 1.5 has some fixes in it that would resolve this, and maybe one more in 1.5.1 for some fig users.

[dnephin]

I suspect (based on the early comments on this thread, and my personal experience helping people new to fig) that this is often an issue with copying old volumes during fig up instead of creating things from scratch.

If you have an existing container with data volumes, and you change the content of those volumes, when you fig up, you'll still get the old data, which makes it seem like the volume is not mounted. The workaround of fig rm seems to have worked for a few people.

I think #858 helps with this.

There is still a case where the volume path hasn't changed but the contents or structure of the data volume have changed. That one is harder to deal with and maybe just needs to be called out better in the docs.

[nomadster]

@dnephin it's not my case.
fig stop && fig up

doesn't mount the volume. If i go to the folder where the volume should be mounted I don't have any data at all.

[dnephin]

@nomadster can you docker inspect <container name> and include a link to a pastebin with the output?

fig stop && fig up is actually what I described, what if you fig stop; fig rm --force; fig up (this will remove data volume containers).

[nomadster]

Cannot reproduce right now, I've switched to docker 1.1.2 and fig 0.5.2 to be able to work :)

[morvans]

@aanand, I've got a scenario with fig where volumes aren't re-mounted correctly upon fig up when docker daemon has been restarted between stop/kill and up. I can provide you (privately preferably) the full log with docker inspect outputs. It forces me to fig rm my containers every morning after booting my workstation (event if containers where "gracefully" fig-stopped before the reboot or docker daemon restart)

[aanand]

@morvans Does the problem only surface when the docker daemon has been restarted? If so, that's a separate issue, and it's worth investigating whether that behaviour can be reproduced using Docker without Fig.

Can you produce a failing case with private information stripped out?

[morvans]

Ok I'll try to get back to you with a stripped down fig.yml and/or a test case involving docker alone.

[dmitry-saritasa]

I can't mount relative folder through fig on centos7:

[root@localhost using-fig]# uname -a
Linux localhost.localdomain 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost fig-master]# fig --version
fig 1.0.1

[root@localhost fig-master]# docker version
Client version: 1.3.2
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 39fa2fa/1.3.2
OS/Arch (client): linux/amd64
Server version: 1.3.2
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 39fa2fa/1.3.2
[root@localhost fig-master]#

fig.yml:

/home/apps/using-fig/fig.yml
es:
    image: library/elasticsearch:1.4.3
    volumes:
        - elasticsearch:/data
    ports:
        - "9200:9200"
        - "9300:9300"

[root@localhost using-fig]# pwd
/home/apps/using-fig

[root@localhost using-fig]# ls -l
total 4
drwxr-xr-x. 3 root root 41 Feb 13 17:30 elasticsearch
-rwxr-xr-x. 1 root root 141 Feb 13 17:16 fig.yml
drwxr-xr-x. 2 root root 59 Feb 13 15:32 web

[root@localhost using-fig]# ls -l elasticsearch/
total 4
drwxrwxrwx. 2 root root 6 Feb 13 16:59 data
-rwxr-xr-x. 1 root root 49 Feb 13 16:23 elasticsearch.yml

so when I run fig up -d

[root@localhost using-fig]# fig up -d
Recreating usingfig_es_1...

[root@localhost using-fig]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e7f8547a4ad1 library/elasticsearch:1.4.3 "elasticsearch" 5 seconds ago Up 4 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp usingfig_es_1

[root@localhost using-fig]# docker exec -ti usingfig_es_1 bash
root@e7f8547a4ad1:/# ls -l /data/
ls: cannot open directory /data/: Permission denied

however if I change fig to absolute path

fig.yml:

/home/apps/using-fig/fig.yml
es:
    image: library/elasticsearch:1.4.3
    volumes:
        - /elasticsearch/data:/data
    ports:
        - "9200:9200"
        - "9300:9300"

everything works just fine

[nomadster]

@dnephin I've managed to find some time to check for the bug:

$ docker version
Client version: 1.3.2
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 39fa2fa/1.3.2
OS/Arch (client): linux/amd64
Server version: 1.3.2
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 39fa2fa/1.3.2
----
$ fig --version
fig 1.0.1

I've deleted every single image I had and built them from scratch then run fig up (some container use links, I can provide the fig.yml privately if needed).
Everything is ok:

$ nsenter -m -u -p -n -i -t $PID
container$ ls /var/lib/mysql
my_db_from_volume_1
...
my_db_from_volume_N

Now I fig stop all the containers and fig up them again (not running fig rm --force)
And the bugs doesn't happen: everything just works fine.
All the data is where I wanted it to be.

Just in case this will happen again later on I've pasted the output of the command you asked here

[dnephin]

Ok, this sounds a lot like re-using old volumes, and might be fixed by #858

[morvans]

@nomadster, if you restart the docker daemon between fig stop and fig up, are the volumes re-mounted correctly ?

[nomadster]

@morvans

fig stop
systemctl stop docker #Stopped Docker Application Container Engine
systemctl start docker #Started Docker Application Container Engine.
fig up -d container1 #(ups other container via links, same command as above)

Some of the container doesn't start correctly. I suppose because the volumes are not mounted correctly.

[morvans]

FWIW, I updated to docker 1.5.0 and the issue seems to be gone.
I'm able to fig up stopped containers without fig rm them beforehand, and volumes are re-mounted correctly.

[UnknownHero]

yes, 1.4.0 not work , but 1.5.0 work fine

[Vrakfall]

@morvans Yeah, I think the problem was mainly coming from docker itself and the last big update was meant to solve big mounting issues.

[aanand]

I'm closing this issue as it seems to be a confusion of different reports - if anyone can produce a minimal failing case, please open a new one.

[CWSpear]

I'm seeing some behavior that seems related and I have a reduced test case here: #2776

[knvpk]

Hi, im experienceing with same problem, after restarting my server the containers are starting but not mounting the volumes, the changes i have done to my server is updated to Ubuntu server 15.10 and docker-compose version is 1.6.0, but when i goto my docker-compose folder and start them then volumes are mounting correctly.

[dnephin]

You might be hitting #2912