Local directories periodically fail to be mounted.

[dancrumb]

I'm seeing an issue whereby a local directory that I mount to a container is, intermittently, not actually mounted.

Dockerfile

FROM busybox
MAINTAINER Dan Rumney <email>

ADD loadsnapshot.sh /loadsnapshot.sh
RUN ["chmod", "u+x", "/loadsnapshot.sh"]

VOLUME ["/snapshot"]

ENTRYPOINT ["/loadsnapshot.sh"]

loadsnapshot.sh

#!/bin/sh

if [ "$( ls -A /snapshot)" ]; then
  echo "Loading snapshot..."
  # Do stuff
else
  echo "No snapshot to load"
fi

fig.yml

pdsvol:
 image: busybox
 volumes:
 - "/opt/alfresco/alf_data"
 - "/data"
 - "/mysqlbackup"
 - "/ldapbackup"
loader:
 image: "docker.myregistry.com/snapshot.loader:3.5.0"
 volumes_from: 
 - pdsvol
 volumes:
 - "/opt/snapshots/pepper-demo-site/:/snapshot/"

The goal here (which may be obvious) is to start up a data container (pdsvol) and then populate it with some data that's running on my machine. pdsvol is then shared by a bunch of other containers.

The way I run this is to call

fig up pdsvol

and then

fig run --rm loader

What I expect to see is

builder@beast:/opt/docker-vm$ fig run --rm loader
Loading snapshot...
... stuff ...
Removing dockervm_loader_run_1...

and, sometimes I do. However, sometimes I see:

builder@beast:/opt/docker-vm$ fig run --rm loader
No snapshot to load
Removing dockervm_loader_run_1...

After a lot of debug effort, I was able to run my command without the --rm and then inspect the remaining containers.

When I diffed the results I saw::

Successful

"Volumes": {
        "/data": "/var/lib/docker/vfs/dir/6c28277cadd21ce4696a8efd122219dcd5c5a7379f77fcff4e5b68f178c1e069",
        "/ldapbackup": "/var/lib/docker/vfs/dir/b45d3ac4989c6748cb62c52c89002ea67952696c40916a3dc7fb1574836080a5",
        "/mysqlbackup": "/var/lib/docker/vfs/dir/3b7eaaae588970cbf54373d14422aaa4167c5c74c52719d5b07f373b68fc92b4",
        "/opt/alfresco/alf_data": "/var/lib/docker/vfs/dir/54a74ac4dce9515e6635b2da0516c76762c43526a1ad207fcfa8e9b49ab4201f",
        "/snapshot": "/opt/snapshots/pepper-demo-site"
    },

Failed

"Volumes": {
        "/data": "/var/lib/docker/vfs/dir/6c28277cadd21ce4696a8efd122219dcd5c5a7379f77fcff4e5b68f178c1e069",
        "/ldapbackup": "/var/lib/docker/vfs/dir/b45d3ac4989c6748cb62c52c89002ea67952696c40916a3dc7fb1574836080a5",
        "/mysqlbackup": "/var/lib/docker/vfs/dir/3b7eaaae588970cbf54373d14422aaa4167c5c74c52719d5b07f373b68fc92b4",
        "/opt/alfresco/alf_data": "/var/lib/docker/vfs/dir/54a74ac4dce9515e6635b2da0516c76762c43526a1ad207fcfa8e9b49ab4201f",
        "/snapshot": "/var/lib/docker/vfs/dir/edd43cd06f2384c18340ee100debee85a2dd4b46c03cf9ad6e141addbbf67e18"
    },

So, it appears that the wrong volume is being mounted at /snapshot

I see this very regularly. If I run my command 10 times, I'll see it a couple of times.

If I run the docker equivalent, I never see this issue.

[aanand]

Could you reduce your fig.yml to the miminal amount of configuration necessary to reproduce the bug, and remove anything domain-specific (e.g. use build: ./build instead of image: ..., mount from ./data instead of /opt/snapshots/pepper-demo-site/)?

[dancrumb]

Put the minimal recreate into https://github.com/dancrumb/github-fig-443

[dancrumb]

The system that I am seeing this on:

$ docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.1
Git commit (client): fa7b24f
OS/Arch (client): linux/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.3.1
Git commit (server): fa7b24f

My local system, that I have not been able to recreate this on:

$ docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070

[dancrumb]

Let me know if you want me to run any instrumented versions of fig to get more information here.

[dancrumb]

Were you able to recreate this issue locally?
This is pretty much a deal-breaker for us, because it means we can't use it on our CI server.

[GrantGochnauer]

+1

[dancrumb]

Unfortunately, we're going to have to abandon fig because of this instability. Switching to crane.

I'm happy so provide more data, if you need it, but it'll take me longer, since I won't have fig running any more.

[sachinkagarwal]

+1

[rolo]

I switched to crane and ended up hitting the same issue and was eventually able to replicate it via Docker directly. See my comments here #328 (comment)

Upgrading to the latest Docker dev version fixed the issue for me. Build instructions here, https://docs.docker.com/contributing/devenvironment/

[thaJeztah]

Docker 1.3 was just released today, so you probably don't even have to build it yourself

[rolo]

Oh, brilliant :)

[bfirsh]

@rolo Let me know if Docker 1.3 fixes it and I can close this!

[rolo]

@bfirsh Yep, have just tested with Docker 1.3 and all seems fine. Cheers!

[bfirsh]

Woot!

[ChiChou]

I'm using Docker 1.3 and first time I ran fig everything's fine. But when I have restarted boot2docker I met the same problem.

[leeahoward]

I am having this problem on osx. Docker 1.3.1, fig 1.0.1, boot2docker 1.3.1
It works if I use the equivalent docker command, but using the fig config it fails to mount the host volume.

[richstandbrook]

I'm also getting this issue. Docker 1.3.0 boot2docker 1.3.0 and fig 1.0.1. Running OS X Yosemite.

[filosganga]

I have the same issue with Docker 1.4.1 boot2docker v1.4.1 and fig 1.0.1. Running OSX 10.9.5.

[mikepurvis]

Docker 1.4.1 on Ubuntu 14.04; I believe I'm still seeing this.

[jaycyb]

+1 still happening here, please re-open the issue.

$ docker version
Client version: 1.4.0
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 4595d4f
OS/Arch (client): linux/amd64
Server version: 1.4.0
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 4595d4f

$ fig --version
fig 1.0.1

Update: after reverting back to Docker 1.3.1 and it works fine

[jaycyb]

looks like it's being addressed on #622

[gquemener]

I'm also having the same issue.

Here's my configuration:

$ docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.4
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.0
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 4595d4f

$ fig --version
fig 1.0.1

$ cat fig.yml
webdev:
    build: ./docker/apache-dev
    command: "/usr/sbin/apache2ctl -D FOREGROUND"
    ports:
        - "80:80"
    volumes:
        - "./apache-logs:/var/log/apache2"
        - "./:/var/www"
    working_dir: /var/www
    links:
        - dbdev
dbdev:
    build: ./docker/mysql
    ports:
        - "3306:3306"
    volumes:
        - ./app/database:/var/lib/mysql

and current dir is not mounted in /var/www

$ fig run webdev ls -l /var/www
total 0

$ ls -l
total 30236
....

Here's webdev inspect result:

[{
    "AppArmorProfile": "",
    "Args": [
        "-D",
        "FOREGROUND"
    ],
    "Config": {
        "AttachStderr": true,
        "AttachStdin": false,
        "AttachStdout": true,
        "Cmd": [
            "/usr/sbin/apache2ctl",
            "-D",
            "FOREGROUND"
        ],
        "CpuShares": 0,
        "Cpuset": "",
        "Domainname": "",
        "Entrypoint": [],
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "APACHE_RUN_USER=www-data",
            "APACHE_RUN_GROUP=www-data",
            "APACHE_LOG_DIR=/var/log/apache2",
            "SERVER_NAME=localhost",
            "APACHE_LOG_LEVEL=info",
            "REFRESHED_AT=2014-11-12"
        ],
        "ExposedPorts": {
            "80/tcp": {}
        },
        "Hostname": "5191f6ad9b09",
        "Image": "my_project_webdev",
        "MacAddress": "",
        "Memory": 0,
        "MemorySwap": 0,
        "NetworkDisabled": false,
        "OnBuild": null,
        "OpenStdin": false,
        "PortSpecs": null,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": {
            "/var/log/apache2": {},
            "/var/www": {}
        },
        "WorkingDir": "/var/www"
    },
    "Created": "2014-12-19T13:35:41.445992488Z",
    "Driver": "devicemapper",
    "ExecDriver": "native-0.2",
    "HostConfig": {
        "Binds": [
            "/home/gildas/projects/my_project/apache-logs:/var/log/apache2:rw",
            "/home/gildas/projects/my_project:/var/www:rw"
        ],
        "CapAdd": null,
        "CapDrop": null,
        "ContainerIDFile": "",
        "Devices": null,
        "Dns": null,
        "DnsSearch": null,
        "ExtraHosts": null,
        "IpcMode": "",
        "Links": [
            "/my_project_dbdev_1:/my_project_webdev_1/my_project_dbdev_1",
            "/my_project_dbdev_1:/my_project_webdev_1/dbdev",
            "/my_project_dbdev_1:/my_project_webdev_1/dbdev_1"
        ],
        "LxcConf": null,
        "NetworkMode": "bridge",
        "PortBindings": {
            "80/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "80"
                }
            ]
        },
        "Privileged": false,
        "PublishAllPorts": false,
        "RestartPolicy": {
            "MaximumRetryCount": 0,
            "Name": ""
        },
        "SecurityOpt": null,
        "VolumesFrom": [
            "c0574d4f51ca1d0705ce7248281f1c3c26a3c5da57c1acecc821f2a0e21be5a8"
        ]
    },
    "HostnamePath": "/var/lib/docker/containers/5191f6ad9b09f32d1e7fdc9cba191970175ec49096aa1d354c5b7121af174ac0/hostname",
    "HostsPath": "/var/lib/docker/containers/5191f6ad9b09f32d1e7fdc9cba191970175ec49096aa1d354c5b7121af174ac0/hosts",
    "Id": "5191f6ad9b09f32d1e7fdc9cba191970175ec49096aa1d354c5b7121af174ac0",
    "Image": "7d06a618867fa2c70578909a948314f3075a5febb78fcf43cd459c9c60199d57",
    "MountLabel": "",
    "Name": "/my_project_webdev_1",
    "NetworkSettings": {
        "Bridge": "docker0",
        "Gateway": "172.17.42.1",
        "IPAddress": "172.17.0.60",
        "IPPrefixLen": 16,
        "MacAddress": "02:42:ac:11:00:3c",
        "PortMapping": null,
        "Ports": {
            "80/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "80"
                }
            ]
        }
    },
    "Path": "/usr/sbin/apache2ctl",
    "ProcessLabel": "",
    "ResolvConfPath": "/var/lib/docker/containers/5191f6ad9b09f32d1e7fdc9cba191970175ec49096aa1d354c5b7121af174ac0/resolv.conf",
    "State": {
        "Error": "",
        "ExitCode": 0,
        "FinishedAt": "0001-01-01T00:00:00Z",
        "OOMKilled": false,
        "Paused": false,
        "Pid": 26303,
        "Restarting": false,
        "Running": true,
        "StartedAt": "2014-12-19T13:35:42.112575725Z"
    },
    "Volumes": {
        "/var/log/apache2": "/var/lib/docker/vfs/dir/8d8ace23570f715d1b25b68b5bed9503abbd5e22fbb1707757f7972d7f61050b",
        "/var/www": "/var/lib/docker/vfs/dir/17436c6b2c98d10bd253d66453844904de5051c0514e48078c001ff6ce90007d"
    },
    "VolumesRW": {
        "/var/log/apache2": true,
        "/var/www": true
    }
}
]

[cpuguy83]

@gquemener You've only updated your client to 1.4.1, you need server 1.4.1

[gquemener]

Oh gosh, I totally missed that!

Thanks @cpuguy83 :D