Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix volumes on recreate #711

Merged
merged 2 commits into from
Jan 15, 2015
Merged

Conversation

dnephin
Copy link

@dnephin dnephin commented Dec 6, 2014

Resolves: #447 (and probably some other related issues)

Instead of using volumes_from on the intermediate container, inspect the volumes, and copy them over the to intermediate container and from the intermediate to the final container.

@bfirsh
Copy link

bfirsh commented Dec 9, 2014

This is a great idea. Needs a rebase though. :(

@dnephin
Copy link
Author

dnephin commented Dec 9, 2014

I need to stop adding tests at the bottom of the file so I don't have to rebase everytime.

Rebased

@bfirsh
Copy link

bfirsh commented Dec 9, 2014

cc @aanand

@dnephin
Copy link
Author

dnephin commented Dec 10, 2014

I suppose this could use some integration tests showing that it solves the issue.

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
@dnephin dnephin force-pushed the fix_volumes_on_recreate branch 3 times, most recently from b2efda5 to d273947 Compare January 14, 2015 20:53
@dnephin
Copy link
Author

dnephin commented Jan 14, 2015

Ok, this is rebased, and I've added an integration test (which fails on master).

Signed-off-by: Daniel Nephin <dnephin@gmail.com>
@dnephin
Copy link
Author

dnephin commented Jan 14, 2015

Oh right, I have to actually start the container in older versions of docker. This test was passing for docker 1.4.1. Pushed another update to get the test working for docker 1.2.0

@aanand
Copy link

aanand commented Jan 15, 2015

LGTM

aanand added a commit that referenced this pull request Jan 15, 2015
@aanand aanand merged commit 55095ef into docker:master Jan 15, 2015
@aanand
Copy link

aanand commented Jan 15, 2015

spinal-tap-11-1321038225

@aanand aanand added this to the 1.1.0 milestone Jan 15, 2015
@aanand
Copy link

aanand commented Jan 16, 2015

Hmm, tests are failing for me against Docker 1.4.1.

@dnephin
Copy link
Author

dnephin commented Jan 16, 2015

Hmm, I didn't actually test it on 1.4.1 after making the change for 1.2.
Does it work without the start_container line?
On Jan 16, 2015 11:09 AM, "Aanand Prasad" notifications@github.com wrote:

Hmm, tests are failing for me against Docker 1.4.1.


Reply to this email directly or view it on GitHub
#711 (comment).

@cpuguy83
Copy link
Contributor

I've not reviewed this code but... there exists a bug in 1.4.1 where on start Docker will always try to re-apply everything in volumes-from. If the volumes-from container no longer exists, it will error out.
Fix is slated for 1.5.0

This may be related.

@dnephin
Copy link
Author

dnephin commented Jan 19, 2015

@aanand tests on master are working for me with docker 1.4.1

@aanand
Copy link

aanand commented Jan 19, 2015

I can reproduce it on master on a fresh Digital Ocean machine with Docker 1.4.1 - here's the relevant bit of script/test output:

+ docker run -v /var/run/docker.sock:/var/run/docker.sock --rm --entrypoint nosetests fig
......stty: standard input: Inappropriate ioctl for device
.stty: standard input: Inappropriate ioctl for device
.stty: standard input: Inappropriate ioctl for device
....................F...............................F.......................................................................................................................
======================================================================
FAIL: test_project_up_recreates_containers (tests.integration.project_test.ProjectTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/tests/integration/project_test.py", line 112, in test_project_up_recreates_containers
    self.assertEqual(db_container.get('Volumes./etc'), db_volume_path)
AssertionError: u'/var/lib/docker/vfs/dir/5dbe0ffc9cecbe2b464bbeb38b2a5557d860db46da112d490fbed6 [truncated]... != u'/var/lib/docker/vfs/dir/65e3c1cc2d1ca83f5f63eb375b66a3eb880a4e0b6ab7d0f5871d83 [truncated]...
- /var/lib/docker/vfs/dir/5dbe0ffc9cecbe2b464bbeb38b2a5557d860db46da112d490fbed6069e9be6f8
+ /var/lib/docker/vfs/dir/65e3c1cc2d1ca83f5f63eb375b66a3eb880a4e0b6ab7d0f5871d83a245e08b31

-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/images/json?only_ids=0&all=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=0&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 441
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
fig.service: INFO: Creating figtest_db_1...
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/create?name=figtest_db_1 HTTP/1.1" 201 90
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/d81d3a4699414fbc9c2782de2c1f76bdbf410ac11fff7c14bf122fd03a19a5ea/json HTTP/1.1" 200 1688
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/d81d3a4699414fbc9c2782de2c1f76bdbf410ac11fff7c14bf122fd03a19a5ea/start HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=0&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 659
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=0&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 659
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=0&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 659
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/d81d3a4699414fbc9c2782de2c1f76bdbf410ac11fff7c14bf122fd03a19a5ea/json HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
fig.service: INFO: Creating figtest_web_1...
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/create?name=figtest_web_1 HTTP/1.1" 201 90
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/cbfb5988012d160c2aeaa7e8f628ab62e0cdf6496d8510ccefb6248cff6444a0/json HTTP/1.1" 200 1574
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/cbfb5988012d160c2aeaa7e8f628ab62e0cdf6496d8510ccefb6248cff6444a0/start HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
fig.service: INFO: Recreating figtest_db_1...
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/d81d3a4699414fbc9c2782de2c1f76bdbf410ac11fff7c14bf122fd03a19a5ea/stop?t=10 HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/create HTTP/1.1" 201 90
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/0ef6711c1da844a1498f650107355d22250ad11e1f6f026a784212b73f13ef45/json HTTP/1.1" 200 1565
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/d81d3a4699414fbc9c2782de2c1f76bdbf410ac11fff7c14bf122fd03a19a5ea/json HTTP/1.1" 200 2013
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/0ef6711c1da844a1498f650107355d22250ad11e1f6f026a784212b73f13ef45/start HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/0ef6711c1da844a1498f650107355d22250ad11e1f6f026a784212b73f13ef45/wait HTTP/1.1" 200 17
requests.packages.urllib3.connectionpool: DEBUG: "DELETE /run/docker.sock/v1.14/containers/d81d3a4699414fbc9c2782de2c1f76bdbf410ac11fff7c14bf122fd03a19a5ea?force=False&link=False&v=False HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/create?name=figtest_db_1 HTTP/1.1" 201 90
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/f38622b9f3ff995e0de9c065d4bfd80cbe9c1eb4c1c9b0dd408f8d0e8c379649/json HTTP/1.1" 200 1688
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/0ef6711c1da844a1498f650107355d22250ad11e1f6f026a784212b73f13ef45/json HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/f38622b9f3ff995e0de9c065d4bfd80cbe9c1eb4c1c9b0dd408f8d0e8c379649/start HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "DELETE /run/docker.sock/v1.14/containers/0ef6711c1da844a1498f650107355d22250ad11e1f6f026a784212b73f13ef45?force=False&link=False&v=False HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=0&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 870
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=0&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 870
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/f38622b9f3ff995e0de9c065d4bfd80cbe9c1eb4c1c9b0dd408f8d0e8c379649/json HTTP/1.1" 200 None
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_recreate_containers (tests.integration.service_test.ServiceTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/tests/integration/service_test.py", line 164, in test_recreate_containers
    self.assertEqual(new_container.get('Volumes')['/etc'], volume_path)
AssertionError: u'/var/lib/docker/vfs/dir/2afba3ba6532907fd4d7db9ec9036ac753f516d24b8f13c80b0367 [truncated]... != u'/var/lib/docker/vfs/dir/ea73d18136c20432013ecd7ba247849be929eeac258da7ac21f0e1 [truncated]...
- /var/lib/docker/vfs/dir/2afba3ba6532907fd4d7db9ec9036ac753f516d24b8f13c80b0367127fc82c71
+ /var/lib/docker/vfs/dir/ea73d18136c20432013ecd7ba247849be929eeac258da7ac21f0e1fc1856634e

-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/b0aa75d1037b9acaaef6b327d3580c516bccb6378fd6f33b36d4ee40d6cb0d0a/kill HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "DELETE /run/docker.sock/v1.14/containers/b0aa75d1037b9acaaef6b327d3580c516bccb6378fd6f33b36d4ee40d6cb0d0a?force=False&link=False&v=False HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/images/json?only_ids=0&all=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/create?name=figtest_db_1 HTTP/1.1" 201 90
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/aee0f255efc43c339aa6fe40c37dc2bdc1efb0c865462cee5c56c923111eded0/json HTTP/1.1" 200 1683
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/aee0f255efc43c339aa6fe40c37dc2bdc1efb0c865462cee5c56c923111eded0/start HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/aee0f255efc43c339aa6fe40c37dc2bdc1efb0c865462cee5c56c923111eded0/json HTTP/1.1" 200 2045
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
fig.service: INFO: Recreating figtest_db_1...
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/aee0f255efc43c339aa6fe40c37dc2bdc1efb0c865462cee5c56c923111eded0/stop?t=10 HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/create HTTP/1.1" 201 90
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/323ac32cff164d182b13a4202fe38065ddf88ebd6a3aedef3e261d10544a0310/json HTTP/1.1" 200 1563
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/aee0f255efc43c339aa6fe40c37dc2bdc1efb0c865462cee5c56c923111eded0/json HTTP/1.1" 200 2008
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/323ac32cff164d182b13a4202fe38065ddf88ebd6a3aedef3e261d10544a0310/start HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/323ac32cff164d182b13a4202fe38065ddf88ebd6a3aedef3e261d10544a0310/wait HTTP/1.1" 200 17
requests.packages.urllib3.connectionpool: DEBUG: "DELETE /run/docker.sock/v1.14/containers/aee0f255efc43c339aa6fe40c37dc2bdc1efb0c865462cee5c56c923111eded0?force=False&link=False&v=False HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/json?all=1&limit=-1&trunc_cmd=1&size=0 HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/create?name=figtest_db_1 HTTP/1.1" 201 90
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/e1e8813013ba99b0a7278d8d57047c6861703c59753eba57217a1ddfd24c46e9/json HTTP/1.1" 200 1683
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/323ac32cff164d182b13a4202fe38065ddf88ebd6a3aedef3e261d10544a0310/json HTTP/1.1" 200 None
requests.packages.urllib3.connectionpool: DEBUG: "POST /run/docker.sock/v1.14/containers/e1e8813013ba99b0a7278d8d57047c6861703c59753eba57217a1ddfd24c46e9/start HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "DELETE /run/docker.sock/v1.14/containers/323ac32cff164d182b13a4202fe38065ddf88ebd6a3aedef3e261d10544a0310?force=False&link=False&v=False HTTP/1.1" 204 0
requests.packages.urllib3.connectionpool: DEBUG: "GET /run/docker.sock/v1.14/containers/e1e8813013ba99b0a7278d8d57047c6861703c59753eba57217a1ddfd24c46e9/json HTTP/1.1" 200 None
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 180 tests in 216.408s

FAILED (failures=2)

@dnephin
Copy link
Author

dnephin commented Jan 20, 2015

@aanand I was able to reproduce the error on another host

I believe the issue is related to this TODO I left (service.py#L297)

The intermediate container has the correct volumes, but the final container is being passed the config which creates a new volume first (before the correct volume is applied during start_container()). So when start_container() tries to apply volumes they are ignored instead of overriding.

I'll submit another PR to address the 1.4.1 issue.

dnephin pushed a commit to dnephin/compose that referenced this pull request Jan 20, 2015
…reate"

This reverts commit 55095ef, reversing
changes made to 72095f5.

Signed-off-by: Daniel Nephin <dnephin@yelp.com>
aanand added a commit that referenced this pull request Jan 20, 2015
Revert #711 from dnephin/fix_volumes_on_recreate
yuval-k pushed a commit to yuval-k/compose that referenced this pull request Apr 10, 2015
Fix volumes on recreate
Signed-off-by: Yuval Kohavi <yuval.kohavi@gmail.com>
yuval-k pushed a commit to yuval-k/compose that referenced this pull request Apr 10, 2015
…reate"

This reverts commit 55095ef, reversing
changes made to 72095f5.

Signed-off-by: Daniel Nephin <dnephin@yelp.com>

Signed-off-by: Yuval Kohavi <yuval.kohavi@gmail.com>
yuval-k pushed a commit to yuval-k/compose that referenced this pull request Apr 10, 2015
…nges

Revert docker#711 from dnephin/fix_volumes_on_recreate
Signed-off-by: Yuval Kohavi <yuval.kohavi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

reload fig.yml to mount the changed volumes
4 participants