docker compose up recreates running container that does not have their configs changed in docker-compose.yml #9600
Comments
|
I can't reproduce locally. Can you provide a sample compose file? Once without network then with for the |
|
@debdutdeb I tried with your compose file, and ran The first run, both of the containers came up. This is expected. The second run, both of them are recreated, but I did not modify anything. Afterwards, I tried it again, it does not happen. Even with modified compose file, it doesn't always occur. |
|
Noticed i was using 2.0.something - old fork of mine. Checking again with 2.6.x. |
|
Looks like you're right 👀 |
|
But can't reproduce reliably though. |
|
Yea, it does not always occur, yet I don't see a pattern as to why it occurs when it occurs |
|
I think the mechanism for determining config changes is by hashing the config and storing that hash in the label You can check if the label changes between invocations to further verify what's going on. Different compose executable versions will sometimes produce different hash outputs on the same config file, so make sure tests are done with a consistent binary version. |
|
Is this a duplicate of #9357? |
|
I'm facing this issue in docker-compose 2.11.2. To reliably reproduce : Compose file Pre-condition:
Step:
Note
|
|
I'm getting the same issue and all images are available locally. In my case, it's triggered by a I checked the hash, it does not change across the unexpected recreates. EDIT: as pointed out in #10068, in my case the result of v2.12.0 |
|
Seeing this issue on v2.14.0 too - any word on a fix? We have a use cae where we need to gradually add some services to the compose file and start them only after confirming the previous service is happy and healthy. |
|
After I commented on this, I discovered the flag “-- no-recreate” which seems to solve the problem for me. But I don’t understand why this behaviour isn’t the default. Note we just moved from compose 1.x to latest 2.x so still figuring out a few changes. |
That flag shouldn't be the default, you're just working around the bug. |
|
Hi guys, this is a pretty bad regression on well established behaviour of docker compose. We will likey need to return to version 1.x until this is addressed - what needs to happen to have the core developers take a serious look at this? |
@sillidev does it not make sense to recreate those containers if the one you're running ( |
|
@debdutdeb No, this is very definitely broken. |
|
I'll have to compare with v1's behaviour.. don't remember that. |
|
@debdutdeb It worked in 2.2.x I believe. This is a regression. It's also nonsense behaviour, there's no way on earth it is desirable. |
|
Here is my example for you. It definely seems to be something that only occurs after an initial docker-compose up when all container images are initially pulled. As an example here is a bringing up our docker compose stack ( i have redacted unimportant details like container names and aws account id): docker-compose.yml: terminal log: Then down one of the running containers and use docker-compose up -d to bring it back. You will notice that this version recreated ALL the containers where as the docker-compose 1.x only restarts the stopped container: If I then stop one container again, and then use docker-compose up -d to restart it, it now behaves as per ver 1.x and simply restarts the one stopped container: I can consistently reproduce this, their is a clear issue here. Note that this also occurs if after an initial up, where all images are pulled, you then edit the docker-compose file and update the image tag of one of the containers and do a docker-compose up -d it will recreate all containers, howeve if you then try again, it will only recreate the container with the change tag as one would expect. Example of this: There seems to be some bug here somewhere |
|
I see. I'll try to track it down tonight once I'm off. |
|
just tried my local fork, couldn't reproduce with the above steps version: '3.2'
services:
router:
image: httpd:2.4
database:
image: httpd:2.4
one:
image: httpd:2.4
two:
image: httpd:2.4 |
|
Can't reproduce the tag change either; services:
one:
image: httpd:2.4
two:
image: httpd:2.4
three:
image: httpd:2.4
four:
image: httpd:2.4out - [I] ⋊> ~/D/R/compose on recreate docker rmi httpd:2.4 httpd:2 13:54:04
Untagged: httpd:2
Untagged: httpd@sha256:87a012bf99bf5e3e0f628ac1f69abbeab534282857fba3a359ca3a3f4a02429a
Deleted: sha256:dc0ffcb6f5c2ffbf82b074a8a875fb771f92cbff7a4f765a8e909544b34e88f5
Deleted: sha256:a46de20bb0879acebd7d4019b20949372e983c9b2734287b5f645d9f4c295473
Deleted: sha256:d8ebf31c170c9c8880f78f88d8c17590764dabc8e7f581fe43f3c9cc991ab810
Deleted: sha256:422dbb594c3ecc7e51ad4f9d53db0b2db7f0bfb9e834750d74c96b3e5742b467
Deleted: sha256:4308bb6033634dee7bac4308a5f7f9548f1ddff2e61e3e5d0d1a94ae24913681
Deleted: sha256:afd7e44a4e086678d666c3721e0e45dc85045db6952832135ea2d32e1a4db966
Error: No such image: httpd:2.4
[I] ⋊> ~/D/R/compose on recreate ./bin/build/docker-compose -f /tmp/compose.yml up -d --remove-orphans 13:54:05
[+] Running 9/9
⠿ one Pulled 8.0s
⠿ three Pulled 8.0s
⠿ two Pulled 8.0s
⠿ 934ce60d1040 Pull complete 4.5s
⠿ e0c9391b348d Pull complete 4.6s
⠿ e106b5ae0073 Pull complete 4.7s
⠿ 339abac02bc5 Pull complete 5.2s
⠿ 9e37b0a5502a Pull complete 5.2s
⠿ four Pulled 8.0s
[+] Running 5/5
⠿ Network tmp_default Created 0.0s
⠿ Container tmp-two-1 Started 0.8s
⠿ Container tmp-three-1 Started 0.5s
⠿ Container tmp-one-1 Started 0.7s
⠿ Container tmp-four-1 Started 0.6s
[I] ⋊> ~/D/R/compose on recreate gsed '0,/2.4/ s/2.4/2/1' /tmp/compose.yml -i 13:54:17
[I] ⋊> ~/D/R/compose on recreate ./bin/build/docker-compose -f /tmp/compose.yml up -d --remove-orphans 13:54:30
[+] Running 1/1
⠿ one Pulled 3.0s
[+] Running 4/4
⠿ Container tmp-two-1 Running 0.0s
⠿ Container tmp-three-1 Running 0.0s
⠿ Container tmp-one-1 Started 1.3s
⠿ Container tmp-four-1 Running 0.0sif I introduce a dependency array - services:
one:
image: httpd:2.4
depends_on:
- two
two:
image: httpd:2.4
three:
image: httpd:2.4
depends_on:
- one
four:
image: httpd:2.4out - [I] ⋊> ~/D/R/compose on recreate ./bin/build/docker-compose -f /tmp/compose.yml up -d --remove-orphans 13:56:19
[+] Running 5/5
⠿ Network tmp_default Created 0.0s
⠿ Container tmp-four-1 Started 0.4s
⠿ Container tmp-two-1 Started 0.4s
⠿ Container tmp-one-1 Started 0.5s
⠿ Container tmp-three-1 Started 0.7s
[I] ⋊> ~/D/R/compose on recreate gsed '0,/2.4/ s/2.4/2/1' /tmp/compose.yml -i 13:56:21
[I] ⋊> ~/D/R/compose on recreate ./bin/build/docker-compose -f /tmp/compose.yml up -d --remove-orphans 13:56:34
[+] Running 4/4
⠿ Container tmp-four-1 Running 0.0s
⠿ Container tmp-two-1 Running 0.0s
⠿ Container tmp-one-1 Started 2.5s
⠿ Container tmp-three-1 Started 1.5sYou'll see it now recreates |
|
I can reproduce on the compose shipped with docker desktop though; 2.6.1 |
|
@debdutdeb So we are on: and can reproduce it pretty consistently. We are running on centos7 (its a packer built AMI), with docker installed as follows: and compose installed as follows: what version is your local fork based on and we can install that version and try. We can consistently reproduce this so let me know if other logs or something would also help. |
|
try v2.15.0 - the very latest one; |
|
It did happen to me with newer docker-compose binaries. The interesting thing is that some (semi-random) containers got restarted even when working correctly. |
|
@debdutdeb I don't see anything in the 2.15.0 changelog which hints that this issue was fixed. What makes you suggest that? |
|
@benlangfeld I tested against the latest tag. Couldn't reproduce the issue with the steps provided, if you look further up the thread, I shared my observations a while ago. I don't remember exactly but I did track down a specific function call path (mustRecreate?) that i noticed got updated - which made me think the issue probably was fixed already. Will try to retrace when I'm at my desk again. |
|
Created two new issues (as above) to track additional issues @ndeloof |
|
@ndeloof how likely are we to get a point release including the bugfix? |
|
I noticed if I do |
|
Same here, the recreation is making me crazy 🤣 |
|
As per my comment and testing here -> #10308 (comment) this looks resolved, but maybe @imjuzcy and @benlangfeld can review/test/confirm? |
|
I've tried a few times with v2.17.0-rc.1, and it does seem to be resolved. But as described above, it happens randomly, and so far there's no pattern that has been identified when or why it'll happen, so might need more input on whether it is actually resolved. |
|
I think that a very viable trigger is that the image gets updated.
Imagine the following flow:
- deploy a stack using compost
- run image update (with an actual update to the image)
- run compose up (which restarts the relevant parts in the stack)
- on subsequent runs of 'up' without any additional change - the stack gets
restarted.
…On Thu, Mar 9, 2023, 10:25 AM imjuzcy ***@***.***> wrote:
I've tried a few times with v2.17.0-rc.1, and it does seem to be resolved.
But as described above, it happens randomly, and so far there's no pattern
that has been identified when or why it'll happen, so might need more input
on whether it is actually resolved.
—
Reply to this email directly, view it on GitHub
<#9600 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABUMMVXYNRMTWT3USO4HQ73W3GOY5ANCNFSM5Z3CIGAQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Not necessarily. Below is my To build
After building |
|
So after the test above, I changed the binary back from v2.6.0 to v2.17.0-rc.1, and ran |
|
@imjuzcy please check the |
|
Ah, got it. But back to v2.17.0-rc.1, it's still solid for me, no unnecessary recreates so far. |
|
I also updated to the rc, and working very nicely with nicer coloured output 😆 I've got auto-updates for compose, so it seems this got broken in v2.16.0... |
|
Closing this issue then, feel free to comment / open a follow-up if this strikes back with 2.17+ |
|
Sorry,but how can i install the v2.17.0 version in debian? |
|
@leranp download binary from https://github.com/docker/compose/releases and install under |
|
@ndeloof You missed the |
|
Any tip for installing on Ubuntu? Or even just how to downgrade on Ubuntu to avoid this regression |
|
@aehlke https://docs.docker.com/compose/install/linux/#install-the-plugin-manually Basically the instructions provided just above your reply. Did you try them before asking? Personally I'll just wait for the next release. |
|
Yes I had tried it without luck. Looks like my system has something wrong with it if those are also the ubuntu intructions. Thank you |
|
thankfully just came across this issue and I managed to use apt (on Ubuntu 22.10) to update from |
|
Upgrading the package on my Debian 11 server also fixed this issue. |
|
Not sure if it is a similar issue, but today I face the problem running |
and others
Description
docker compose upstops and recreates running containers that do not have their config changed.Steps to reproduce the issue:
docker compose updocker-compose.ymldocker compose upagainDescribe the results you received:
A lot of containers that are not related to the service that has its config changed are recreated and restarted.
Describe the results you expected:
Only the container of the service that has new config should be recreated and those that depend on it should be restarted.
Additional information you deem important (e.g. issue happens only occasionally):
After switching to v2 it happens almost all of the time. Previously on v1 there was no such problem.
Output of
docker compose version:Output of
docker info:Additional environment details:
The text was updated successfully, but these errors were encountered: