-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
failed to get network during CreateEndpoint #888
failed to get network during CreateEndpoint #888
Comments
Once create the container, same with the docker command: # worked
$ docker restart 0_zookeeper_1
0_zookeeper_1
# failed
$ docker restart 0_zookeeper_1
Error response from daemon: Cannot restart container 0_zookeeper_1: failed to get network during CreateEndpoint: network zreirydw66jtrf0z1kjx8lnco not found
# worked
$ docker restart 0_zookeeper_1
0_zookeeper_1
# failed
$ docker restart 0_zookeeper_1
Error response from daemon: Cannot restart container 0_zookeeper_1: failed to get network during CreateEndpoint: network zreirydw66jtrf0z1kjx8lnco not found |
@dingzhengkai thanks for raising this issue, it seems to be related to the
I'm not too familiar with this code flow but I narrowed the race to this line by adding delays ( any hints @cpuguy83 ? |
There seems to be a race between detach and attach. I'm not even sure how the literal detach from the node is happening. The |
libnetwork just seems all wrong: In deleting an object from the kv store it does an atomic operation, but if there is a failure it just retries by fetching the latest version. It doesn't check what's changed it just goes along with the delete. |
Can someone fix it? Thanks. |
@thaJeztah,@tiborvass can you help with this? |
A fix would be highly appreciated, for now I only had it in my pycharm dev environment, so it was uncomfortable to start twice to get the container running, but now I' hitting this also in some CI/CD chains which is not that easy to fix. |
I do not believe I have software with more than +5 million pageviews per month caught on exactly that! does anyone have any solution? |
@cpuguy83 I don't think this is related to the KV store logic Took another look at the logs and it might be related to the async nature in which we create and delete the
|
Just want to add, that I have been having this issue for close to a year. docker restart will not work, however a docker stop and docker start is successful. Ive read many posts that all touch on the subject, and this post falls inline with our findings on the race condition. How do we go about gettting the escalated to the right person and getting a fix in the master branch? |
I have a few customers having the same issue (they use I opened debug mode on docker engine, and found that it is a race condition. The issue happened in the following time order for example:
I think there's a issue in swarmkit related code: |
We use workaround for this issue, we create network in global mode, so after CI/CD there is no errors |
full diff: moby/libnetwork@2e24aed...9e99af2 - moby/libnetwork#2548 Add docker interfaces to firewalld docker zone - fixes docker/for-linux#957 DNS Not Resolving under Network [CentOS8] - fixes moby/libnetwork#2496 Port Forwarding does not work on RHEL 8 with Firewalld running with FirewallBackend=nftables - store.getNetworksFromStore() remove unused error return - moby/libnetwork#2554 Fix 'failed to get network during CreateEndpoint' - fixes/addresses docker/for-linux#888 failed to get network during CreateEndpoint - moby/libnetwork#2558 [master] bridge: disable IPv6 router advertisements - moby/libnetwork#2563 log error instead if disabling IPv6 router advertisement failed - fixes docker/for-linux#1033 Shouldn't be fatal: Unable to disable IPv6 router advertisement: open /proc/sys/net/ipv6/conf/docker0/accept_ra: read-only file system Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: moby/libnetwork@2e24aed...9e99af2 - moby/libnetwork#2548 Add docker interfaces to firewalld docker zone - fixes docker/for-linux#957 DNS Not Resolving under Network [CentOS8] - fixes moby/libnetwork#2496 Port Forwarding does not work on RHEL 8 with Firewalld running with FirewallBackend=nftables - store.getNetworksFromStore() remove unused error return - moby/libnetwork#2554 Fix 'failed to get network during CreateEndpoint' - fixes/addresses docker/for-linux#888 failed to get network during CreateEndpoint - moby/libnetwork#2558 [master] bridge: disable IPv6 router advertisements - moby/libnetwork#2563 log error instead if disabling IPv6 router advertisement failed - fixes docker/for-linux#1033 Shouldn't be fatal: Unable to disable IPv6 router advertisement: open /proc/sys/net/ipv6/conf/docker0/accept_ra: read-only file system Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Upstream-commit: 219e7e7ddcf5f0314578d2a517fc0832f03622c1 Component: engine
This maybe similar to
moby/moby#35288
moby/libnetwork#2341
moby/libnetwork#2015
Expected behavior
docker-compose -f zk.yml restart
would restart the container correctlyActual behavior
Steps to reproduce the behavior
Output of
docker version
:Output of
docker info
:Output of
docker network ls
:Additional environment details (AWS, VirtualBox, physical, etc.)
zk.yml
No other container on my os.
Is anyone can help?
The text was updated successfully, but these errors were encountered: