17.06 won't deploy stack anymore. no suitable node (unsupported platform on 3 nodes) on armhf docker cluster #2294

trunet · 2017-07-02T12:58:46Z

Hello,

I have an armhf cluster and since 17.06 installed, I'm getting this error and my stacks doesn't come up, it stays as pending forever. There's no constraint. This was working perfectly on 17.03.

Inspecting the task, I'm getting:

        "Status": {
            "Timestamp": "2017-07-02T12:51:00.556959128Z",
            "State": "pending",
            "Message": "no suitable node (unsupported platform on 3 nodes)",
            "ContainerStatus": {},
            "PortStatus": {}
        },

docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 9
Server Version: 17.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local nfs
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: chzqrk30d8aph7ikg60owdbjz
 Is Manager: true
 ClusterID: 1ucdfzovu4whdawkzv8wbfeb6
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 192.168.178.6
 Manager Addresses:
  192.168.178.6:2377
  192.168.178.7:2377
  192.168.178.8:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.34-45
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: armv7l
CPUs: 8
Total Memory: 1.949GiB
Name: odroid01.casaams.wsartori.com
ID: DRWB:PHMO:GCXF:GDTK:QXDF:V2WM:6ZAM:MC5G:GYID:7YOH:PBQC:BOYE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 registry:5000
 127.0.0.0/8
Live Restore Enabled: false

The text was updated successfully, but these errors were encountered:

trunet · 2017-07-02T13:10:21Z

probably introduced on 8edbb92 by @nishanttotla

nishanttotla · 2017-07-02T19:15:22Z

@trunet can you show the output for service inspect and also what docker info on the nodes looks like?

trunet · 2017-07-02T21:15:36Z

docker info is on the first comment.

docker service inspect:

[
    {
        "ID": "n0bhpy9ibyiiv2wkux65x1dsv",
        "Version": {
            "Index": 2286
        },
        "CreatedAt": "2017-07-02T21:13:59.560944718Z",
        "UpdatedAt": "2017-07-02T21:13:59.56740151Z",
        "Spec": {
            "Name": "portal_portal",
            "Labels": {
                "com.docker.stack.image": "arm32v7/httpd:latest",
                "com.docker.stack.namespace": "portal"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "arm32v7/httpd:latest@sha256:90fb0d45f7267ac96523f2030ab842d61c77c65999024b5f67771f599ae4bfa8",
                    "Labels": {
                        "com.docker.stack.namespace": "portal"
                    },
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/mnt/glusterfs/portal",
                            "Target": "/usr/local/apache2/htdocs"
                        }
                    ],
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {}
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Platforms": [
                        {
                            "Architecture": "arm",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "jzho3a68yjsqvakajo7b9yqv3",
                        "Aliases": [
                            "portal"
                        ]
                    }
                ],
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 2
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "ingress"
                    }
                ]
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip",
                "Ports": [
                    {
                        "Protocol": "tcp",
                        "TargetPort": 80,
                        "PublishedPort": 80,
                        "PublishMode": "ingress"
                    }
                ]
            },
            "Ports": [
                {
                    "Protocol": "tcp",
                    "TargetPort": 80,
                    "PublishedPort": 80,
                    "PublishMode": "ingress"
                }
            ],
            "VirtualIPs": [
                {
                    "NetworkID": "sposg3gl89sjhlhy1hdpqbwt6",
                    "Addr": "10.255.0.8/16"
                },
                {
                    "NetworkID": "jzho3a68yjsqvakajo7b9yqv3",
                    "Addr": "10.0.1.2/24"
                }
            ]
        }
    }
]

nishanttotla · 2017-07-03T17:38:19Z

I see the issue.

It seems like the node reports

OSType: linux
Architecture: armv7l

but the image reports

OSType: linux
Architecture: arm

and that causes scheduling to fail. We will need to add more normalization for architecture names. I'll do that but it would be useful to have a full list of architecture name variations.

cc @thaJeztah @aaronlehmann currently we only do this for x86_64 and amd64: https://github.com/docker/swarmkit/blob/master/manager/scheduler/filter.go#L288

trunet · 2017-07-03T17:56:05Z

I can think of aarch64 armv7l, but they are completely different architecture, aarch64 is 64-bits, armv7l is 32-bits.

also, check this: https://github.com/docker-library/official-images#architectures-other-than-amd64

trunet · 2017-07-03T17:59:44Z

@nishanttotla Is there a way to build the image with armv7l architecture instead of only arm, I didn't find anything on commit, push on build to add this? how docker hub is handling that, I suppose all https://hub.docker.com/r/arm32v7/ should uses armv7l?

nishanttotla · 2017-07-03T18:02:18Z

@trunet I'm not sure about that, but we'll see what's the best way to resolve this. Until then, you can get around this issue by using the --no-resolve-image flag on service create/update which will not add platform information to your service spec.

trunet · 2017-07-03T18:05:55Z

@nicolaka as I'm using stack, I didn't find the --no-resolve-image equivalent on compose 3.3 reference.

trunet · 2017-07-03T18:09:14Z

@nicolaka found it on https://docs.docker.com/engine/reference/commandline/stack_deploy/ thanks man

nishanttotla · 2017-07-03T18:11:04Z

@trunet apologies, yes for stacks you can use --resolve-image=never for now.

thaJeztah · 2017-07-03T21:13:17Z

We will need to add more normalization for architecture names.

Is there a list of valid architectures in the image's data? (Same for nodes)?

ARM is specifically tricky, as there's a lot of variations, and ways to express them (armhf, armv6, armv7 etc.)

If there's a list of valid values, we can decide to ignore other values

nishanttotla · 2017-07-05T21:54:06Z

Related: opencontainers/image-spec#661

@trunet I'm working on figuring out what the right fix is for this case.

stevvooe · 2017-07-05T22:35:52Z

@tianon Do you know how Config.Architecture is getting set for images in arm32v7? Are these just using GOARCH directly or is there a special build process?

tianon · 2017-07-05T22:40:58Z

@stevvooe that'd be GOARCH directly -- the builder for those actually used to be a 64bit build (and thus was setting Architecture to arm64), but I'm now using a 32bit build of Docker on a 64bit machine and using linux32 to ensure the kernel claims it's also 32bit (just to help cut down on the types of things that can go wrong when cross-building like this), so they should all be set to arm (which is as specific as an individual image manifest can get without a manifest list, as you know)

I think way back in Docker's history, someone thought it'd be helpful to include "multiarch" metadata on images, and just threw in GOOS and GOARCH as-is, and it hasn't been adjusted since. IMO that'd be better served by having a full platform object ala manifest lists, but figured for individual non-multiarch images it really didn't make much difference either way (especially for image-spec's 1.0).

tianon · 2017-07-05T22:42:30Z

In the manifest list pushing code we've got (which again, as you know, isn't deployed to library/ just yet), arm32v7 turns into something like the following:

platform:
  os: linux
  architecture: arm
  variant: v7

tianon · 2017-07-05T22:48:49Z

Ah, see moby/moby@f9359f5 for where those fields were first populated with GOARCH (appears it was hard-coded to x86_64 before, and no OS info).

stevvooe · 2017-07-05T22:50:35Z

@tianon The problem with GOARCH on images, is that we lose variant when assembling Platform from an image configuration: https://github.com/moby/moby/blob/master/api/server/router/distribution/distribution_routes.go#L120.

For the vast number of cases, I think we can assume variant is v7 but I am sure that will break.

tianon · 2017-07-05T23:02:58Z

Indeed -- I'm 100% with you on GOARCH being insufficient, which is why I wish individual non-list manifests had a full platform object so that this could be set properly. 😞

To give another example that's completely broken, all the images under i386 will have Architecture set to amd64, since I don't build them on real 32bit hardware (and didn't want to compile a custom Docker binary just to do those builds). They only exist (in my mind) as a rough sanity check for a simple way to test multiarch (since they can build on any amd64 machine as-is), but their metadata being wrong irks me a lot (it'll be correct when they're included in a manifest list, as seen under trollin, but the individual image metadata will be wrong until/unless we get additional support in Docker itself for informing it what it should be putting there).

It's even conceivable to build s390x images on an amd64 daemon, as long as they don't have any RUN lines (ie, FROM scratch, ADD tarball /, etc), or as long as appropriate qemu bits are in the right places somehow.

Here's a more concrete example:

$ manifest-tool inspect trollin/httpd:latest
Name:   trollin/httpd:latest (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:6f43aac63ceb4fbfee6a724f3d8aeec4366bb83cd7ef37a9905ccf2256ba8557
 * Contains 6 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:8f58a3ef340038615498cead8b83fa3b31e4fe5c16961c6c3635e973ac9303ed
1  Mfst Length: 1780
1     Platform:
1           -      OS: linux
1           - OS Vers: 
1           - OS Feat: []
1           -    Arch: amd64
1           - Variant: 
1           - Feature: 
1     # Layers: 7
         layer 1: digest = sha256:9f0706ba7422412cd468804fee456786f88bed94bf9aea6dde2a47f770d19d27
         layer 2: digest = sha256:47bacf36113fe830a77280f81a8203a228fa4bb4536f145a333709b9da0f7cf7
         layer 3: digest = sha256:56798d8e5a3081371b84de38415c30c0f1ae034e4d57d82383c468a95521fc53
         layer 4: digest = sha256:94b25413538ac04aa893f729c216553d9c86b0ea69318cfe903e16eb078492b7
         layer 5: digest = sha256:97d879f4e260dd8d46a4eaf0c29f253a501fa82337840c30a6274de23227ab6e
         layer 6: digest = sha256:2a4f7e960a3e3b32308101dec9c1525b961a242dca799737d2ad0c3e19350b42
         layer 7: digest = sha256:12f5eb5312902eedee33d3b8d0f2abc7750f3ae7e32733b26547371f8eb6b034

2    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
2       Digest: sha256:90fb0d45f7267ac96523f2030ab842d61c77c65999024b5f67771f599ae4bfa8
2  Mfst Length: 1780
2     Platform:
2           -      OS: linux
2           - OS Vers: 
2           - OS Feat: []
2           -    Arch: arm
2           - Variant: v7
2           - Feature: 
2     # Layers: 7
         layer 1: digest = sha256:72c70f9f7d679945bc71d954dc0c7de236e0067af495d09e9bea24f497cc79b7
         layer 2: digest = sha256:c3529be5f104abaa0a6777ce5aa3d62a14d637886a4eebc1fc8ed730c4009654
         layer 3: digest = sha256:d8da62524d5de9ecb71f0046c66d3a291e6b2f81496a07d28e30b4e81f04ac51
         layer 4: digest = sha256:7d7da9871d55dac5ea74a2b5f684f5c94bb5ca9b51ff5f2a81330facfcf67122
         layer 5: digest = sha256:c2c771d498cd865c5dff06aa02340b0c859cebcbe6a88246c7e87e287ab70c48
         layer 6: digest = sha256:de5a89172d87cbecee5e7924eb3c003458ad751d51e25eaed5d8a89912ef3ccb
         layer 7: digest = sha256:a922909dfad204174854c24774fc54c9016d96d587480b6d40c0c56ae73612a7

3    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
3       Digest: sha256:21920abf229fd278a820287cb8fbe9db206ecbbb4b5bd8a34aab0af8b9c0f470
3  Mfst Length: 1780
3     Platform:
3           -      OS: linux
3           - OS Vers: 
3           - OS Feat: []
3           -    Arch: arm64
3           - Variant: v8
3           - Feature: 
3     # Layers: 7
         layer 1: digest = sha256:0dcb325d306f51780dbde5818f38ed8b0e8f478b56132f46bb5c30385f86eaa6
         layer 2: digest = sha256:0a2d58b6ee283f9f76e8f06d3fac65e85b9e289df29ca44dc9314586e055cecc
         layer 3: digest = sha256:eac331a2af3233030fcf1ad5300f9684767ebc08b8447e52b00783cb43de360a
         layer 4: digest = sha256:b382ea768f87d2a04cac208183cd6ee2b487748fb7fe33a9f0e6c4ab0912a029
         layer 5: digest = sha256:313fc6a3542b845ea9d51e727b1b0270de45d71e58a4cbd5e68c427761ce8107
         layer 6: digest = sha256:1a9a6cd7e8894b6345a3d83a4cefba76609c8e096d5163a67c0dee33c7927ffb
         layer 7: digest = sha256:26af40e76c32da3f9b7eac5f62d73ee6d51c30f0b257e37596da4bcbb7e34877

4    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
4       Digest: sha256:261d6c4753982e203308fc5c637a74bdcd986a2edbab7d8aac39a391cae7a683
4  Mfst Length: 1780
4     Platform:
4           -      OS: linux
4           - OS Vers: 
4           - OS Feat: []
4           -    Arch: 386
4           - Variant: 
4           - Feature: 
4     # Layers: 7
         layer 1: digest = sha256:2fa359c89a0e952ec2fe14e3c584ee13d6ec919c73a7dcac34ba320a459e2a62
         layer 2: digest = sha256:19378a3c2be7bef3f223e1ca4a0391c966e6330d8692ed17042471f25509713b
         layer 3: digest = sha256:9e17073700efe44252f1c00a14cf28030458d929d0de85f4873189e23b218530
         layer 4: digest = sha256:38c6ce3cfa9a4835b70425932f1da52dc06e8416369192b5b2fd0768956b907d
         layer 5: digest = sha256:23815ae9a9b769433bf361d90f0654af265d56f7792633aaae368cc17d8af505
         layer 6: digest = sha256:7951cc644f4c9b2aa30c2a8653ce4d2b7ac8390fb72dab2e6f5d3f522bbe738d
         layer 7: digest = sha256:381d02ad82455287029be0a8cf1a1cae9915c26289bac3c24ed7636ba77c6795

5    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
5       Digest: sha256:343ea34611a882700bf162ad929c42259446088d9933991addc64bc90af8f17e
5  Mfst Length: 1780
5     Platform:
5           -      OS: linux
5           - OS Vers: 
5           - OS Feat: []
5           -    Arch: ppc64le
5           - Variant: 
5           - Feature: 
5     # Layers: 7
         layer 1: digest = sha256:a5561821dba4ceb47be1d2f5f108a24b391df9d6a3a764d2c04ea8ac29410625
         layer 2: digest = sha256:39c4390b614987e9576d36928eb4d12a6d5e160fb83b5095198efd400b6b8c5b
         layer 3: digest = sha256:d14c381984db757a959e0f35eb170b4863f5c5e8570702bb7529f4f88998f73a
         layer 4: digest = sha256:90d23acf5ff39880a4a787a1d41e6bc4b204a7e01fc2e60183892c20aa60c5c4
         layer 5: digest = sha256:77e6d79cd73f74c4071006e03344deccab113906c139d8755461d4fd72604b1e
         layer 6: digest = sha256:e3649c2d25a8bf9910fe5f9f940924ef00277ca9a003c5fcaaeb487f116def62
         layer 7: digest = sha256:67e875c10908fb1f66c807700e155e51ccc053f596e93f0d0f1aff26240aa46f

6    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
6       Digest: sha256:3c121d29d5f0c2bb4fdc54fa2e206bb9d6a4b751e29de571b4ef5af7b80c20f6
6  Mfst Length: 1780
6     Platform:
6           -      OS: linux
6           - OS Vers: 
6           - OS Feat: []
6           -    Arch: s390x
6           - Variant: 
6           - Feature: 
6     # Layers: 7
         layer 1: digest = sha256:29420dd727d39cbedfb85562111f49e24b0b96adda04de4663d2099fbbf4f993
         layer 2: digest = sha256:0eb60e029192948c200dc12526bcb19c36747de3cedae3822c3009d6250b95b9
         layer 3: digest = sha256:f2cee80bd07440bd13eb077947699252cbfba9b263af2a43bd94f31378a5c35c
         layer 4: digest = sha256:b628a0b163be3123753102d5269918189e0ff4f2cecc25b8ebb685a7e292435e
         layer 5: digest = sha256:9c67f623c538f560836daac0ea7ab1ffa883c09b87ae7b426fed7e4b828e466b
         layer 6: digest = sha256:0bf51bf880e17e555e66c3af906cd603deae0af6095d1816ffb364b9d16e426d
         layer 7: digest = sha256:16c1cb76b65df18dfffd6e1f23e843c3ddebd950f5e8ea24fd98be6c8b5997ad

(which was built by combining amd64/httpd:latest, arm32v7/httpd:latest, etc)

The info for busybox:latest is likely even better, since it'll have several arm variants:

$ manifest-tool inspect trollin/busybox:latest | grep -A7 'Platform:'
1     Platform:
1           -      OS: linux
1           - OS Vers: 
1           - OS Feat: []
1           -    Arch: amd64
1           - Variant: 
1           - Feature: 
1     # Layers: 1
--
2     Platform:
2           -      OS: linux
2           - OS Vers: 
2           - OS Feat: []
2           -    Arch: arm
2           - Variant: v5
2           - Feature: 
2     # Layers: 1
--
3     Platform:
3           -      OS: linux
3           - OS Vers: 
3           - OS Feat: []
3           -    Arch: arm
3           - Variant: v6
3           - Feature: 
3     # Layers: 1
--
4     Platform:
4           -      OS: linux
4           - OS Vers: 
4           - OS Feat: []
4           -    Arch: arm
4           - Variant: v7
4           - Feature: 
4     # Layers: 1
--
5     Platform:
5           -      OS: linux
5           - OS Vers: 
5           - OS Feat: []
5           -    Arch: arm64
5           - Variant: v8
5           - Feature: 
5     # Layers: 1
--
6     Platform:
6           -      OS: linux
6           - OS Vers: 
6           - OS Feat: []
6           -    Arch: 386
6           - Variant: 
6           - Feature: 
6     # Layers: 1
--
7     Platform:
7           -      OS: linux
7           - OS Vers: 
7           - OS Feat: []
7           -    Arch: ppc64le
7           - Variant: 
7           - Feature: 
7     # Layers: 1
--
8     Platform:
8           -      OS: linux
8           - OS Vers: 
8           - OS Feat: []
8           -    Arch: s390x
8           - Variant: 
8           - Feature: 
8     # Layers: 1

tianon · 2017-07-05T23:04:04Z

For building the individual architecture images, I'm not doing anything special or custom beyond docker build, docker tag, docker push.

stevvooe · 2017-07-06T00:23:20Z

Indeed -- I'm 100% with you on GOARCH being insufficient

For arm, I should have qualified. All other architectures have sufficient information.

We'll either need this on the manifest or all the way down on the config (not sure if we really want to take that on). Seems, as is, there is no way to resolve the arm properties from the config itself and that the contents of Config.Architecture is generally dubious (at least for cross-builds).

Do we need to file a bug with OCI image-spec?

stevvooe · 2017-07-07T17:51:31Z

@tianon ^

As far as a solution, I think we should remove the automatic constraint on architecture unless reporting on a multi-platform image. That will allow swarm to meet the image matching use case for windows/linux.

tianon · 2017-07-07T17:57:47Z

I think that makes sense, especially since if it's not a manifest list, the notion of it being "multiarch" is kind of bogus anyhow (since the arch information is likely also bogus, or at least misleading). The OS field might be accurate, but it's hard to say in that case. These architecture constraints can be directly added by hand, right?

As for filing a bug with the image-spec, I think that makes sense, but I didn't want to try and rock that boat pre-1.0 (since as shown by the commit I linked, it's a real old boat), and hadn't looked to see whether the image config itself was covered by the spec (but it makes sense for it to be). 😇

tianon · 2017-07-07T17:58:46Z

(I have to confess that I haven't tried building FROM scratch on Windows to see if I could cross-build a Linux image that way. 😅)

stevvooe · 2017-07-07T18:06:46Z

@tianon The image config is covered by the spec, unfortunately. For the cases where cross builds are happening, the image will have to be modified post-build. In the case where we just want to route a workload a host that can run that image, we likely don't have enough information, as that will have to be wrapped in a manifest list (index) to provide the actual platform information in the ARM case.

broersa · 2017-07-07T18:49:42Z

I ran into the same issue today. I run a manager on amd64 and have a node which is armv6l. It's a raspberrypi model b+. I can run the image on the node itself, but when I want to run it as a swarm service it says:
"Message": "no suitable node (unsupported platform on 4 nodes)"

Btw. I also have two amd64 nodes.

It seems it can't map the arm image on armv6l in the swarm scheduler.

StefanScherer · 2017-07-09T20:56:43Z

The Scaleway C1 machine shows armv7l, as well as Raspberry Pi 2/3
Raspberry Pi 0/1 show armv6l
Docker images have arm as architecture.

So probably the platformEqual() should have the same normalization as for x86_64 -> amd64:

	switch imgPlatform.Architecture {
	case "x86_64":
		imgPlatform.Architecture = "amd64"
	case "armv6l", "armv7l":
		imgPlatform.Architecture = "arm"
	case "aarch64":
		imgPlatform.Architecture = "arm64"			
	}
	switch nodePlatform.Architecture {
	case "x86_64":
		nodePlatform.Architecture = "amd64"
	case "armv6l", "armv7l":
		nodePlatform.Architecture = "arm"
	case "aarch64":
		nodePlatform.Architecture = "arm64"			
	}

Something like this?

stevvooe · 2017-07-31T20:52:13Z

@alexellis I agree. This is not a good situation and I am worried that having these constraints will create mutual incompatibilities.

Could you see if moby/moby#34021 will work?

Toshik · 2017-08-20T19:09:35Z

I have all nodes Orange Pi PC with platform linux armv7l and have to use --no-resolve-image, otherwise swarm services are in constant pending state.

Docker: 17.06.1-ce

alexellis · 2017-08-20T19:10:55Z

@stevvooe the PR by @nishanttotla fixes the issue - I've spent the whole day testing it... :-/

Is the Docker team aware that fast ARMv7 units are available on Scaleway's infrastructure for next to nothing? It could help with future CI and testing.

moby/moby#34021 (comment)

@Toshik I don't believe your work-around is good for stacks?

Toshik · 2017-08-20T19:16:30Z

@alexellis, no, it is ugly workaround. And I had to recreate all services with just docker service create command :(

alexellis · 2017-08-20T19:18:24Z

Just take this PR it fixes everything.

Reverts raspbian changes back to https://apt.dockerproject.org, due to issues with swarm in the newest release 17.07.0-ce related to issue moby/swarmkit#2294. This will stay in place for raspbian until that issue is resolved. Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>

nishanttotla · 2017-09-20T19:26:19Z

Closing since moby/moby#34021 was merged. A longer term resolution of ARM variants will take time, but it is no longer the case that tasks don't run, so we can consider this issue resolved.

6be709c0 · 2017-10-02T22:29:47Z

Same problem on 17.09, have to rollback on 17.06

nishanttotla · 2017-10-02T22:48:40Z

@MLescaudron this should be fixed now. Can you share the output of docker info on one of your worker nodes that has ARM, and the output of docker service inspect <service name> for the service that fails to run?

6be709c0 · 2017-10-02T22:51:20Z

Docker info (previously I was in 17.09.2-ce)

Containers: 21
 Running: 11
 Paused: 0
 Stopped: 10
Images: 14
Server Version: 17.06.2-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: vjh2cujrci6sjus9lyhjm87po
 Is Manager: true
 ClusterID: b70lb8nciap52uncs76xnfkfl
 Managers: 1
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Root Rotation In Progress: false
 Node Address: 145.239.13.20
 Manager Addresses:
  145.239.13.20:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-81-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 14.34GiB
Name: ovh-prod-1
ID: TXNV:VBKT:MFJD:4K7W:G3JH:WDB7:LZZF:3SWD:LVQO:6SFP:WOPE:AN7E
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: nelioapp
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

docker service inspect

[
    {
        "ID": "0rvq0tmowj9nfn2gjgp0hzmyd",
        "Version": {
            "Index": 69574
        },
        "CreatedAt": "2017-09-26T22:31:04.605418562Z",
        "UpdatedAt": "2017-10-02T22:13:46.09779364Z",
        "Spec": {
            "Name": "tech_blog",
            "Labels": {
                "com.docker.stack.image": "ghost:alpine",
                "com.docker.stack.namespace": "tech"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "ghost:alpine@sha256:b7c4fcb9e78518ea85fcba80969f445301b38f76833a6668eb268da7b6e7f3a4",
                    "Labels": {
                        "com.docker.stack.namespace": "tech"
                    },
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/home/nelio/tech.nelio.io/",
                            "Target": "/var/lib/ghost/content"
                        }
                    ],
                    "StopGracePeriod": 10000000000,
                    "DNSConfig": {}
                },
                "Resources": {},
                "RestartPolicy": {
                    "Condition": "any",
                    "Delay": 5000000000,
                    "MaxAttempts": 0
                },
                "Placement": {
                    "Constraints": [
                        "node.role == manager"
                    ],
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "omgpnj44fbx9y7khp6pxmptq7",
                        "Aliases": [
                            "blog"
                        ]
                    }
                ],
                "LogDriver": {
                    "Options": {
                        "max-size": "50m"
                    }
                },
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "UpdateConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "RollbackConfig": {
                "Parallelism": 1,
                "FailureAction": "pause",
                "Monitor": 5000000000,
                "MaxFailureRatio": 0,
                "Order": "stop-first"
            },
            "EndpointSpec": {
                "Mode": "vip"
            }
        },
        "PreviousSpec": {
            "Name": "tech_blog",
            "Labels": {
                "com.docker.stack.image": "ghost:alpine",
                "com.docker.stack.namespace": "tech"
            },
            "TaskTemplate": {
                "ContainerSpec": {
                    "Image": "ghost:alpine@sha256:b7c4fcb9e78518ea85fcba80969f445301b38f76833a6668eb268da7b6e7f3a4",
                    "Labels": {
                        "com.docker.stack.namespace": "tech"
                    },
                    "Privileges": {
                        "CredentialSpec": null,
                        "SELinuxContext": null
                    },
                    "Mounts": [
                        {
                            "Type": "bind",
                            "Source": "/home/nelio/tech.nelio.io/",
                            "Target": "/var/lib/ghost/content"
                        }
                    ]
                },
                "Resources": {},
                "Placement": {
                    "Constraints": [
                        "node.role == manager"
                    ],
                    "Platforms": [
                        {
                            "Architecture": "amd64",
                            "OS": "linux"
                        }
                    ]
                },
                "Networks": [
                    {
                        "Target": "omgpnj44fbx9y7khp6pxmptq7",
                        "Aliases": [
                            "blog"
                        ]
                    }
                ],
                "LogDriver": {
                    "Options": {
                        "max-size": "50m"
                    }
                },
                "ForceUpdate": 0,
                "Runtime": "container"
            },
            "Mode": {
                "Replicated": {
                    "Replicas": 1
                }
            },
            "EndpointSpec": {
                "Mode": "vip"
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "vip"
            },
            "VirtualIPs": [
                {
                    "NetworkID": "omgpnj44fbx9y7khp6pxmptq7",
                    "Addr": "10.0.0.5/24"
                }
            ]
        }
    }
]

nishanttotla · 2017-10-02T23:01:13Z

@MLescaudron your node reports linux/x86_64 and your service placement constraints report linux/amd64. This is a compatible combination, and your service should run. There maybe another reason why it's failing.

6be709c0 · 2017-10-02T23:41:52Z

Ok, so I have unistall and reinstall docker-ce on each node.
When I launch a service, docker ps return an empty response and the state is on pending mode.
Same problem as @trunet, except don't have the architecture problem

beatkyo · 2017-10-03T23:06:36Z

Same here with 17.09

Docker info: linux aarch64
Docker inspect task: linux arm64
Docker inspect container: linux arm64

Task status: "State": "pending", "Message": "unsupported platform on 1 node"

nishanttotla · 2017-10-03T23:13:41Z

It seems that moby/moby#34021 should have excluded more cases in addition to just "arm", like "arm64".

cc @thaJeztah @andrewhsu

alexellis · 2017-10-09T20:40:49Z

This appears to be broken again on ARM64 :-( in particular I have a weird situation where it will work on Dieter's ARM64 RPi image but not on Packet's Qualcomm machine - both are "aarch64" in uname.

Packet:

root@48-cores:~# docker service create --name func_prometheus  luxas/prometheus-arm64:v1.5.2
DEBU[2017-10-09T20:36:21.741818950Z] Calling GET /_ping                           
DEBU[2017-10-09T20:36:21.742930000Z] Calling GET /_ping                           
DEBU[2017-10-09T20:36:21.746669250Z] Calling GET /v1.32/distribution/luxas/prometheus-arm64:v1.5.2/json 
DEBU[2017-10-09T20:36:22.147220750Z] Calling POST /v1.32/services/create          
DEBU[2017-10-09T20:36:22.147412700Z] form data: {"EndpointSpec":{"Mode":"vip"},"Labels":{},"Mode":{"Replicated":{}},"Name":"func_prometheus","TaskTemplate":{"ContainerSpec":{"DNSConfig":{},"Image":"luxas/prometheus-arm64:v1.5.2@sha256:7011cf4a94d350cc6719b7a87eaa2cd7cbcc19bcfcf081eae4b9b6ae5f9e00d0"},"ForceUpdate":0,"Placement":{"Platforms":[{"Architecture":"amd64","OS":"linux"}]},"Resources":{"Limits":{},"Reservations":{}}}} 
DEBU[2017-10-09T20:36:22.161471000Z] Service hzxpjiytu97jfz931oduom8o8 was scaled up from 0 to 1 instances  module=node node.id=xl2vxso5gh504rosy2u9nyqkc
hzxpjiytu97jfz931oduom8o8
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
root@48-cores:~# DEBU[2017-10-09T20:36:22.246894100Z] no suitable node available for task           module=node node.id=xl2vxso5gh504rosy2u9nyqkc task.id=yj0vos4kyogoklebm3dvw50xb
DEBU[2017-10-09T20:36:22.246938550Z] no suitable node available for task           module=node node.id=xl2vxso5gh504rosy2u9nyqkc task.id=syppv5tnsr77hzst6tahy0imq
DEBU[2017-10-09T20:36:22.336576750Z] no suitable node available for task           module=node node.id=xl2vxso5gh504rosy2u9nyqkc task.id=syppv5tnsr77hzst6tahy0imq
DEBU[2017-10-09T20:36:22.336625100Z] no suitable node available for task           module=node node.id=xl2vxso5gh504rosy2u9nyqkc task.id=yj0vos4kyogoklebm3dvw50xb
docker version
DEBU[2017-10-09T20:36:36.377204000Z] Calling GET /_ping                           
DEBU[2017-10-09T20:36:36.378193650Z] Calling GET /v1.32/version                   
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:10 2017
 OS/Arch:      linux/arm64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:38:22 2017
 OS/Arch:      linux/arm64
 Experimental: false
root@48-cores:~# DEBU[0173] containerd: process exited                    id=0574e809eb8b8787684c3e76649466cdc7ea05c1d18d56be032fbf2c91fa9a5d pid=init status=255 systemPid=7790
DEBU[2017-10-09T20:38:15.945790550Z] libcontainerd: received containerd event: &types.Event{Type:"exit", Id:"0574e809eb8b8787684c3e76649466cdc7ea05c1d18d56be032fbf2c91fa9a5d", Status:0xff, Pid:"init", Timestamp:(*timestamp.Timestamp)(0x442166c150)} 
DEBU[0173] containerd: process exited                    id=3a80448475e5b29e4a7ecb8b1dd0f11b1baaffb731871a6d02386be50fd75ca2 pid=init status=255 systemPid=8386
DEBU[2017-10-09T20:38:15.965611750Z] libcontainerd: received containerd event: &types.Event{Type:"exit", Id:"3a80448475e5b29e4a7ecb8b1dd0f11b1baaffb731871a6d02386be50fd75ca2", Status:0xff, Pid:"init", Timestamp:(*timestamp.Timestamp)(0x442166c410)} 

DEBU[2017-10-09T20:36:36.378193650Z] Calling GET /v1.32/version                   
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:10 2017
 OS/Arch:      linux/arm64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:38:22 2017
 OS/Arch:      linux/arm64
 Experimental: false```

stevvooe · 2017-10-09T22:10:13Z

@alexellis Looking at the patch that should have fixed this, it seems like this case is not covered. Although, it seems like the placement constraint clearly reads amd64, from the log message.

vielmetti · 2017-10-09T22:58:25Z

OK, there's something funny going on here.

ed$ docker run --rm mplatform/mquery luxas/prometheus-arm64:v1.5.2
Manifest List: No
 Supports: amd64/linux

It looks like @luxas has a prometheus-arm64 image that advertises that it supports amd64/linux.

stevvooe · 2017-10-09T23:03:53Z

Looking at the configuration, the architecture is reported as amd64:

$ ctr fetch-object docker.io/luxas/prometheus-arm64@sha256:7663b9e02bac80a6b5e48ffb0602c2d515261dc2afcf10cb819fc563b8bc4126 | jq .
INFO[0000] resolving                                     ref="docker.io/luxas/prometheus-arm64@sha256:7663b9e02bac80a6b5e48ffb0602c2d515261dc2afcf10cb819fc563b8bc4126"
INFO[0001] fetching                                      ref="docker.io/luxas/prometheus-arm64@sha256:7663b9e02bac80a6b5e48ffb0602c2d515261dc2afcf10cb819fc563b8bc4126"
{
  "architecture": "amd64",
...

Were these "cross-built" binaries?

vielmetti · 2017-10-09T23:08:43Z

Looks like @luxas has a multiarch image, cf

ed$ docker run --rm mplatform/mquery luxas/prometheus:v2.0.0-rc.0
Manifest List: Yes
Supported platforms:
 - amd64/linux
 - arm/linux (variant: undefined)
 - arm64/linux (variant: undefined)

also

ed$ docker run --rm mplatform/mquery luxas/prometheus:v1.7.1
Manifest List: Yes
Supported platforms:
 - amd64/linux
 - arm/linux (variant: undefined)
 - arm64/linux (variant: undefined)

luxas · 2017-10-10T17:42:28Z

@vielmetti @stevvooe Yes, I'm crossbuilding everything. That's by far the best way to do things IMO. Don't know if docker lets me set the os/arch (I don't think so?)

stevvooe · 2017-10-11T00:36:19Z

@luxas Cross-building won't work unless you re-pack into manifest list or re-write the configuration with the correct architecture.

@vielmetti There is no variant field in the base image config, so it won't be defined. We can add one there if this is a huge problem.

luxas · 2017-10-11T17:26:57Z

@luxas Cross-building won't work unless you re-pack into manifest list or re-write the configuration with the correct architecture.

@stevvooe Can you point to docs how to mutate OS/arch for a given (non-manifest-list) image? e.g. for luxas/prometheus-arm64

vielmetti · 2017-10-13T19:02:26Z

This patch #2411 addresses the issue.

trunet mentioned this issue Jul 2, 2017

Platform Filter for Service Scheduling #1981

Merged

nishanttotla self-assigned this Jul 2, 2017

nishanttotla mentioned this issue Jul 7, 2017

Clear Architecture field in platform constraint for arm architectures moby/moby#34021

Merged

firecyberice mentioned this issue Jul 19, 2017

Wrong Image Architecture hypriot/rpi-traefik#29

Closed

thaJeztah added the kind/bug label Aug 1, 2017

seemethere mentioned this issue Aug 29, 2017

Revert raspbian changes docker/docker-install#25

Merged

seemethere mentioned this issue Aug 30, 2017

Update instructions for Raspberry Pi docker/docs#4315

Merged

jcberthon mentioned this issue Sep 17, 2017

manifest list (multiarch) picks wrong arch on ARMv7 moby/moby#34875

Closed

nishanttotla closed this as completed Sep 20, 2017

vielmetti mentioned this issue Oct 9, 2017

"no suitable node available for task" on arm64 due to non-normalized platform name #2401

Closed

17.06 won't deploy stack anymore. no suitable node (unsupported platform on 3 nodes) on armhf docker cluster #2294

17.06 won't deploy stack anymore. no suitable node (unsupported platform on 3 nodes) on armhf docker cluster #2294

Comments

trunet commented Jul 2, 2017

trunet commented Jul 2, 2017

nishanttotla commented Jul 2, 2017

trunet commented Jul 2, 2017

nishanttotla commented Jul 3, 2017

trunet commented Jul 3, 2017 • edited Loading

trunet commented Jul 3, 2017

nishanttotla commented Jul 3, 2017 • edited Loading

trunet commented Jul 3, 2017

trunet commented Jul 3, 2017

nishanttotla commented Jul 3, 2017

thaJeztah commented Jul 3, 2017

nishanttotla commented Jul 5, 2017

stevvooe commented Jul 5, 2017

tianon commented Jul 5, 2017

tianon commented Jul 5, 2017

tianon commented Jul 5, 2017

stevvooe commented Jul 5, 2017

tianon commented Jul 5, 2017

tianon commented Jul 5, 2017

stevvooe commented Jul 6, 2017

stevvooe commented Jul 7, 2017

tianon commented Jul 7, 2017

tianon commented Jul 7, 2017

stevvooe commented Jul 7, 2017

broersa commented Jul 7, 2017

StefanScherer commented Jul 9, 2017

stevvooe commented Jul 31, 2017

Toshik commented Aug 20, 2017 • edited Loading

alexellis commented Aug 20, 2017 • edited Loading

Toshik commented Aug 20, 2017

alexellis commented Aug 20, 2017

nishanttotla commented Sep 20, 2017

6be709c0 commented Oct 2, 2017

nishanttotla commented Oct 2, 2017

6be709c0 commented Oct 2, 2017

nishanttotla commented Oct 2, 2017

6be709c0 commented Oct 2, 2017 • edited Loading

beatkyo commented Oct 3, 2017

nishanttotla commented Oct 3, 2017 • edited Loading

alexellis commented Oct 9, 2017 • edited Loading

stevvooe commented Oct 9, 2017

vielmetti commented Oct 9, 2017

stevvooe commented Oct 9, 2017

vielmetti commented Oct 9, 2017

luxas commented Oct 10, 2017

stevvooe commented Oct 11, 2017

luxas commented Oct 11, 2017

vielmetti commented Oct 13, 2017

trunet commented Jul 3, 2017 •

edited

Loading

nishanttotla commented Jul 3, 2017 •

edited

Loading

Toshik commented Aug 20, 2017 •

edited

Loading

alexellis commented Aug 20, 2017 •

edited

Loading

6be709c0 commented Oct 2, 2017 •

edited

Loading

nishanttotla commented Oct 3, 2017 •

edited

Loading

alexellis commented Oct 9, 2017 •

edited

Loading