[WIP] Cli: Support --cap-add, --cap-drop and --privileged on services

[olljanat]

- What I did

• Added --cap-add, --cap-drop and --privileged flags to docker service create
• Added --cap-add and --cap-drop to docker service update

Closes moby/moby#25885 moby/moby#24862 moby/swarmkit#1030

- How I did it
Included logic which converts -cap-add, --cap-drop and --privileged flags to exact capabilities list

- How to verify it
Create service based on default capabilities with minor adjustments:

docker service create --name test --cap-add "NET_ADMIN" --cap-drop "CAP_CHOWN" ollijanatuinen/capsh

Update capabilities for existing service:

docker service update --cap-drop "CAP_NET_RAW" --cap-add "SYS_NICE" test

Create service with all capabilities and drop some capabilities from it:

docker service create --name test2 --privileged ollijanatuinen/capsh
docker service update --cap-drop "CAP_AUDIT_READ" --cap-drop "CAP_BLOCK_SUSPEND" test2

- A picture of a cute animal (not mandatory but encouraged)

WIP because based on #1940 and it needs to be merged first.

[fmiguelez]

Hi @olljanat

I need this feature to get "jmap" working on my container using image "gettyimages/spark:2.2.1-hadoop-2.7" and deployed using docker swarm.

I followed your suggestion by downloading your docker-cli program from https://github.com/olljanat/cli/releases/download/beta1/docker and latest (nightly) versions of the rest.

When I try to run "service update" it seems to work but it does not enable required capability on my service (SYS_PTRACE):

appusr@od-node0:~/docker$ /usr/bin/docker service update --cap-add SYS_PTRACE od_spark-worker od_spark-worker overall progress: 3 out of 3 tasks 1/3: running [==================================================>] 2/3: running [==================================================>] 3/3: running [==================================================>] verify: Service converged appusr@od-node0:~/docker$ docker exec -it od_spark-worker.1.wrt54f0xdtepx3wdim8yber8u bash root@spark-worker-1:/usr/spark-2.2.1# cat /proc/sys/kernel/yama/ptrace_scope 1 root@spark-worker-1:/usr/spark-2.2.1# echo 0 > /proc/sys/kernel/yama/ptrace_scope bash: /proc/sys/kernel/yama/ptrace_scope: Read-only file system
I have also tried adding "cap_add" and "privileged: true" to service definition in stack with same result.

These are my versions (I have only upgraded one node with docker versions):

---

appusr@od-node0:~/docker$ docker version
Client:
Version: 19.09.0-dev
API version: 1.41
Go version: go1.12.12
Git commit: 341dadf
Built: Sat Nov 23 11:12:27 2019
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 0.0.0-20190727010531-15bdbd76a5
API version: 1.41 (minimum version 1.12)
Go version: go1.12.6
Git commit: 15bdbd7
Built: Sun Jul 28 07:35:46 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683

---

Any suggestions?

[olljanat]

@fmiguelez you can use my ollijanatuinen/capsh image to see that capabilities are really updated. It run command capsh --print inside of container and you can see them with command docker logs <container id>

Then comes to your "jmap" issue check first that you get it working with docker run command as it can be that you need also some other security settings (most probably you need provide some values for --sysctl ). If that does not help then ping me on Docker community Slack (as that should like a issue which is not directly related to this PR and I would like to keep this discussion clear for everyone who is following it).

[information-security]

@olljanat Hi
I need to load a kernel module so I need privileged flag in swarm mode. For that I have an image that is working fine with docker run -it --privileged --network host --name stp_module 127.0.0.1:2000/inap:latest. I have replaced the /usr/bin/dockerd and /usr/bin/docker with those you suggested. Albeit the stack is deployed without any complaints about --privileged flag, it has no effect on actual container.

Here are some info that might help investigate the problem:

1. Your capsh output no matter what combination of cap_add or cap_drop or privileged flags I use (it is always the same):

Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=0(root)
gid=0(root)
groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)

2. My docker-compose.yml

  docker service create \
   --constraint 'node.hostname == stp2' \
   --name stp_module \
   --privileged \
   --replicas 1 \
   --network host \
   --mount type=bind,src=/home/utob_inap.log,dst=/var/log/uinap.log \
   127.0.0.1:2000/inap:latest

Note 1: --network host is also obligatory in my case because my kernel module is talking to a network-like device.
Note 2: I have also tested with stack deploy command and same results.

3. My container error log upon loading kernel module:
   modprobe: ERROR: could not insert 'utob_inap': Operation not permitted

4. docker container inspect's output in summary:

            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Privileged": false,

5. docker version of the node with hostname as stp2:

Client:
 Version:           19.09.0-dev
 API version:       1.41
 Go version:        go1.12.12
 Git commit:        341dadfb
 Built:             Sat Nov 23 11:12:27 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          master-dockerproject-2019-12-11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.4
  Git commit:       1347481
  Built:            Wed Dec 11 23:58:15 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

6. My setup is a two node swarm having both as managers. dockerd and docker on one of the nodes are not changed but I made sure that service is deployed on the node with stp2 hostname which has proper versions.

[olljanat]

@information-security if you have multiple swarm nodes then you must update dockerd to all of them. Other why you cannot use API version 1.41 which is needed by this. Latest code even give warnings about it but you don't see it because that test binary was built before I added that logic.

If that does not help then plz ping me on Docker community Slack. I want keep this discussion clear for maintainers who do actual code review.