Skip to content

Commit

Permalink
Apply CRI to support CRI-O
Browse files Browse the repository at this point in the history
  • Loading branch information
ssup2 committed Sep 20, 2022
1 parent 96cae4c commit 370a6b9
Show file tree
Hide file tree
Showing 13 changed files with 1,491 additions and 380 deletions.
13 changes: 10 additions & 3 deletions Dockerfile-cnsenter
Original file line number Diff line number Diff line change
@@ -1,13 +1,20 @@
# Build cnsenter
FROM golang:1.16 as builder
WORKDIR /workspace

# Build cnsenter
ARG VERSION
COPY . .
RUN CGO_ENABLED=0 GO111MODULE=on go build -a -ldflags="-X 'github.com/ssup2/kpexec/pkg/cmd/cnsenter.version=${VERSION}'" -o cnsenter cmd/cnsenter/main.go

# Download crictl
FROM alpine:3.13.1 as downloader
ENV CRICTL_VERSION v1.24.1
RUN wget https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz
RUN tar zxvf crictl-${CRICTL_VERSION}-linux-amd64.tar.gz -C /usr/local/bin

# Build image
FROM alpine:3.13.1
COPY --from=builder /workspace/cnsenter /usr/bin/cnsenter
COPY --from=builder /workspace/cnsenter /usr/local/bin/cnsenter
COPY --from=downloader /usr/local/bin/crictl /usr/local/bin/crictl

CMD ["cnsenter"]
CMD ["cnsenter"]
11 changes: 9 additions & 2 deletions Dockerfile-cnsenter-tools
Original file line number Diff line number Diff line change
@@ -1,15 +1,22 @@
# Build cnsenter
FROM golang:1.16 as builder
WORKDIR /workspace

# Build cnsenter
ARG VERSION
COPY . .
RUN CGO_ENABLED=0 GO111MODULE=on go build -a -ldflags="-X 'github.com/ssup2/kpexec/pkg/cmd/cnsenter.version=${VERSION}'" -o cnsenter cmd/cnsenter/main.go

# Download crictl
FROM alpine:3.13.1 as downloader
ENV CRICTL_VERSION v1.24.1
RUN wget https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz
RUN tar zxvf crictl-${CRICTL_VERSION}-linux-amd64.tar.gz -C /usr/local/bin

# Build image
# Reference - https://github.com/nicolaka/netshoot
FROM alpine:3.13.1
COPY --from=builder /workspace/cnsenter /usr/bin/cnsenter
COPY --from=builder /workspace/cnsenter /usr/local/bin/cnsenter
COPY --from=downloader /usr/local/bin/crictl /usr/local/bin/crictl
COPY scripts/remount-proc-exec /usr/bin/remount-proc-exec
RUN apk add \
apache2-utils \
Expand Down
20 changes: 12 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

![kpexec Demo](image/kpexec_Demo.gif)

**kpexec** is a K8s cli that runs commands in a container with high privileges. It runs a highly privileged container on the same node as the target container and joins into the namespaces of the target container (IPC, UTS, PID, net, mount). This is useful for debugging where you often need to execute commands with high privileges. Also, kpexec has a **tools mode**, which adds useful debugging tools into the debugged container. The tools mode is useful when there necessary debugging tools are missing in the target container.
**kpexec** is a K8s cli that runs commands in a container with high privileges **without SSH**. It runs a highly privileged container on the same node as the target container and joins into the namespaces of the target container (IPC, UTS, PID, net, mount). This is useful for debugging where you often need to execute commands with high privileges. Also, kpexec has a **tools mode**, which adds useful debugging tools into the debugged container. The tools mode is useful when there necessary debugging tools are missing in the target container.

In contrast, kubectl-exec runs the command with the same privileges as the container. For example, if a container does not have network privileges, the command executed by kubectl-exec also has no network privileges. This makes debugging the pod difficult. If you use kpexec instead of kubectl-exec, you can easily get high privileges for debugging.

Expand All @@ -16,9 +16,9 @@ kpexec now supports the following CPU architectures.

## Check & Install

Since kpexec uses kubectl internally, **kubectl** installation and **kubeconfig** files must be properly configured before using kpexec. Whenever kpexec is executed, kpexec creates a **cnsenter (Container Namespace Enter) pod** to executes cnsenter. cnsenter is a command to exec command in the target container. The cnsenter pod must be created with **hostPID** and **Privileged** Option. Therefore, before using kpexec, you should check if the pod options mentioned are available in your K8s cluster.
Since kpexec uses kubectl internally, **kubectl** installation and **kubeconfig** files must be properly configured before using kpexec. Whenever kpexec is executed, kpexec creates a **cnsenter (Container Namespace Enter) pod** to executes cnsenter. cnsenter is a command to exec command in the target container through **CRI (Container Runtime Interface)**.

Fortunately, in most K8s clusters including managed K8s clusters by public cloud service such as EKS, AKS and GKE, the pod options mentioned available without configuration. Therefore, kpexec can also be used in most K8s clusters without configuration.
The cnsenter pod must be created with **hostPID** and **Privileged** Option. Therefore, before using kpexec, you should check if the pod options mentioned are available in your K8s cluster. Fortunately, in most K8s clusters including managed K8s clusters by public cloud service such as EKS, AKS and GKE, the pod options mentioned available without configuration. Therefore, kpexec can also be used in most K8s clusters without any configuration.

### Download Binary

Expand Down Expand Up @@ -98,12 +98,16 @@ $ kpexec -it mypod -c bash-container -- bash
$ kubectl pexec -it mypod -c bash-container -- bash

# Enable tools mode.
$ kpexec -it -T mypod -c golang-container -- bash
$ kubectl pexec -it -T mypod -c golang-container -- bash
$ kpexec -it -T mypod -c bash-container -- bash
$ kubectl pexec -it -T mypod -c bash-container -- bash

# Set cnsenter pod's image
$ kpexec -it -T --cnsenter-image=ssup2/my-cnsenter-tools:latest mypod -c golang-container -- bash
$ kubectl pexec -it -T --cnsenter-image=ssup2/my-cnsenter-tools:latest mypod -c golang-container -- bash
$ kpexec -it -T --cnsenter-image=ssup2/my-cnsenter-tools:latest mypod -c bash-container -- bash
$ kubectl pexec -it -T --cnsenter-image=ssup2/my-cnsenter-tools:latest mypod -c bash-container -- bash

# Set CRI socket path / containerd socket path
$ kpexec -it -T --cri /run/my/containerd.sock -c bash-container -- bash
$ kubectl pexec -it -T --cri /run/my/containerd.sock -c bash-container -- bash

# kpexec removes the cnsetner pod it created after executing the command.
# If cnsenter pods remain due to external factors, you can remove all remaining cnsenter pods
Expand All @@ -116,7 +120,7 @@ $ kubectl pexec --cnsenter-gc

![kpexec Operation](image/kpexec_Operation.png)

The figure above shows the operation processs of kpexec. At first, kpexec obtains the information of target pod from K8s API Server and finds out which Node the target pod exists in. After that, kpexec creates a cnsenter pod in the node where target pod exists and executes cnsetner. cnsenter gets the target container's pid and root directory information from containerd. Then cnsetner executes the command in the target container based on the obtained information.
The figure above shows the operation processs of kpexec. At first, kpexec obtains the information of target pod from K8s API Server and finds out which Node the target pod exists in. After that, kpexec creates a cnsenter pod in the node where target pod exists and executes cnsetner. cnsenter gets the target container's pid and root directory information from container runtime through CRI (Container Runtime Interface). Then cnsetner executes the command in the target container based on the obtained information.

cnsenter pod uses the below images defaultly. The cnsenter pod image can be set with the '--cnsetner-image' option.
* default mode - ssup2/cnsenter:[kpexec version]
Expand Down
31 changes: 12 additions & 19 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,18 @@ module github.com/ssup2/kpexec
go 1.16

require (
github.com/Microsoft/hcsshim v0.8.7 // indirect
github.com/containerd/cgroups v0.0.0-20210114181951-8a68de567b68 // indirect
github.com/containerd/containerd v1.4.3
github.com/containerd/continuity v0.0.0-20200107194136-26c1120b8d41 // indirect
github.com/containerd/fifo v0.0.0-20191213151349-ff969a566b00 // indirect
github.com/containerd/ttrpc v0.0.0-20200121165050-0be804eadb15 // indirect
github.com/containerd/typeurl v0.0.0-20200205145503-b45ef1f1f737 // indirect
github.com/docker/go-events v0.0.0-20190806004212-e31b211e4f1c // indirect
github.com/gogo/googleapis v1.3.2 // indirect
github.com/opencontainers/image-spec v1.0.1 // indirect
github.com/opencontainers/runc v0.1.1 // indirect
github.com/opencontainers/selinux v1.8.0 // indirect
github.com/spf13/cobra v1.1.1
github.com/syndtr/gocapability v0.0.0-20180916011248-d98352740cb2 // indirect
go.etcd.io/bbolt v1.3.3 // indirect
gotest.tools/v3 v3.0.3 // indirect
k8s.io/api v0.18.15
k8s.io/apimachinery v0.18.15
k8s.io/client-go v0.18.15
github.com/containerd/containerd v1.6.8
github.com/google/go-cmp v0.5.8 // indirect
github.com/spf13/cobra v1.1.3
github.com/stretchr/testify v1.7.1 // indirect
github.com/tidwall/gjson v1.14.3
golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f // indirect
google.golang.org/protobuf v1.28.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/api v0.22.5
k8s.io/apimachinery v0.22.5
k8s.io/client-go v0.22.5
)

replace github.com/docker/distribution => github.com/docker/distribution v0.0.0-20191216044856-a8371794149d
Loading

0 comments on commit 370a6b9

Please sign in to comment.