Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: manual of container labels and annotaions under cri #2672

Closed
wants to merge 1 commit into from
Closed

docs: manual of container labels and annotaions under cri #2672

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions cri/annotations/annotations.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,10 @@ const (

// PidsLimitExtendAnnotation is the extend annotation of pids limit
PidsLimitExtendAnnotation = "io.alibaba.pouch.resources.pids-limit"

// PassthruKey specify whether an interface is pass through to qemu
PassthruKey = "io.alibaba.pouch.vm.passthru"

// PassthruIP is the IP for container
PassthruIP = "io.alibaba.pouch.vm.passthru.ip"
)
14 changes: 4 additions & 10 deletions cri/v1alpha2/cri.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ const (
pouchRuntimeName = "pouch"
kubeletRuntimeAPIVersion = "0.1.0"

// kubePrefix is used to idenfify the containers/sandboxes on the node managed by kubelet.
// kubePrefix is used to identify the containers/sandboxes on the node managed by kubelet.
kubePrefix = "k8s"

// annotationPrefix is used to distinguish between annotations and labels.
Expand Down Expand Up @@ -73,12 +73,6 @@ const (

// networkNotReadyReason is the reason reported when network is not ready.
networkNotReadyReason = "NetworkPluginNotReady"

// passthruKey to specify whether a interface is passthru to qemu
passthruKey = "io.alibaba.pouch.vm.passthru"

// passthruIP is the IP for container
passthruIP = "io.alibaba.pouch.vm.passthru.ip"
)

var (
Expand Down Expand Up @@ -578,8 +572,8 @@ func (c *CriManager) PodSandboxStatus(ctx context.Context, r *runtime.PodSandbox
}
}

if v, exist := annotations[passthruKey]; exist && v == "true" {
ip = annotations[passthruIP]
if v, exist := annotations[anno.PassthruKey]; exist && v == "true" {
ip = annotations[anno.PassthruIP]
}

status := &runtime.PodSandboxStatus{
Expand Down Expand Up @@ -674,7 +668,7 @@ func (c *CriManager) CreateContainer(ctx context.Context, r *runtime.CreateConta
sandboxMeta.NetNS = containerNetns(sandbox)

labels := makeLabels(config.GetLabels(), config.GetAnnotations())
// Apply the container type lable.
// Apply the container type label.
labels[containerTypeLabelKey] = containerTypeLabelContainer
// Write the sandbox ID in the labels.
labels[sandboxIDLabelKey] = podSandboxID
Expand Down
55 changes: 40 additions & 15 deletions docs/kubernetes/pouch_cri_annotations_supported.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,18 @@

* [Overview](#overview "Overview")
* [The Annotations Supported](#the-annotations-supported "The Annotations Supported")
* [Make runtime choosing supported](#make-runtime-choosing-supported "Make runtime choosing supported")
* [Make lxcfs configurable supported](#make-lxcfs-configurable-supported "Make lxcfs configurable supported")
* [Pull Request](#pull-request "Pull Request")
* [Runtime choosing](#runtime-choosing "Runtime choosing")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use RuntimeClass in Kubernetes to support runv/Kata which is described in #2314? @starnop @zhuangqh

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use annotations to support runtime choosing for legacy kubernetes. RuntimeHandler of kubernetes 1.12 is also supported

Copy link
Collaborator

@allencloud allencloud Mar 14, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that, I think we should add detailed instructions in this document.

< 1.12, suggest using annotation;
>= 1.12, use runtime class

In addition, we should have a plan whether to deprecate the annotation for future.

* [LXCFS switcher](#lxcfs-switcher "LXCFS switcher")
* [VM passthrough config](#vm-passthrough "VM passthrough config")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we need set correctly markdown anchor point here and follows, e.g. VM passthrough config (#vm-passthrough-config "VM passthrough config") 👍

* [The container labels rule](#container-labels-rule "The container labels rule")
* [Used by PouchContainer implementation](#label-used-by-pouch-container "Used by PouchContainer implementation")
* [Generated from kubernetes spec](#label-generated-from-kubernetes-spec "Generated from kubernetes spec")

## Overview

Currently, PouchContainer has lots of advantages over other container runtimes, such as:

* resource review isolation via lxcfs
* resource view isolation via LXCFS
* runtime choosing for runc-based container or runv-based containers
* and so on

Expand All @@ -28,12 +31,14 @@ So, we need to accomplish the following things:

| Requirement | Field definition | Supported Kubernetes Version | Pull Request |
|-----------------------------------|----------------------------------------------|------------------------------|-------------------------------------------|
| make runtime choosing supported | KubernetesRuntime = "io.kubernetes.runtime" | V1.6 + | https://github.com/alibaba/pouch/pull/1593 |
| make lxcfs configurable supported | LxcfsEnabled = "io.kubernetes.lxcfs.enabled" | V1.10 + | https://github.com/alibaba/pouch/pull/2210 |
| Runtime choosing | io.kubernetes.runtime | V1.6 + | https://github.com/alibaba/pouch/pull/1593 |
| LXCFS switcher | io.kubernetes.lxcfs.enabled | V1.10 + | https://github.com/alibaba/pouch/pull/2210 |
| VM passthrough config swither| io.alibaba.pouch.vm.passthru | V1.10+ | https://github.com/alibaba/pouch/pull/2437 |
| VM passthrough IP | io.alibaba.pouch.vm.passthru.ip | V1.10+ | https://github.com/alibaba/pouch/pull/2437 |

NOTES: The way to specify runtime using **KubernetesRuntime annotation is Deprecated**. It is recommended to use [RuntimeClass](https://v1-12.docs.kubernetes.io/docs/concepts/containers/runtime-class) which is an alpha feature for selecting the container runtime configuration to use to run a pod’s containers.
NOTES: **Specify runtimes using `io.kubernetes.runtime` annotation is Deprecated**. It is recommended to use [RuntimeClass](https://v1-12.docs.kubernetes.io/docs/concepts/containers/runtime-class) which is an alpha feature for selecting the container runtime configuration to use to run a pod’s containers.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RuntimeClass is a stable feature and the url is deprecated, could you help to fix them? Also there have a merge conflict, thanks to rebase it!


### Make runtime choosing supported
### Runtime choosing

#### What To Solve

Expand Down Expand Up @@ -98,17 +103,17 @@ Linux pouch-runtime-76c8d4d79b-6l5w7 4.12.4-hyper #18 SMP Mon Sep 4 15:10:13 CST

```

### Make lxcfs configurable supported
### LXCFS switcher

#### What To Solve

1. Support resource review isolation via lxcfs in CRI Manager by making lxcfs configurable supported.
1. Support resource view isolation via LXCFS in CRI Manager by making LXCFS configurable supported.

#### How to verify it

1. Prerequisites Installation and make sure your lxcfs service is running.
1. Prerequisites Installation and make sure your LXCFS service is running.

2. Enable pouchd lxcfs (with --enable-lxcfs flag).
2. Enable pouchd LXCFS (with --enable-lxcfs flag).

3. After setting up your kubernetes cluster, you can create a deployment like this :

Expand Down Expand Up @@ -160,7 +165,27 @@ MemAvailable: 261368 kB
......
```

## Pull Request
### VM passthrough config

* feature: make runtime choosing supported [#1593](https://github.com/alibaba/pouch/pull/1593)
* feature: make lxcfs configurable supportd in CRI [#2210](https://github.com/alibaba/pouch/pull/2210)
#### What To Solve

pass through some config to qemu

1. `io.alibaba.pouch.vm.passthru` specify whether a config should be passed through to qemu
2. `io.alibaba.pouch.vm.passthru.ip` specify the IP of the container.

## The container labels rule

### Used by PouchContainer implementation

| LabelKey | Usage |
| --- | --- |
| io.kubernetes.pouch.type | identify whether a container is a sandbox or a regular container |
| io.kubernetes.sandbox.id | attaching to a regular container specify which sandbox it belongs to |
| io.kubernetes.container.logpath | log path of the container |

### Generated from kubernetes spec

PouchContainer would generate some labels according to the annotations in spec by attaching the prefix `annotation.` to each annotation key.

For example, annotation `"io.kubernetes.container.restartCount": "0"` would be converted to label `"annotation.io.kubernetes.container.restartCount": "0"`.