add advanced cpuset manger

[szy441687879]

Advanced CPUSet Manager

• Provide a new mechanism to manage cpuset bypass, to do extra things that kubelet cannot do, such as revise cpuset when pod running, share the cpuset assigned with others and relax restrictions of cpuset allocation.

Table of Contents

• Advanced CPUSet Manager
  • Table of Contents
  • Motivation
    • Goals
    • Non-Goals/Future Work
  • Proposal
    • Relax restrictions of cpuset allocation
    • Add new annotation to describe the requirement of cpuset contorl manger
    • Advanced CPU Manager component
    • User Stories
      • Story 1
      • Story 2
    • Risks and Mitigations

Motivation

Some latency-sensitive application have lower lantency and cpu usage when running with specific cores, which results in fewer context switchs and higer cache affinity.
But kubelet will always exclude assigned cores in shared cores, which may waste resources.Offline and other online pods can running on the codes actually. In our experiment, for the most part, it is barely noticeable for performance of service.

Goals

• Provide a new mechanism to manage cpuset bypass
• Provide a new cpuset control way "shared"
• Allow revise cpuset when pod running
• Relax restrictions of cpuset allocation

Non-Goals/Future Work

• Solve the conflict with kubelet static cpuset control (need close)
• Numa manager will support in future, CCX/CCD manager can also considered

Proposal

Relax restrictions of cpuset allocation

Kubelet allocate cpus for containers included in a pod of "Guaranteed" QoS class and the container's resource limit for the CPU resource is an integer greater than or equal to one.

With Crane, containers only to neeed CPU resource limit is an integer greater than or equal to one and equal to cpu resource request.

Add new annotation to describe the requirement of cpuset contorl manger

apiVersion: v1
kind: Pod
metadata:
  annotations:
    qos.gocrane.io/cpu-manager: none/exclusive/share

Provide three policy for cpuset control:

• none: containers of this pod shares a set of cpus not allocated top exclusive policy containers
• exclusive: containers of this pod exclude the CPUs allocated, other containers not allowed to use.
• share: containers of this pod runs in the CPUs allocated, but other containers can alse use.

Advanced CPU Manager component

• Crane-agent use podLister informs to sense the creation of pod.
• Crane-agent use alloca when pod is binded, and loop in cycle to addContainer(change cpuset) until contanerd is created
• Update/Delete pod take effect in reconcile state.
• state.State reference kubelet and topology_cpu_assign copied from kubelet

User Stories

• users can update pod annotaion to control cpuset policy flexibly

Story 1

make some pods to run with exclusive Cpus to compare with others

Story 2

make pod from exclusive to share,so that offline can use the cpu

Risks and Mitigations

• kubelet cpu manger policy need to be set to none or will be conflicted with crane-agent
• if crane-agent can not allocate CPUs for pods, it will not refuse pod to start as kubelet

[szy441687879]

2. if a pod qos policy is set as exclusive and if pod request all cpu on a node, after this pod exclusively bind all the cpus, all scheduler knows there is no allocatable resource on the node and filter out the node from candidate list?

@szy441687879 I have two general questions:

1. when bind cpuset, how the cpu index is chosen
2. if a pod qos policy is set as exclusive and if pod request all cpu on a node, after this pod exclusively bind all the cpus, all scheduler knows there is no allocatable resource on the node and filter out the node from candidate list?

1. It's the same with kubelet, take by topology.Acquire whole sockets > Acquire whole cores > Acquire single threads
2. It is not conflicted with scheduler, if a pod request to bind all cpus, the node will not have extra cpu resources remains .

[kitianFresh]

exclusive: containers of this pod exclude the CPUs allocated, other containers not allowed to use.

With Crane, containers only to neeed CPU resource limit is an integer greater than or equal to one and equal to cpu resource request.

I want to know if the pod has request do not equal to limit, limit =4, request= 2, then i patch it use exclude binding cpuset, what will you do. If you binding 4 cpus excluding, then it must conflict with the scheduler expected behavior. because scheduler just compute 2 cores as comsumed cpu number, but you exclude 4 cpus.

[szy441687879]

I want to know if the pod has request do not equal to limit, limit =4, request= 2, then i patch it use exclude binding cpuset, what will you do. If you binding 4 cpus excluding, then it must conflict with the scheduler expected behavior. because scheduler just compute 2 cores as comsumed cpu number, but you exclude 4 cpus.

@kitianFresh
It will not take effect now, container's cpu request must equal limit.
But in the future，tencent os will support burst to other cpusets even binding cpus.

[mfanjie]

I am basically ok for the change, several comments

1. if we are copying some source code with any customized logic, suggest to vendor them
2. please clean up logs following crane code standards

[mfanjie]

@szy441687879 need a rebase and could you squash the commits?

[szy441687879]

@szy441687879 need a rebase and could you squash the commits?

@mfanjie done

[mfanjie]

/lgtm