Because the CRI interface of Kubernetes cannot meet the customized development of Kubelet at present, it is necessary to provide the required functions of Kubelet by extending the API of CRI. On top of CRI's existing functionality, use the field extension or add method to make CRI meet the requirements.
Kubernetes Version: V1.10.0+
- Support the ability to update the container quotadir.
- Support more container resource fields define in OCI spec.
- Support the ability to update container runtime spec annotations.
- Add the
DiskQuotafield inLinuxContainerResources, referring to the definition of Resources in moby for better compatibility. The new fields are as follows： - Add some fields from OCI spec
The changes need to be made in the proto file are as follows:
message UpdateContainerResourcesRequest {
// ID of the container to update.
string container_id = 1;
// Resource configuration specific to Linux containers.
LinuxContainerResources linux = 2;
// Annotations contains arbitrary metadata for the container.
// Note: spec_annotations is the metadata for container runtime, such as runc. Not
// the 'annotations' for Kubernetes objects.
map<string, string> spec_annotations = 3;
}
message LinuxContainerResources {
......
//***New Fields***//
// DiskQuota constrains the disk
map<string,string> disk_quota = 100;
// Block IO weight (relative weight vs. other containers)
uint32 blkio_weight = 101;
repeated WeightDevice blkio_weight_device = 102;
repeated ThrottleDevice blkio_device_read_bps = 103;
repeated ThrottleDevice blkio_device_write_bps = 104;
repeated ThrottleDevice blkio_device_read_IOps = 105;
repeated ThrottleDevice blkio_device_write_IOps = 106;
// Kernel memory limit (in bytes)
int64 kernel_memory = 107;
// Memory soft limit (in bytes)
int64 memory_reservation = 108;
// Tuning container memory swappiness behaviour
Int64Value memory_swappiness = 109;
// List of ulimits to be set in the container
repeated Ulimit ulimits = 110;
}
// WeightDevice is a structure that holds device:weight pair
message WeightDevice {
// Path of weightdevice.
string path = 1;
// Weight of weightdevice.
uint32 Weight = 2;
}
// ThrottleDevice is a structure that holds device:rate_per_second pair
message ThrottleDevice {
// Path of throttledevice.
string path = 1;
// Rate of throttledevice.
uint64 rate = 2;
}
// Ulimit is a human friendly version of Rlimit.
message Ulimit {
// Name of ulimit.
string name = 1;
// Hard limit of ulimit.
int64 hard = 2;
// Soft limit of Ulimit.
int64 soft = 3;
}
- Support to the ability to acquire the volumes of the Container.
- Support to the ability to acquire the Resource of the Container.
- Pass the quotaID generated when the container is created for disk reuse.
- Support to the ability to acquire the envs of the Container.
In the container inplace upgrade process, we read the attributes from the old one, and attach it to the new one.
- The
ContainerStatusstruct is used only inContainerStatusResponsein CRI, so the volumes of the container can be obtained by directly addingvolumefield to theContainerStatusstruct. - Add Resources field to support the retrieval of container's resource.
- When get ContainerStatus, the return object of
ContainerStatusResponsewill contain the field ofQuotaId. - When get ContainerStatus, the return object of
ContainerStatusResponsewill contain the field ofEnvs.
// ContainerStatus represents the status of a container.
type ContainerStatus struct {
......
//***New Fields***//
// Volumes of container
Volumes map[string]struct{} `protobuf:"bytes,100,opt,name=volumes,json=volumes" json:"volumes,omitempty"`
// Resources specification for the container
Resources *LinuxContainerResources `protobuf:"bytes,101,opt,name=resources" json:"resources,omitempty"`
// QuotaId of the container
QuotaId string `protobuf:"bytes,102,opt,name=quota_id,json=quotaId,proto3" json:"quota_id,omitempty"`
// List of environment variable to set in the container.
Envs []*KeyValue `protobuf:"bytes,103,rep,name=envs" json:"envs,omitempty"`
}
The changes need to be made in the proto file are as follows:
// ContainerStatus represents the status of a container.
message ContainerStatus {
......
//***New Fields***//
// Volumes of container
map<string, Volume> volumes= 100;
// Resources specification for the container
LinuxContainerResources resources = 101;
// QuotaId of the container
string quota_id = 102;
// List of environment variable to set in the container.
repeated KeyValue envs = 103;
}
message Volume {
}
- Support the ability to acquire the volumes of the Image.
In the container inplace upgrade process, the volumes of the new container needs to remain consistent with that of the old container, so the volumes of the old container needs to be read.
- Add
volumesfield in the Image struct.
// Basic information about a container image.
type Image struct {
......
//***New Fields***//
// Volumes of image
Volumes map[string]struct{} `protobuf:"bytes,7,opt,name=volumes,json=volumes" json:"volumes,omitempty"`
}
The changes need to be made in the proto file are as follows:
// Basic information about a container image.
message Image {
......
//***New Fields***//
// Volumes of image
map<string, Volume> volumes= 100;
}
- Support the ability to set DiskQuota, NetPriority.
LinuxContainerConfigcontainsLinuxContainerResources, which have changed in UpdateContainerResources().So after changing LinuxContainerResources, the Create process already supports the setting of DiskQuota.- For missing fields are as follows (not all):
- NetPriority : Set network priorities
- QuotaId : When creating container, pass parameters of the DiskQuota and QuotaId. (When QuotaId is -1, QuotaId will be automatically generated)
type ContainerConfig struct {
......
//***New Fields***//
// NetPriority of the container
NetPriority int64 `protobuf:"bytes,100,opt,name=net_priority" json:"net_priority,omitempty"`
// QuotaId of the container
QuotaId string `protobuf:"bytes,101,opt,name=quota_id,json=quotaId,proto3" json:"quota_id,omitempty"`
}
The changes need to be made in the proto file are as follows:
message ContainerConfig {
......
//***New Fields***//
// NetPriority of the containeri
int64 net_priority = 100;
// QuotaId of the container
string quota_id = 101;
}
- After kubelet performs upgrade, the container upgraded cannot delete the anonymous volume inherited.
- Provides an interface for removing volume.
- The containerstatus interface supports querying volume by name.
The changes need to be made in the proto file are as follows:
service VolumeService {
// RemoveVolume volume an volume
rpc RemoveVolume(RemoveVolumeRequest) returns (RemoveVolumeResponse) {}
}
message RemoveVolumeRequest {
// name of the volume to remove
string volume_name = 1;
}
message RemoveVolumeResponse {}
Add name field in the Mount struct：
// Mount specifies a host volume to mount into a container.
message Mount {
......
//***New Fields***//
// Name of volume
string name = 100;
}
- StartPodSandbox restarts a sandbox pod which was stopped by accident and setup the network with network plugin.
It will fail to get IP after PodSandbox restarts because of the external factors such as shutdown the host. It is a solution to avoid container reallocating.
- Provides an interface for starting PodSandbox specified.
The changes need to be made in the proto file are as follows:
service RuntimeService {
...
// Start a sandbox pod which was forced to stop by external factors.
// Network plugin returns same IPs when input same pod names and namespaces
rpc StartPodSandbox(StartPodSandboxRequest) returns (StartPodSandboxResponse) {}
...
}
message StartPodSandboxRequest {
// ID of the PodSandbox to start.
string pod_sandbox_id = 1;
}
message StartPodSandboxResponse {}
- PauseContainer pause a container.
- UnpauseContainer unpause a container.
Under serverless situation, we may pre-allocate a batch of container which were ready to serve, waiting online. Using pause container to balance the resource cost and application start-up time.
- Extend the RuntimeService interface
The changes need to be made in the proto file are as follows:
service RuntimeService {
...
// PauseContainer pauses the container.
rpc PauseContainer(PauseContainerRequest) returns (PauseContainerResponse) {}
// UnpauseContainer unpauses the container.
rpc UnpauseContainer(UnpauseContainerRequest) returns (UnpauseContainerResponse) {}
...
}
message PauseContainerRequest {
// ID of the container to pause.
string container_id = 1;
}
message PauseContainerResponse {}
message UnpauseContainerRequest {
// ID of the container to unpause.
string container_id = 1;
}
message UnpauseContainerResponse {}
- feature: extend cri apis for special needs #1617
- feature: extend cri apis for remove volume #2124
- feature: extend cri apis for support quotaID #2138
- feature: extend cri apis for get envs #2163
- feature: extend cri apis for support StartPodSandbox #2242
- feature: extend cri apis for support pause/unpause container #2623