Skip to content

Commit

Permalink
[en] Add cluster-scheduler-plugin.md to kpanda
Browse files Browse the repository at this point in the history
  • Loading branch information
windsonsea committed Mar 28, 2024
1 parent 2ed59af commit ad03fc3
Show file tree
Hide file tree
Showing 3 changed files with 224 additions and 56 deletions.
164 changes: 164 additions & 0 deletions docs/en/docs/kpanda/user-guide/clusters/cluster-scheduler-plugin.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
# Deploy Second Scheduler scheduler-plugins in a Cluster

This page describes how to deploy a second scheduler scheduler-plugins in a cluster.

## Why do we need scheduler-plugins?

The cluster created through the platform will install the native K8s scheduler, but the native scheduler has many limitations:

- The native scheduler cannot meet scheduling requirements, you can choose to use
[CoScheduling](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/coscheduling),
[CapacityScheduling](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master/pkg/capacityscheduling)
and other scheduler-plugins plugins
- In special scenarios, a new scheduler is needed to complete scheduling tasks without affecting the process of the native scheduler
- Distinguish schedulers with different functionalities by switching scheduler names to achieve different scheduling scenarios

This page takes the scenario of using the vgpu scheduler while combining the cosheduling plugin capability of scheduler-plugins as an example to introduce how to install and use scheduler-plugins.

## Install scheduler-plugins

### Prerequisites

- kubean is a new feature introduced in version v0.13.0, please ensure that the version is >= this version when managing the cluster.
- The installation version of scheduler-plugins is v0.27.8, please ensure that the cluster version is compatible with it.
Refer to the document [Compatibility Matrix](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master?tab=readme-ov-file#compatibility-matrix).

### Installation Process

1. Add the scheduler-plugins parameter in **Create Cluster** -> **Advanced Configuration** -> **Custom Parameters**.

```yaml
scheduler_plugins_enabled:true
scheduler_plugins_plugin_config:
- name: Coscheduling
args:
permitWaitingTimeSeconds: 10 # default is 60
```
Parameters:
- `scheduler_plugins_enabled` When set to true, enables the scheduler-plugins capability.
- You can enable or disable certain plugins by setting the `scheduler_plugins_enabled_plugins` or `scheduler_plugins_disabled_plugins` options.
See [K8s Official Plugin Names](https://github.com/kubernetes-sigs/scheduler-plugins?tab=readme-ov-file#plugins) for reference.
- If you need to set parameters for custom plugins, please configure `scheduler_plugins_plugin_config`,
for example: set the `permitWaitingTimeoutSeconds` parameter for coscheduling.
See [K8s Official Plugin Configuration](https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/manifests/coscheduling/scheduler-config.yaml) for reference.

<!-- ![Add scheduler-plugins parameter](../../images/cluster-scheduler-plugin-01.png) -->

2. After successful cluster creation, the system will automatically install the scheduler-plugins and
controller component loads. You can check the load status in the corresponding cluster's stateless loads.

<!-- ![Check plugin load status](../../images/cluster-scheduler-plugin-02.png) -->

## Using scheduler-plugins

Here is an example of how to use scheduler-plugins in conjunction with the coscheduling plugin while using the vgpu scheduler.

1. Install vgpu in the Helm template and set the values.yaml parameters.

- `schedulerName: scheduler-plugins-scheduler`: This is the scheduler name for scheduler-plugins installed by kubean, and currently cannot be modified.
- `scheduler.kubeScheduler.enabled: false`: Do not install kube-scheduler and use vgpu-scheduler as a separate extender.

<!-- ![Install vgpu plugin](../../images/cluster-scheduler-plugin-03.png) -->

1. Extend vgpu-scheduler on scheduler-plugins.

```bash
[root@master01 charts]# kubectl get cm -n scheduler-plugins scheduler-config -ojsonpath="{.data.scheduler-config\.yaml}"
```

```yaml
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
leaderElection:
leaderElect: false
profiles:
# Compose all plugins in one profile
- schedulerName: scheduler-plugins-scheduler
plugins:
multiPoint:
enabled:
- name: Coscheduling
- name: CapacityScheduling
- name: NodeResourceTopologyMatch
- name: NodeResourcesAllocatable
disabled:
- name: PrioritySort
pluginConfig:
- args:
permitWaitingTimeSeconds: 10
name: Coscheduling
```

Modify configmap of scheduler-config for scheduler-plugins:

```bash
[root@master01 charts]# kubectl get cm -n scheduler-plugins scheduler-config -ojsonpath="{.data.scheduler-config\.yaml}"
```

```yaml
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
leaderElection:
leaderElect: false
profiles:
# Compose all plugins in one profile
- schedulerName: scheduler-plugins-scheduler
plugins:
multiPoint:
enabled:
- name: Coscheduling
- name: CapacityScheduling
- name: NodeResourceTopologyMatch
- name: NodeResourcesAllocatable
disabled:
- name: PrioritySort
pluginConfig:
- args:
permitWaitingTimeSeconds: 10
name: Coscheduling
extenders:
- urlPrefix: "${urlPrefix}"
filterVerb: filter
bindVerb: bind
nodeCacheCapable: true
ignorable: true
httpTimeout: 30s
weight: 1
enableHTTPS: true
tlsConfig:
insecure: true
managedResources:
- name: nvidia.com/vgpu
ignoredByScheduler: true
- name: nvidia.com/gpumem
ignoredByScheduler: true
- name: nvidia.com/gpucores
ignoredByScheduler: true
- name: nvidia.com/gpumem-percentage
ignoredByScheduler: true
- name: nvidia.com/priority
ignoredByScheduler: true
- name: cambricon.com/mlunum
ignoredByScheduler: true
```

1. After installing vgpu-scheduler, the system will automatically create a service (svc), and the urlPrefix specifies the URL of the svc.

!!! note

- The svc refers to the pod service load. You can use the following command in the namespace where the nvidia-vgpu plugin is installed to get the external access information for port 443.

```shell
kubectl get svc -n ${namespace}
```

- The urlPrefix format is https://${ip address}:${port}.

1. Restart the scheduler pod of scheduler-plugins to load the new configuration file.

!!! note

When creating a vgpu application, you do not need to specify the scheduler name. The vgpu-scheduler webhook
will automatically change the scheduler's name to "scheduler-plugins-scheduler" without manual specification.
1 change: 1 addition & 0 deletions docs/en/navigation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ nav:
- Choose Runtime: kpanda/user-guide/clusters/runtime.md
- Cluster Versions: kpanda/user-guide/clusters/cluster-version.md
- Integrate Rancher Cluster: kpanda/user-guide/clusters/integrate-rancher-cluster.md
- Deploy 2nd Scheduler in a Cluster: kpanda/user-guide/clusters/cluster-scheduler-plugin.md
- Namespaces:
- Create Namespace: kpanda/user-guide/namespaces/createns.md
- Exclusive Node to Namespace: kpanda/user-guide/namespaces/exclusive.md
Expand Down
115 changes: 59 additions & 56 deletions docs/zh/docs/kpanda/user-guide/clusters/cluster-scheduler-plugin.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# 如何在集群中部署第二调度器 scheduler-plugins

本文介绍如何在集群中部署第二个调度器 scheduler-plugins。

## 为什么需要 scheduler-plugins?

通过平台创建的集群中会安装 K8s 原生的调度器,但是原生的调度器存在很多的局限性:
Expand All @@ -15,13 +17,13 @@

## 安装 scheduler-plugins

**前置条件**
### 前置条件

- kubean 是在 v0.13.0 版本推出的新功能,选择管理集群时请确保版本是否 >= 此版本。
- 安装 scheduler-plugins 版本为 v0.27.8,请确保集群版本是否与它兼容。
参考文档 [Compatibility Matrix](https://github.com/kubernetes-sigs/scheduler-plugins/tree/master?tab=readme-ov-file#compatibility-matrix)

**安装流程**
### 安装流程

1.**创建集群** -> **高级配置** -> **自定义参数** 中添加 scheduler-plugins 参数

Expand Down Expand Up @@ -57,86 +59,87 @@
- `scheduler.kubeScheduler.enabled: false`,不安装 kube-scheduler,将 vgpu-scheduler 作为单独的 extender。

![安装 vgpu 插件](../../images/cluster-scheduler-plugin-03.png)
![安装 vgpu 插件](../../images/cluster-scheduler-plugin-04.png)

1. 在 scheduler-plugins 上扩展 vgpu-scheduler。

```bash
[root@master01 charts]# kubectl get cm -n scheduler-plugins scheduler-config -ojsonpath="{.data.scheduler-config\.yaml}"
```

```yaml
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
leaderElection:
leaderElect: false
leaderElect: false
profiles:
# Compose all plugins in one profile
- schedulerName: scheduler-plugins-scheduler
plugins:
multiPoint:
enabled:
- name: Coscheduling
- name: CapacityScheduling
- name: NodeResourceTopologyMatch
- name: NodeResourcesAllocatable
disabled:
- name: PrioritySort
# Compose all plugins in one profile
- schedulerName: scheduler-plugins-scheduler
plugins:
multiPoint:
enabled:
- name: Coscheduling
- name: CapacityScheduling
- name: NodeResourceTopologyMatch
- name: NodeResourcesAllocatable
disabled:
- name: PrioritySort
pluginConfig:
- args:
permitWaitingTimeSeconds: 10
- args:
permitWaitingTimeSeconds: 10
name: Coscheduling
```

修改 scheduler-plugins 的 scheduler-config 的 cofigmap 参数,如下:
修改 scheduler-plugins 的 scheduler-config 的 configmap 参数,如下:

```bash
[root@master01 charts]# kubectl get cm -n scheduler-plugins scheduler-config -ojsonpath="{.data.scheduler-config\.yaml}"
```

```yaml
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
leaderElection:
leaderElect: false
leaderElect: false
profiles:
# Compose all plugins in one profile
- schedulerName: scheduler-plugins-scheduler
plugins:
multiPoint:
enabled:
- name: Coscheduling
- name: CapacityScheduling
- name: NodeResourceTopologyMatch
- name: NodeResourcesAllocatable
disabled:
- name: PrioritySort
# Compose all plugins in one profile
- schedulerName: scheduler-plugins-scheduler
plugins:
multiPoint:
enabled:
- name: Coscheduling
- name: CapacityScheduling
- name: NodeResourceTopologyMatch
- name: NodeResourcesAllocatable
disabled:
- name: PrioritySort
pluginConfig:
- args:
permitWaitingTimeSeconds: 10
- args:
permitWaitingTimeSeconds: 10
name: Coscheduling
extenders:
- urlPrefix: "${urlPrefix}"
filterVerb: filter
bindVerb: bind
nodeCacheCapable: true
ignorable: true
httpTimeout: 30s
weight: 1
enableHTTPS: true
tlsConfig:
insecure: true
managedResources:
- name: nvidia.com/vgpu
ignoredByScheduler: true
- name: nvidia.com/gpumem
ignoredByScheduler: true
- name: nvidia.com/gpucores
ignoredByScheduler: true
- name: nvidia.com/gpumem-percentage
ignoredByScheduler: true
- name: nvidia.com/priority
ignoredByScheduler: true
- name: cambricon.com/mlunum
ignoredByScheduler: true
- urlPrefix: "${urlPrefix}"
filterVerb: filter
bindVerb: bind
nodeCacheCapable: true
ignorable: true
httpTimeout: 30s
weight: 1
enableHTTPS: true
tlsConfig:
insecure: true
managedResources:
- name: nvidia.com/vgpu
ignoredByScheduler: true
- name: nvidia.com/gpumem
ignoredByScheduler: true
- name: nvidia.com/gpucores
ignoredByScheduler: true
- name: nvidia.com/gpumem-percentage
ignoredByScheduler: true
- name: nvidia.com/priority
ignoredByScheduler: true
- name: cambricon.com/mlunum
ignoredByScheduler: true
```

1. 安装完 vgpu-scheduler 后,系统会自动创建 svc,urlPrefix 指定 svc 的 url。
Expand All @@ -149,7 +152,7 @@
kubectl get svc -n ${namespace}
```

- urlprifix 格式为 https://${ip 地址}:${端口}
- urlprifix 格式为 `https://${ip 地址}:${端口}`


1. 将 scheduler-plugins 的 scheduler Pod 重启,加载新的配置文件。
Expand Down

0 comments on commit ad03fc3

Please sign in to comment.