Skip to content

Commit 1746434

Browse files
bart0shbart0sh
committed
implement DSA plugin
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
1 parent cc02785 commit 1746434

File tree

12 files changed

+754
-0
lines changed

12 files changed

+754
-0
lines changed

‎.github/workflows/ci.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ jobs:
7575
- intel-deviceplugin-operator
7676
- intel-sgx-plugin
7777
- intel-sgx-initcontainer
78+
- intel-dsa-plugin
7879

7980
# Demo images
8081
- crypto-perf

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Table of Contents
1919
* [QAT device plugin](#qat-device-plugin)
2020
* [VPU device plugin](#vpu-device-plugin)
2121
* [SGX device plugin](#sgx-device-plugin)
22+
* [DSA device pugin](#dsa-device-plugin)
2223
* [Device Plugins Operator](#device-plugins-operator)
2324
* [Demos](#demos)
2425
* [Developers](#developers)
@@ -163,6 +164,11 @@ operator deployment and NFD is configured to register the SGX EPC memory extende
163164
Containers requesting SGX EPC resources in the cluster use `sgx.intel.com/epc` resource which is of
164165
type [memory](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory).
165166

167+
168+
### DSA device plugin
169+
170+
The [DSA device plugin](cmd/dsa_plugin/README.md) supports acceleration using the Intel Data Streaming accelerator(DSA).
171+
166172
## Device Plugins Operator
167173

168174
Currently the operator has limited support for the QAT, GPU, FPGA and SGX device plugins:

‎build/docker/intel-dsa-plugin.Dockerfile‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# CLEAR_LINUX_BASE and CLEAR_LINUX_VERSION can be used to make the build
2+
# reproducible by choosing an image by its hash and installing an OS version
3+
# with --version=:
4+
# CLEAR_LINUX_BASE=clearlinux@sha256:b8e5d3b2576eb6d868f8d52e401f678c873264d349e469637f98ee2adf7b33d4
5+
# CLEAR_LINUX_VERSION="--version=29970"
6+
#
7+
# This is used on release branches before tagging a stable version.
8+
# The master branch defaults to using the latest Clear Linux.
9+
ARG CLEAR_LINUX_BASE=clearlinux/golang:latest
10+
11+
FROM ${CLEAR_LINUX_BASE} as builder
12+
13+
ARG CLEAR_LINUX_VERSION=
14+
15+
RUN swupd update --no-boot-update ${CLEAR_LINUX_VERSION}
16+
17+
ARG DIR=/intel-device-plugins-for-kubernetes
18+
ARG GO111MODULE=on
19+
WORKDIR $DIR
20+
COPY . .
21+
22+
RUN mkdir /install_root \
23+
&& swupd os-install \
24+
${CLEAR_LINUX_VERSION} \
25+
--path /install_root \
26+
--statedir /swupd-state \
27+
--no-boot-update \
28+
&& rm -rf /install_root/var/lib/swupd/*
29+
30+
RUN cd cmd/dsa_plugin; GO111MODULE=${GO111MODULE} go install; cd -
31+
RUN chmod a+x /go/bin/dsa_plugin \
32+
&& install -D /go/bin/dsa_plugin /install_root/usr/local/bin/intel_dsa_device_plugin \
33+
&& install -D ${DIR}/LICENSE /install_root/usr/local/share/package-licenses/intel-device-plugins-for-kubernetes/LICENSE \
34+
&& scripts/copy-modules-licenses.sh ./cmd/dsa_plugin /install_root/usr/local/share/
35+
36+
FROM scratch as final
37+
COPY --from=builder /install_root /
38+
ENTRYPOINT ["/usr/local/bin/intel_dsa_device_plugin"]

‎cmd/dsa_plugin/README.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Intel DSA device plugin for Kubernetes
2+
3+
Table of Contents
4+
5+
* [Introduction](#introduction)
6+
* [Installation](#installation)
7+
* [Deploy with pre-built container image](#deploy-with-pre-built-container-image)
8+
* [Getting the source code](#getting-the-source-code)
9+
* [Verify node kubelet config](#verify-node-kubelet-config)
10+
* [Deploying as a DaemonSet](#deploying-as-a-daemonset)
11+
* [Build the plugin image](#build-the-plugin-image)
12+
* [Deploy plugin DaemonSet](#deploy-plugin-daemonset)
13+
* [Deploy by hand](#deploy-by-hand)
14+
* [Build the plugin](#build-the-plugin)
15+
* [Run the plugin as administrator](#run-the-plugin-as-administrator)
16+
* [Verify plugin registration](#verify-plugin-registration)
17+
18+
## Introduction
19+
20+
The DSA device plugin for Kubernetes supports acceleration using the Intel Data Streaming accelerator(DSA).
21+
22+
The DSA plugin discovers DSA work queues and presents them as a node resources.
23+
24+
## Installation
25+
26+
The following sections detail how to obtain, build, deploy and test the DSA device plugin.
27+
28+
Examples are provided showing how to deploy the plugin either using a DaemonSet or by hand on a per-node basis.
29+
30+
### Deploy with pre-built container image
31+
32+
[Pre-built images](https://hub.docker.com/r/intel/intel-dsa-plugin)
33+
of this component are available on the Docker hub. These images are automatically built and uploaded
34+
to the hub from the latest master branch of this repository.
35+
36+
Release tagged images of the components are also available on the Docker hub, tagged with their
37+
release version numbers in the format `x.y.z`, corresponding to the branches and releases in this
38+
repository. Thus the easiest way to deploy the plugin in your cluster is to run this command
39+
40+
```bash
41+
$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/dsa_plugin?ref=<REF>
42+
daemonset.apps/intel-dsa-plugin created
43+
```
44+
45+
Where `<REF>` needs to be substituted with the desired git ref, e.g. `master`.
46+
47+
Nothing else is needed. But if you want to deploy a customized version of the plugin read further.
48+
49+
### Getting the source code
50+
51+
```bash
52+
$ export INTEL_DEVICE_PLUGINS_SRC=/path/to/intel-device-plugins-for-kubernetes
53+
$ git clone https://github.com/intel/intel-device-plugins-for-kubernetes ${INTEL_DEVICE_PLUGINS_SRC}
54+
```
55+
56+
### Verify node kubelet config
57+
58+
Every node that will be running the dsa plugin must have the
59+
[kubelet device-plugins](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)
60+
configured. For each node, check that the kubelet device plugin socket exists:
61+
62+
```bash
63+
$ ls /var/lib/kubelet/device-plugins/kubelet.sock
64+
/var/lib/kubelet/device-plugins/kubelet.sock
65+
```
66+
67+
### Deploying as a DaemonSet
68+
69+
To deploy the dsa plugin as a daemonset, you first need to build a container image for the
70+
plugin and ensure that is visible to your nodes.
71+
72+
#### Build the plugin image
73+
74+
The following will use `docker` to build a local container image called
75+
`intel/intel-dsa-plugin` with the tag `devel`.
76+
77+
The image build tool can be changed from the default `docker` by setting the `BUILDER` argument
78+
to the [`Makefile`](Makefile).
79+
80+
```bash
81+
$ cd ${INTEL_DEVICE_PLUGINS_SRC}
82+
$ make intel-dsa-plugin
83+
...
84+
Successfully tagged intel/intel-dsa-plugin:devel
85+
```
86+
87+
#### Deploy plugin DaemonSet
88+
89+
You can then use the [example DaemonSet YAML](/deployments/dsa_plugin/base/intel-dsa-plugin.yaml)
90+
file provided to deploy the plugin. The default kustomization that deploys the YAML as is:
91+
92+
```bash
93+
$ kubectl apply -k deployments/dsa_plugin
94+
daemonset.apps/intel-dsa-plugin created
95+
```
96+
97+
### Deploy by hand
98+
99+
For development purposes, it is sometimes convenient to deploy the plugin 'by hand' on a node.
100+
In this case, you do not need to build the complete container image, and can build just the plugin.
101+
102+
#### Build the plugin
103+
104+
First we build the plugin:
105+
106+
```bash
107+
$ cd ${INTEL_DEVICE_PLUGINS_SRC}
108+
$ make dsa_plugin
109+
```
110+
111+
#### Run the plugin as administrator
112+
113+
Now we can run the plugin directly on the node:
114+
115+
```bash
116+
$ sudo -E ${INTEL_DEVICE_PLUGINS_SRC}/cmd/dsa_plugin/dsa_plugin
117+
device-plugin registered
118+
```
119+
120+
### Verify plugin registration
121+
122+
You can verify the plugin has been registered with the expected nodes by searching for the relevant
123+
resource allocation status on the nodes:
124+
125+
```bash
126+
$ kubectl get nodes -o=jsonpath="{range .items[*]}{.metadata.name}{'\n'}{' i915: '}{.status.allocatable.dsa\.intel\.com/*}{'\n'}"
127+
master
128+
dsa.intel.com/wq-user-dedicated: 1
129+
dsa.intel.com/wq-user-shared: 1
130+
```

‎cmd/dsa_plugin/dsa_plugin.go‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
// Copyright 2020 Intel Corporation. All Rights Reserved.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package main
16+
17+
import (
18+
"flag"
19+
"os"
20+
21+
dpapi "github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin"
22+
"github.com/intel/intel-device-plugins-for-kubernetes/pkg/idxd"
23+
24+
"k8s.io/klog"
25+
)
26+
27+
const (
28+
// Device plugin settings.
29+
namespace = "dsa.intel.com"
30+
// SysFS directory.
31+
sysfsDir = "/sys/bus/dsa/devices"
32+
// Device directories.
33+
devDir = "/dev/dsa"
34+
// Glob pattern for the state sysfs entry.
35+
statePattern = "/sys/bus/dsa/devices/dsa*/wq*/state"
36+
)
37+
38+
func main() {
39+
var sharedDevNum int
40+
41+
flag.IntVar(&sharedDevNum, "shared-dev-num", 1, "number of containers sharing the same work queue")
42+
flag.Parse()
43+
44+
if sharedDevNum < 1 {
45+
klog.Warning("The number of containers sharing the same work queue must be greater than zero")
46+
os.Exit(1)
47+
}
48+
49+
plugin := idxd.NewDevicePlugin(sysfsDir, statePattern, devDir, sharedDevNum)
50+
if plugin == nil {
51+
klog.Fatal("Cannot create device plugin, please check above error messages.")
52+
}
53+
manager := dpapi.NewManager(namespace, plugin)
54+
manager.Run()
55+
}

‎deployments/dsa_plugin/base/intel-dsa-plugin.yaml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
apiVersion: apps/v1
2+
kind: DaemonSet
3+
metadata:
4+
name: intel-dsa-plugin
5+
labels:
6+
app: intel-dsa-plugin
7+
spec:
8+
selector:
9+
matchLabels:
10+
app: intel-dsa-plugin
11+
template:
12+
metadata:
13+
labels:
14+
app: intel-dsa-plugin
15+
spec:
16+
containers:
17+
- name: intel-dsa-plugin
18+
env:
19+
- name: NODE_NAME
20+
valueFrom:
21+
fieldRef:
22+
fieldPath: spec.nodeName
23+
image: intel/intel-dsa-plugin:devel
24+
imagePullPolicy: IfNotPresent
25+
securityContext:
26+
readOnlyRootFilesystem: true
27+
volumeMounts:
28+
- name: devfs
29+
mountPath: /dev/dsa
30+
readOnly: true
31+
- name: chardevs
32+
mountPath: /dev/char
33+
readOnly: true
34+
- name: sysfs
35+
mountPath: /sys/bus/dsa
36+
readOnly: true
37+
- name: kubeletsockets
38+
mountPath: /var/lib/kubelet/device-plugins
39+
volumes:
40+
- name: devfs
41+
hostPath:
42+
path: /dev/dsa
43+
- name: chardevs
44+
hostPath:
45+
path: /dev/char
46+
- name: sysfs
47+
hostPath:
48+
path: /sys/bus/dsa
49+
- name: kubeletsockets
50+
hostPath:
51+
path: /var/lib/kubelet/device-plugins
52+
nodeSelector:
53+
kubernetes.io/arch: amd64

‎deployments/dsa_plugin/base/kustomization.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
resources:
2+
- intel-dsa-plugin.yaml

‎deployments/dsa_plugin/kustomization.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
bases:
2+
- base

‎deployments/dsa_plugin/overlays/namespace_kube-system/add-namespace-kube-system.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
apiVersion: apps/v1
2+
kind: DaemonSet
3+
metadata:
4+
name: intel-dsa-plugin
5+
namespace: kube-system

‎deployments/dsa_plugin/overlays/namespace_kube-system/kustomization.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
bases:
2+
- ../../base
3+
patches:
4+
- add-namespace-kube-system.yaml

0 commit comments

Comments
 (0)