-
-
Notifications
You must be signed in to change notification settings - Fork 39
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Advertise the readiness of the driver on each node. (#72)
* Advertise the readiness of the driver on each node. Set the label gcs.csi.ofek.dev/driver-ready={true,false} on each node running a controller * upgrade driver-ready set failure to warns * Document how gcs.csi.ofek.dev/driver-ready can be used. * documentation reword * According to JSON patch RFC, add should be used here, not replace even if k8s implementation seems forgiveful. * Honnor driver-name flag when setting the driver-ready label on nodes.
- Loading branch information
Showing
4 changed files
with
101 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,47 @@ | ||
# Troubleshooting | ||
## Warnings events from pods scheduled on newly started nodes when mounting csi-gcs volumes | ||
|
||
----- | ||
Warnings, like the one below, can be seen from pods scheduled on newly started nodes. | ||
|
||
## TODO | ||
``` | ||
MountVolume.MountDevice failed for volume "xxxx" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name gcs.csi.ofek.dev not found in the list of registered CSI drivers | ||
``` | ||
|
||
Those warnings are temporary and reflect the csi-gcs driver is still starting. Kubernetes will retry until the csi-gcs driver is ready. | ||
|
||
It's possible to avoid those warnings by adding a node selector or affinity using the node label `gcs.csi.ofek.dev/driver-ready=true`. | ||
|
||
> Adding such node selector or affinity will trade the time spend waiting for volume mounting retries with time waiting for scheduling. | ||
> The exact label added is `<driver name>/driver-ready`, by default `<driver name>` is `gcs.csi.ofek.dev` | ||
``` | ||
apiVersion: v1 | ||
kind: Pod | ||
metadata: | ||
name: pod-mount-csi-gcs-volume | ||
spec: | ||
// ... | ||
nodeSelector: | ||
gcs.csi.ofek.dev/driver-ready: "true" | ||
``` | ||
|
||
``` | ||
apiVersion: v1 | ||
kind: Pod | ||
metadata: | ||
name: pod-mount-csi-gcs-volume | ||
spec: | ||
// ... | ||
affinity: | ||
nodeAffinity: | ||
requiredDuringSchedulingIgnoredDuringExecution: | ||
nodeSelectorTerms: | ||
- matchExpressions: | ||
- key: gcs.csi.ofek.dev/driver-ready | ||
operator: In | ||
values: | ||
- "true" | ||
``` | ||
|
||
You can also add an admission mutating webhook to automatically inject such node selector or affinity in all pods mounting csi-gcs volumes. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters