Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Advertise the readiness of the driver on each node. #72

Merged
merged 6 commits into from
May 14, 2021

Conversation

dysosmus
Copy link
Contributor

@dysosmus dysosmus commented May 4, 2021

This PR set the label gcs.csi.ofek.dev/driver-ready={true,false} on each node running csi-gcs to reflect the state of readiness of the driver.

This allows pods to avoid nodes where csi-gcs is still starting up.

Copy link
Owner

@ofek ofek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I'll let @maennchen give the final approval.

Can you please briefly explain/show how this would be used?

@dysosmus
Copy link
Contributor Author

dysosmus commented May 8, 2021

Hi, indeed I forgot to provide a use case.

When pods get scheduled on freshly created nodes and the driver just started many pods will temporarily fail with MountVolume.MountDevice failed for volume xxxx usually because the driver is still registering, for example. It only persists for a few seconds. However, we noticed at the company where I work, it has a tendency to make our users worried.

Having the driver readiness signalled at node level with a label allow us to inject proper affinity/node selector in pods mounting csi-gcs volume and ensure pods only get scheduled after the driver is ready to mount volumes. This results in a smoother user experience.
It's of course a trade-off of longer scheduling time vs temporary warnings at pod startup.

@ofek
Copy link
Owner

ofek commented May 8, 2021

Great! Can you please rebase/merge master and add such info to https://github.com/ofek/csi-gcs/blob/master/docs/troubleshooting.md#todo

(and rename header)

dysosmus added 2 commits May 8, 2021 16:16
Set the label gcs.csi.ofek.dev/driver-ready={true,false}
on each node running a controller
@dysosmus dysosmus force-pushed the feat/label-ready-node branch from 3a4bf25 to fc6cc3f Compare May 8, 2021 14:37
@dysosmus dysosmus force-pushed the feat/label-ready-node branch from fc6cc3f to 8c2f50d Compare May 8, 2021 14:38
@dysosmus
Copy link
Contributor Author

dysosmus commented May 8, 2021

I updated the documentation. If I have the time, and if you are interested, I can contribute the mutating webhook too in a separate PR.

@ofek
Copy link
Owner

ofek commented May 8, 2021

Yes please!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants