Flexvolume volume-plugin-dir is read-only on coreos #57

bcg62 · 2017-11-08T21:03:13Z

This affects things like rook
https://rook.github.io/docs/rook/master/k8s-pre-reqs.html

dghubble · 2017-11-09T05:31:33Z

Agreed, the kubelet flag and mount could be added across platforms. I've had to set it manually to evaluate rook so it'd be nice to standardize it and shouldn't cause any downsides for those not using volume plugins.

Calico and flannel have established a pattern where they use a sidecar pod to install needed binaries on hosts /opt/cni/bin (which happens to be the default ). Flex volume plugins are similar and I think special enough to merit the same treatment - sidecar installs to known location, kubelet mounts.

I'm a bit partial to continuing to use /opt/name for these sorts of cases, but /var/lib/kubelet/volumeplugins (from rook docs) isn't bad either.

It might be a nice exercise to ensure it works for rook and some other popular flex volume plugin too, though I've only used rook myself (and torus back in the day).

cc @barakmich

dghubble · 2017-12-02T22:58:04Z

I'd like to include #61 on bare-metal or maybe all platforms in the next release. I'm also not a heavy rook user at the moment, any chance you've looked at the change or have opinions? Which platform(s) are you seeking?

bcg62 · 2017-12-03T22:03:04Z

lgtm, my use case at the moment is bare metal

dghubble · 2017-12-05T21:38:23Z

Added on bare-metal. I'll verify with rook soon, but these flags are working and sensible to always have enabled.

ivan4th · 2018-02-21T11:35:45Z

This problem is not limited to bare-metal. For example, I want to use DigitalOcean block storage via external-storage, and to do so I also need to be able to install a flexvolume plugin.

ivan4th · 2018-02-21T13:30:42Z

In addition to that, DigitalOcean block storage plugin executable needs to be accessible by kube-controller-manager, it's possible to achieve that by for example mounting /var/lib/kubelet/volumeplugins from the host under /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ in the kube-controller-manager pod. I can try and make a PR to support it but I'm not sure what's the best way to add mounts to the controller manager pod.

dghubble · 2018-02-21T16:47:04Z

I have no objection to providing the --volume-plugin-dir=/var/lib/kubelet/volumeplugins flag to the kubelet on cloud platforms, it corrects the search path for Container Linux. It was simply added to bare-metal first.

I'm not yet familiar with Digital Ocean's plugin so that'd be more of an exploration.

ivan4th · 2018-02-21T19:00:39Z

Thanks. And what's the correct way of modifying mounts in the kube-controller-manager deployment? Do I need to submit a PR for bootkube for this or are there other ways to do it? (Patching the deployment after everything is installed being an obvious one perhaps...)

ivan4th · 2018-02-21T19:01:45Z

(many flexvolume plugins need to be run not only by kubelet but also by kube-controller-manager, and the plugin from external-storage project that handles DigitalOcean volumes is an example of such situation)

dghubble · 2018-02-21T19:30:47Z

You may propose mount changes to the controller manager manifest in upstream bootkube and Typhoon's terraform-render-bootkube. Any changes should be generally applicable and agnostic to platform and plugin. Plugin vendors must document any control-plane modifications they require and provide appropriate sidecars for installing the plugins.

To experiment, you may use kubectl edit to change the manifest, its part of the value of self-hosting control plane components.

At the moment, I'm not sure which Digital Ocean plugin you're referring to. It may be helpful to link what you're trying to use / talking about. Quality of flexvolume plugins varies.

ivan4th · 2018-02-21T19:37:38Z

I mean this plugin. As of adding mounts to the controller-manager deployment via kubectl edit, I already tried it and it worked for me on DO so I was able to deploy helm charts with PVCs etc.

aknuds1 · 2018-04-05T08:44:41Z

If I want to use the DigitalOcean storage provisioner, is it enough to modify it to use /var/lib/kubelet/volumeplugins as its volume plugin directory in this file? Or do I also have to modify the kube-controller-manager as alluded to in this thread?

aknuds1 · 2018-04-05T09:02:19Z

To answer my own question, it does look as if the controller manager also needs to be modified as @ivan4th says, since persistent volumes fail to mount:

Unable to mount volumes for pod "busy-pod_default(501d9e4a-38af-11e8-b39a-4670a3129ce7)": timeout expired waiting for volumes to attach or mount for pod "default"/"busy-pod". list of unmounted volumes=[vol1]. list of unattached volumes=[vol1 default-token-vw56f]

How should I modify the controller manager? I'm not so familiar with the internals of Kubernetes.

klausenbusk · 2018-04-05T09:05:20Z

@aknuds1 Please see kubernetes-retired/external-storage#531, it should contain all the required steps.

aknuds1 · 2018-04-05T09:41:35Z

@klausenbusk Thanks! I can't figure out the exact steps from reading that issue though :/ Sorry if I'm missing the obvious. From what I can tell there are two steps:

Mount /var/lib/kubelet/volumeplugin into kube-controller-manager as /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
Add DIGITALOCEAN_ACCESS_TOKEN env variable to kube-controller-manager

How should I implement the two above steps with Typhoon?

klausenbusk · 2018-04-05T09:50:00Z

How should I implement the two above steps with Typhoon?

kubectl -n kube-system edit deploy kube-controller-manager IIRC

Regard the flexvolume change, pleaae see kubernetes-retired/external-storage#571.

aknuds1 · 2018-04-05T14:19:18Z

@klausenbusk Ah OK, I could add $DIGITALOCEAN_ACESS_TOKEN. However, if I understand kubernetes-retired/external-storage#571 correctly, I should edit /etc/kubernetes/manifests/kube-controller-manager.yaml? I have no such file neither on my master nor my workers.

klausenbusk · 2018-04-05T14:29:05Z

I have no such file neither on my master nor my workers.

kubeadm use static manifest files, bootkube use self-hosted manifest files, so you should use kubectl edit.

aknuds1 · 2018-04-05T14:49:01Z

@klausenbusk OK thanks! I've edited the kube-controller-manager deployment now however, mounting /etc/kubernetes/kubelet-plugins/volume/ to /etc/kubernetes/kubelet-plugins/volume in the containers and adding the flag --flex-volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume, and mounting the persistent volume still doesn't work :(

aknuds1 · 2018-04-05T14:50:25Z

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
  creationTimestamp: 2018-04-05T08:02:29Z
  generation: 3
  labels:
    k8s-app: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
  resourceVersion: "37968"
  selfLink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/kube-controller-manager
  uid: afb44481-38a7-11e8-9b03-4670a3129ce7
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kube-controller-manager
      tier: control-plane
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: kube-controller-manager
        tier: control-plane
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: tier
                  operator: In
                  values:
                  - control-plane
                - key: k8s-app
                  operator: In
                  values:
                  - kube-controller-manager
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - command:
        - ./hyperkube
        - controller-manager
        - --use-service-account-credentials
        - --allocate-node-cidrs=true
        - --cloud-provider=
        - --cluster-cidr=10.2.0.0/16
        - --service-cluster-ip-range=10.3.0.0/16
        - --configure-cloud-routes=false
        - --leader-elect=true
        - --root-ca-file=/etc/kubernetes/secrets/ca.crt
        - --service-account-private-key-file=/etc/kubernetes/secrets/service-account.key
        - --flex-volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume
        env:
        - name: DIGITALOCEAN_ACCESS_TOKEN
          valueFrom:
            secretKeyRef:
              key: access-token
              name: digitalocean
        image: gcr.io/google_containers/hyperkube:v1.10.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10252
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 15
        name: kube-controller-manager
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/kubernetes/secrets
          name: secrets
          readOnly: true
        - mountPath: /etc/ssl/certs
          name: ssl-host
          readOnly: true
        - mountPath: /etc/kubernetes/kubelet-plugins/volume
          name: flexvolume-mount
          readOnly: true
      dnsPolicy: Default
      nodeSelector:
        node-role.kubernetes.io/master: ""
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccount: kube-controller-manager
      serviceAccountName: kube-controller-manager
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      volumes:
      - name: secrets
        secret:
          defaultMode: 420
          secretName: kube-controller-manager
      - hostPath:
          path: /usr/share/ca-certificates
          type: ""
        name: ssl-host
      - hostPath:
          path: /etc/kubernetes/kubelet-plugins/volume/
          type: DirectoryOrCreate
        name: flexvolume-mount
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: 2018-04-05T08:03:23Z
    lastUpdateTime: 2018-04-05T08:03:23Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: 2018-04-05T08:02:40Z
    lastUpdateTime: 2018-04-05T14:44:34Z
    message: ReplicaSet "kube-controller-manager-5d8564d6dc" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 3
  readyReplicas: 2
  replicas: 2
  updatedReplicas: 2

aknuds1 · 2018-04-05T15:16:07Z

@klausenbusk I was able to make it work! I realized I had copied the wrong host volume plugins path from the 571 issue :/

I am still very much interested in taking the next step though and modifying Typhoon (and/or Bootkube) so that this will be automated. Do you know how this could be done?

klausenbusk · 2018-04-05T15:30:52Z

I am still very much interested in taking the next step though and modifying Typhoon (and/or Bootkube) so that this will be automated. Do you know how this could be done?

I know kubeadm contain some logic to add the flexvolume hostPath, maybe bootkube could do the same (in a smarter way, to support read-only /usr)?
kubernetes/kubernetes#51296
kubernetes/kubernetes#55320

The easiest option is probably patching kube-controller-manager.yaml after bootkube render and inject all the required provisioner manifest files (put them in the render output folder, before bootkube start).

klausenbusk · 2018-04-06T13:53:35Z

@poseidon Would it make sense to create a DigitalOcean provisioner addon or include it as default?

aknuds1 · 2018-04-06T13:56:50Z

@klausenbusk Could it complicate things that DigitalOcean are developing their own controller manager which is supposed to implement storage provisioning in the future? I.e., once the latter implements storage provisioning, I think it would be preferable.

klausenbusk · 2018-04-06T14:03:30Z

Could it complicate things that DigitalOcean are developing their own controller manager which is supposed to implement storage provisioning in the future?

I don't think that is gonna happen in the near-future, so we need something else for the time being. The controller manager is also currently blocked on kubernetes/kubernetes#55633, so we can't include it.

I.e., once the latter implements storage provisioning, I think it would be preferable

digitalocean/digitalocean-cloud-controller-manager#19 is worth reading.

aknuds1 · 2018-04-06T15:07:18Z

I see, you clearly have more insight into this issue than I have 🙂 Would there be an easy migration path from the storage provisioner to DigitalOcean's official solution once that arrives you think?

…

On Fri, Apr 6, 2018, 16:03 Kristian Klausen ***@***.***> wrote: Could it complicate things that DigitalOcean are developing their own controller manager which is supposed to implement storage provisioning in the future? I don't think that is gonna happen in the near-future, so we need something else for the time being. The controller manager is also currently blocked on kubernetes/kubernetes#55633 <kubernetes/kubernetes#55633>, so we can't include it. I.e., once the latter implements storage provisioning, I think it would be preferable digitalocean/digitalocean-cloud-controller-manager#19 <digitalocean/digitalocean-cloud-controller-manager#19> is worth reading. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#57 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARK1_JI-ipPAto6kkxK_UrHwY1Juq7Lks5tl3W1gaJpZM4QXCSp> .

klausenbusk · 2018-04-06T15:18:16Z

Would there be an easy migration path from the storage provisioner to DigitalOcean's official solution once that arrives you think?

It should be easy to script a solution. Worst case we let k8s create new PV and copy the data from the old DigitalOcean provisioner PV.

aknuds1 · 2018-04-06T16:44:06Z

@klausenbusk OK great! I would be all for including the storage provisioner then with Typhoon, either as an add-on or built in. It's very useful!

klausenbusk · 2018-04-06T17:51:41Z

I don't think that is gonna happen in the near-future, so we need something else for the time being.

Oh, they are already working on it: https://github.com/digitalocean/digitalocean-cloud-controller-manager/pulls?utf8=%E2%9C%93&q=csi . I didn't expect that. But we are still blocked on kubernetes/kubernetes#55633 sadly :/

klausenbusk · 2018-04-11T21:30:48Z

FYI: https://github.com/digitalocean/csi-digitalocean

dghubble mentioned this issue Nov 11, 2017

Add kubelet --volume-plugin-dir flag on bare-metal #61

Merged

adarshaj mentioned this issue Dec 2, 2017

Added Tectonic documentation based on XenServer rook/rook#1187

Merged

dghubble closed this as completed Dec 5, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flexvolume volume-plugin-dir is read-only on coreos #57

Flexvolume volume-plugin-dir is read-only on coreos #57

bcg62 commented Nov 8, 2017

dghubble commented Nov 9, 2017 •

edited

Loading

dghubble commented Dec 2, 2017

bcg62 commented Dec 3, 2017

dghubble commented Dec 5, 2017

ivan4th commented Feb 21, 2018

ivan4th commented Feb 21, 2018

dghubble commented Feb 21, 2018 •

edited

Loading

ivan4th commented Feb 21, 2018

ivan4th commented Feb 21, 2018

dghubble commented Feb 21, 2018

ivan4th commented Feb 21, 2018

aknuds1 commented Apr 5, 2018

aknuds1 commented Apr 5, 2018 •

edited

Loading

klausenbusk commented Apr 5, 2018

aknuds1 commented Apr 5, 2018

klausenbusk commented Apr 5, 2018

aknuds1 commented Apr 5, 2018

klausenbusk commented Apr 5, 2018

aknuds1 commented Apr 5, 2018

aknuds1 commented Apr 5, 2018

aknuds1 commented Apr 5, 2018

klausenbusk commented Apr 5, 2018

klausenbusk commented Apr 6, 2018

aknuds1 commented Apr 6, 2018 •

edited

Loading

klausenbusk commented Apr 6, 2018

aknuds1 commented Apr 6, 2018 via email

klausenbusk commented Apr 6, 2018

aknuds1 commented Apr 6, 2018

klausenbusk commented Apr 6, 2018

klausenbusk commented Apr 11, 2018

Flexvolume volume-plugin-dir is read-only on coreos #57

Flexvolume volume-plugin-dir is read-only on coreos #57

Comments

bcg62 commented Nov 8, 2017

dghubble commented Nov 9, 2017 • edited Loading

dghubble commented Dec 2, 2017

bcg62 commented Dec 3, 2017

dghubble commented Dec 5, 2017

ivan4th commented Feb 21, 2018

ivan4th commented Feb 21, 2018

dghubble commented Feb 21, 2018 • edited Loading

ivan4th commented Feb 21, 2018

ivan4th commented Feb 21, 2018

dghubble commented Feb 21, 2018

ivan4th commented Feb 21, 2018

aknuds1 commented Apr 5, 2018

aknuds1 commented Apr 5, 2018 • edited Loading

klausenbusk commented Apr 5, 2018

aknuds1 commented Apr 5, 2018

klausenbusk commented Apr 5, 2018

aknuds1 commented Apr 5, 2018

klausenbusk commented Apr 5, 2018

aknuds1 commented Apr 5, 2018

aknuds1 commented Apr 5, 2018

aknuds1 commented Apr 5, 2018

klausenbusk commented Apr 5, 2018

klausenbusk commented Apr 6, 2018

aknuds1 commented Apr 6, 2018 • edited Loading

klausenbusk commented Apr 6, 2018

aknuds1 commented Apr 6, 2018 via email

klausenbusk commented Apr 6, 2018

aknuds1 commented Apr 6, 2018

klausenbusk commented Apr 6, 2018

klausenbusk commented Apr 11, 2018

dghubble commented Nov 9, 2017 •

edited

Loading

dghubble commented Feb 21, 2018 •

edited

Loading

aknuds1 commented Apr 5, 2018 •

edited

Loading

aknuds1 commented Apr 6, 2018 •

edited

Loading