Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot mount volume with non-root user #6

Closed
epalazuelosb opened this issue Apr 17, 2020 · 26 comments
Closed

Cannot mount volume with non-root user #6

epalazuelosb opened this issue Apr 17, 2020 · 26 comments
Assignees
Labels
question Further information is requested

Comments

@epalazuelosb
Copy link

epalazuelosb commented Apr 17, 2020

Steps

  • kubectl create namespace csi-gcs
  • kubectl create secret generic csi-gcs-secret --from-literal=bucket=my-gcs-bucket --from-file=key=./key.json --namespace csi-gcs
  • kubectl apply -f csi-gcs-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-gcs
  namespace: csi-gcs
provisioner: gcs.csi.ofek.dev
volumeBindingMode: Immediate
allowVolumeExpansion: false
reclaimPolicy: Retain
parameters:
  csi.storage.k8s.io/node-publish-secret-name: csi-gcs-secret
  csi.storage.k8s.io/node-publish-secret-namespace: csi-gcs
  • kubectl apply -f csi-gcs-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: csi-gcs-pv
  namespace: csi-gcs
 #annotations:
   #pv.beta.kubernetes.io/gid: "1000"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-gcs
  csi:
    driver: gcs.csi.ofek.dev
    volumeHandle: csi-gcs
    nodePublishSecretRef:
      name: csi-gcs-secret
      namespace: csi-gcs
  • kubectl apply -f csi-gcs-pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-gcs-pvc
  namespace: csi-gcs
  labels:
    app: csi-gcs-example
spec:
  storageClassName: csi-gcs
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi

Tried to chown objects with initcontainers command

  • kubectl apply -f csi-gcs-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: csi-gcs-test
  namespace: csi-gcs
  labels:
    app: csi-gcs-test
spec:
  selector:
    matchLabels:
      app: csi-gcs-test      
  template:
    metadata:
      labels:
        app: csi-gcs-test
    spec:
      initContainers:
      - name: chown-er
        image: busybox:latest
        securityContext:
          runAsUser: 0
        command:
        - /bin/chown
        - -R
        - "1000:1000"
        - /data
        volumeMounts:
        - name: csi-gcs-pvc
          mountPath: /data
      containers:
      - name: writer
        image: busybox
        command:
        - sleep
        - infinity
        volumeMounts:
        - name: csi-gcs-pvc
          mountPath: /data
      - name: reader
        image: busybox
        command:
        - sleep
        - infinity
        volumeMounts:
        - name: csi-gcs-pvc
          mountPath: /data
          readOnly: true
      volumes:
      - name: csi-gcs-pvc
        persistentVolumeClaim:
          claimName: csi-gcs-pvc
  • kubectl get sc,pv,pvc -n csi-gcs
storageclass.storage.k8s.io/csi-gcs                         gcs.csi.ofek.dev        14d
storageclass.storage.k8s.io/singlewriter-standard           pd.csi.storage.gke.io   16d
persistentvolume/csi-gcs-pv                                 50Gi       RWO            Retain           Bound    csi-gcs/csi-gcs-pvc                  csi-gcs                                  14h
persistentvolumeclaim/csi-gcs-pvc   Bound    csi-gcs-pv   50Gi       RWO            csi-gcs        14h
  • kubectl get deploy -n csi-gcs
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
csi-gcs-test   1/1     1            1           60s
  • kubectl -n csi-gcs exec -it $(kubectl -n csi-gcs get po |grep csi | cut -f 1 -d " ") -c writer -- ls -l /data
total 3
-rw-r--r--    1 root     root            11 Apr  2 19:53 hello_world.hel

Is there any way to mount it as non-root user? Tried also with gid annotations on pv and

fsGroup securityContext on deployment, with the same result

annotations:
  pv.beta.kubernetes.io/gid: "1000"
@maennchen
Copy link
Collaborator

@epalazuelosb Does it work if you specify the correct uid, gid, dir-mode & file-mode flags?

https://github.com/ofek/csi-gcs#extra-flags

@maennchen maennchen added the question Further information is requested label Apr 21, 2020
@maennchen maennchen self-assigned this Apr 21, 2020
@epalazuelosb
Copy link
Author

epalazuelosb commented Apr 21, 2020

Yes, I have tried it, but still not working:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: csi-gcs-pv
  annotations:
    pv.beta.kubernetes.io/gid: "1000"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-gcs
  csi:
    driver: gcs.csi.ofek.dev
    volumeHandle: csi-gcs
    volumeAttributes:
      uid: "1000"
      gid: "1000"
      dir-mode: "777"
      file-mode: "664"
    nodePublishSecretRef:
      name: csi-gcs-secret
      namespace: csi-gcs

@epalazuelosb
Copy link
Author

epalazuelosb commented Apr 21, 2020

apiVersion: v1
kind: PersistentVolume
metadata:
  name: csi-gcs-pv
  namespace: liferay7
  annotations:
    pv.beta.kubernetes.io/gid: "1000"
spec:
  accessModes:
    ReadWriteMany
  capacity:
    storage: 50Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-gcs
  csi:
    driver: gcs.csi.ofek.dev
    volumeHandle: csi-gcs
    volumeAttributes:
      uid: "1000"
      gid: "1000"
      dir-mode: "775"
      file-mode: "664"
    nodePublishSecretRef:
      name: csi-gcs-secret
      namespace: csi-gcs

@maennchen
Copy link
Collaborator

maennchen commented Apr 21, 2020

@epalazuelosb You need to set flags. Does this work?

apiVersion: v1
kind: PersistentVolume
metadata:
  name: csi-gcs-pv
  namespace: liferay7
  annotations:
    pv.beta.kubernetes.io/gid: "1000"
spec:
  accessModes:
    ReadWriteMany
  capacity:
    storage: 50Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-gcs
  csi:
    driver: gcs.csi.ofek.dev
    volumeHandle: csi-gcs
    volumeAttributes:
      flags:
        uid: "1000"
        gid: "1000"
        dir-mode: "775"
        file-mode: "664"
    nodePublishSecretRef:
      name: csi-gcs-secret
      namespace: csi-gcs

or maybe this:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: csi-gcs-pv
  namespace: liferay7
  annotations:
    pv.beta.kubernetes.io/gid: "1000"
spec:
  accessModes:
    ReadWriteMany
  capacity:
    storage: 50Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-gcs
  csi:
    driver: gcs.csi.ofek.dev
    volumeHandle: csi-gcs
    volumeAttributes:
      flags:
        - '--uid="1000"'
        - '--gid="1000"'
        - '--dir-mode="775"'
        - '--file-mode="664"'
    nodePublishSecretRef:
      name: csi-gcs-secret
      namespace: csi-gcs

@epalazuelosb
Copy link
Author

I've got a validation error in both cases:

error validating data: ValidationError(PersistentVolume.spec.csi.volumeAttributes.flags): invalid type for io.k8s.api.core.v1.CSIPersistentVolumeSource.volumeAttributes: got "array", expected "string"; if you choose to ignore these errors, turn validation off with --validate=false

But the previous one I have applied was correct:

    volumeAttributes:
      uid: "1000"
      gid: "1000"
      dir-mode: "775"
      file-mode: "664"

So it looks that is correctly configured, but not correctly mounted. Maybe the mistake is on the deployment side?

@maennchen
Copy link
Collaborator

@epalazuelosb The volume attributes you mentioned are not being read. They have no effect.

Try with this?

apiVersion: v1
kind: PersistentVolume
metadata:
  name: csi-gcs-pv
  namespace: liferay7
  annotations:
    pv.beta.kubernetes.io/gid: "1000"
spec:
  accessModes:
    ReadWriteMany
  capacity:
    storage: 50Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-gcs
  csi:
    driver: gcs.csi.ofek.dev
    volumeHandle: csi-gcs
    volumeAttributes:
      flags: '--uid="1000" --gid="1000" --dir-mode="775" --file-mode="664"'
    nodePublishSecretRef:
      name: csi-gcs-secret
      namespace: csi-gcs

@epalazuelosb
Copy link
Author

epalazuelosb commented Apr 21, 2020

@epalazuelosb The volume attributes you mentioned are not being read. They have no effect.

Try with this?

apiVersion: v1
kind: PersistentVolume
metadata:
  name: csi-gcs-pv
  namespace: liferay7
  annotations:
    pv.beta.kubernetes.io/gid: "1000"
spec:
  accessModes:
    ReadWriteMany
  capacity:
    storage: 50Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-gcs
  csi:
    driver: gcs.csi.ofek.dev
    volumeHandle: csi-gcs
    volumeAttributes:
      flags: '--uid="1000" --gid="1000" --dir-mode="775" --file-mode="664"'
    nodePublishSecretRef:
      name: csi-gcs-secret
      namespace: csi-gcs

Now it looks we have something:

csi-gcs-test-8684576d4d-l8d6r	Pod	MountVolume.SetUp failed for volume "csi-gcs-pv" : rpc error: code = Internal desc = Error (code exit status 1) mounting bucket ssgs-cms-document-library-sb:
Incorrect Usage. invalid value "\"1000\"" for flag -uid: parse error

NAME:
   gcsfuse - Mount a GCS bucket locally

USAGE:
   gcsfuse [global options] bucket mountpoint
   
VERSION:
   0.27.0 (Go version go1.13.6)
   
GLOBAL OPTIONS:
   --foreground                 Stay in the foreground after mounting.
   -o value                     Additional system-specific mount options. Be careful!
   --dir-mode value             Permissions bits for directories, in octal. (default: 755)
   --file-mode value            Permission bits for files, in octal. (default: 644)
   --uid value                  UID owner of all inodes. (default: -1)
   --gid value                  GID owner of all inodes. (default: -1)
   --implicit-dirs              Implicitly define directories based on content. See docs/semantics.md
   --only-dir value             Mount only the given directory, relative to the bucket root.
   --billing-project value      Project to use for billing when accessing requester pays buckets. (default: none)
   --key-file value             Absolute path to JSON key file for use with GCS. (default: none, Google application default credentials used)
   --limit-bytes-per-sec value  Bandwidth limit for reading data, measured over a 30-second window. (use -1 for no limit) (default: -1)
   --limit-ops-per-sec value    Operations per second limit, measured over a 30-second window (use -1 for no limit) (default: 5)
   --stat-cache-capacity value  How many entries can the stat cache hold (impacts memory consumption) (default: 4096)
   --stat-cache-ttl value       How long to cache StatObject results and inode attributes. (default: 1m0s)
   --type-cache-ttl value       How long to cache name -> file/dir mappings in directory inodes. (default: 1m0s)
   --temp-dir value             Absolute path to temporary directory for local GCS object copies. (default: system default, likely /tmp)
   --debug_fuse                 Enable fuse-related debugging output.
   --debug_gcs                  Print GCS request and timing information.
   --debug_http                 Dump HTTP requests and responses to/from GCS.
   --debug_invariants           Panic when internal invariants are violated.
   --help, -h                   show help
   --version, -v                print the version
   
invalid value "\"1000\"" for flag -uid: parse error
	FailedMount	Warning	2020-04-21T11:40:58Z

@epalazuelosb
Copy link
Author

Once corrected the quotes it looks to work correctly! Let me check it

@epalazuelosb
Copy link
Author

epalazuelosb commented Apr 21, 2020

Confirmed. The solution provided by @maennchen:

    volumeAttributes:
      flags: '--uid="1000" --gid="1000" --dir-mode="775" --file-mode="664"'

Worked correctly.

edit: fixed typo

@ofek
Copy link
Owner

ofek commented Apr 23, 2020

@epalazuelosb
Copy link
Author

epalazuelosb commented Apr 23, 2020

I have realized that the mount is not working properly:

I0423 18:24:44.766837       1 driver.go:65] Method /csi.v1.Node/NodeUnpublishVolume completed
I0423 18:24:44.850083       1 node.go:155] Method NodeGetCapabilities called with: 
I0423 18:24:44.850153       1 driver.go:65] Method /csi.v1.Node/NodeGetCapabilities completed
I0423 18:05:01.791525       1 driver.go:65] Method /csi.v1.Node/NodeGetCapabilities completed
I0423 18:05:01.997528       1 node.go:21] Method NodePublishVolume called with: {"secrets":"***stripped***","target_path":"/var/lib/kubelet/pods/f36fe5fb-858c-11ea-801d-42010a840156/volumes/kubernetes.io~csi/csi-gcs-pv/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":5}},"volume_context":{"flags":"--uid=1000 --gid=1000 --dir-mode=775 --file-mode=664"},"volume_id":"csi-gcs"}
I0423 18:05:01.998759       1 node.go:60] Using secret name 'key' as the service account key
I0423 18:05:01.998771       1 node.go:67] Saving key contents to a temporary location
I0423 18:05:01.998878       1 node.go:87] Mounting bucket my-bucket to /var/lib/kubelet/pods/f36fe5fb-858c-11ea-801d-42010a840156/volumes/kubernetes.io~csi/csi-gcs-pv/mount with: [--key-file=/tmp/keys/424969854 -o=allow_other --uid=1000 --gid=1000 --dir-mode=775 --file-mode=664]
I0423 18:05:02.194302       1 driver.go:65] Method /csi.v1.Node/NodePublishVolume completed
I0423 18:07:09.871255       1 node.go:108] Method NodeUnpublishVolume called with: volume_id:"csi-gcs" target_path:"/var/lib/kubelet/pods/f36fe5fb-858c-11ea-801d-42010a840156/volumes/kubernetes.io~csi/csi-gcs-pv/mount" 
I0423 18:07:09.896261       1 driver.go:65] Method /csi.v1.Node/NodeUnpublishVolume completed
I0423 18:07:09.973046       1 node.go:155] Method NodeGetCapabilities called with: 
I0423 18:07:09.973093       1 driver.go:65] Method /csi.v1.Node/NodeGetCapabilities completed

I can create directories and files from pods, I can list files created by other users. I cannot see directories created by other users unless I create a directory with the same name. In that case I can see the directory and its contents.

@maennchen
Copy link
Collaborator

@epalazuelosb Can you enable implicit dirs?

@ofek
Copy link
Owner

ofek commented Apr 23, 2020

Also fyi flags doesn't do anything on 0.3.0+ #6 (comment)

@ofek
Copy link
Owner

ofek commented Apr 24, 2020

@epalazuelosb Any luck?

@epalazuelosb
Copy link
Author

epalazuelosb commented Apr 24, 2020

Excuse me. No, I cannot deploy the PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: csi-gcs-pv
  annotations:
    pv.beta.kubernetes.io/gid: "1000"
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 50Gi
    mountOptions:
    - --uid=1000
      --gid=1000
      --dir-mode=0775
      --file-mode=0664
      --implicit-dirs="document_library"
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-gcs
  csi:
    driver: gcs.csi.ofek.dev
    volumeHandle: csi-gcs
    volumeAttributes:
      gid: "1000"
      uid: "1000"
      dirMode: "0775"
      fileMode: "0664"
      implicitDirs: "document_library"
    nodePublishSecretRef:
      name: csi-gcs-secret
error: error validating "csi-gcs-pv.yaml": error validating data: ValidationError(PersistentVolume.spec.capacity.mountOptions): invalid type for io.k8s.apimachinery.pkg.api.resource.Quantity: got "array", expected "string"; if you choose to ignore these errors, turn validation off with --validate=false

@ofek
Copy link
Owner

ofek commented Apr 24, 2020

While mountOptions should be under spec like:

apiVersion: v1
kind: PersistentVolume
spec:
  mountOptions:
  - --gid=63147
  - --dir-mode=0775
  - --file-mode=0664

that shouldn't matter since volumeAttributes looks right and should take precedence anyway.

@maennchen Any thoughts?

@maennchen
Copy link
Collaborator

@ofek The yml above is not valid. Will not work therefore.

@ofek
Copy link
Owner

ofek commented Apr 24, 2020

Oh, yes. @epalazuelosb try removing mountOptions

@epalazuelosb
Copy link
Author

epalazuelosb commented Apr 24, 2020

Excuse, I have tried it before with the same result:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: csi-gcs-pv
  annotations:
    pv.beta.kubernetes.io/gid: "1000"
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 50Gi
    mountOptions:
    - --uid=1000
    - --gid=1000
    - --dir-mode=0775
    - --file-mode=0664
    - --implicit-dirs="document_library"
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-gcs
  csi:
    driver: gcs.csi.ofek.dev
    volumeHandle: csi-gcs
    volumeAttributes:
      gid: "1000"
      uid: "1000"
      dirMode: "0775"
      fileMode: "0664"
      implicitDirs: "document_library"
    nodePublishSecretRef:
      name: csi-gcs-secret
error: error validating "csi-gcs-pv.yaml": error validating data: ValidationError(PersistentVolume.spec.capacity.mountOptions): invalid type for io.k8s.apimachinery.pkg.api.resource.Quantity: got "array", expected "string"; if you choose to ignore these errors, turn validation off with --validate=false

@epalazuelosb
Copy link
Author

Oh, yes. @epalazuelosb try removing mountOptions

Ok, I am trying right now

@ofek
Copy link
Owner

ofek commented Apr 24, 2020

mountOptions doesn't belong under capacity

@epalazuelosb
Copy link
Author

epalazuelosb commented Apr 24, 2020

Now the PVC cannot be provisioned:

csi-gcs-pvc PersistentVolumeClaim waiting for a volume to be created, either by external provisioner "gcs.csi.ofek.dev" or manually created by system administrator ExternalProvisioning Normal 2020-04-24T16:33:14Z

persistentvolume/csi-gcs-pv 50Gi RWX Retain Available csi-gcs 5s storageclass.storage.k8s.io/csi-gcs gcs.csi.ofek.dev 29h

@ofek
Copy link
Owner

ofek commented Apr 24, 2020

@epalazuelosb
Copy link
Author

epalazuelosb commented Apr 24, 2020

Excuse me. I have just updated the csi-gcs deployment. Now it looks to work, but the directory "document_library" created externally stills hidden for the pod:

I0424 16:54:19.123774       1 driver.go:59] Starting Google Cloud Storage CSI Driver - driver: `gcs.csi.ofek.dev`, version: `0.3.0`, gRPC socket: `unix:///csi/csi.sock`
I0424 16:54:19.460643       1 identity.go:36] Method Probe called with: 
I0424 16:54:19.460700       1 driver.go:52] Method /csi.v1.Identity/Probe completed
I0424 16:54:19.461367       1 identity.go:11] Method GetPluginInfo called with: 
I0424 16:54:19.461393       1 driver.go:52] Method /csi.v1.Identity/GetPluginInfo completed
I0424 16:54:19.461884       1 identity.go:20] Method GetPluginCapabilities called with: 
I0424 16:54:19.461923       1 driver.go:52] Method /csi.v1.Identity/GetPluginCapabilities completed
I0424 16:54:19.462333       1 controller.go:172] Method ControllerGetCapabilities called with: {}
I0424 16:54:19.463493       1 driver.go:52] Method /csi.v1.Controller/ControllerGetCapabilities completed
I0424 16:54:17.406222       1 driver.go:59] Starting Google Cloud Storage CSI Driver - driver: `gcs.csi.ofek.dev`, version: `0.3.0`, gRPC socket: `unix:///csi/csi.sock`
I0424 16:54:17.376383       1 driver.go:59] Starting Google Cloud Storage CSI Driver - driver: `gcs.csi.ofek.dev`, version: `0.3.0`, gRPC socket: `unix:///csi/csi.sock`
I0424 16:54:17.405723       1 driver.go:59] Starting Google Cloud Storage CSI Driver - driver: `gcs.csi.ofek.dev`, version: `0.3.0`, gRPC socket: `unix:///csi/csi.sock`
I0424 16:56:21.679362       1 mount_linux.go:153] Detected OS without systemd
I0424 16:56:21.689894       1 mount_linux.go:153] Detected OS without systemd
I0424 16:56:21.700517       1 mount_linux.go:153] Detected OS without systemd
I0424 16:56:21.711005       1 mount_linux.go:153] Detected OS without systemd
I0424 16:56:21.721581       1 mount_linux.go:153] Detected OS without systemd
E0424 16:56:21.722170       1 mounter.go:82] [liferay7-document-library] Mounting file system...
E0424 16:56:21.723746       1 mounter.go:82] [liferay7-document-library] File system has been successfully mounted.
I0424 16:56:21.732226       1 mount_linux.go:153] Detected OS without systemd
I0424 16:56:21.732608       1 driver.go:52] Method /csi.v1.Node/NodePublishVolume completed
E0424 16:57:43.279659       1 mounter.go:82] [my-bucket] fuse: 2020/04/24 16:57:43.279524 *fuseops.GetXattrOp error: function not implemented
I0424 16:54:17.637635       1 driver.go:59] Starting Google Cloud Storage CSI Driver - driver: `gcs.csi.ofek.dev`, version: `0.3.0`, gRPC socket: `unix:///csi/csi.sock`
I0424 16:54:18.000474       1 driver.go:59] Starting Google Cloud Storage CSI Driver - driver: `gcs.csi.ofek.dev`, version: `0.3.0`, gRPC socket: `unix:///csi/csi.sock`
I0424 16:54:17.619218       1 driver.go:59] Starting Google Cloud Storage CSI Driver - driver: `gcs.csi.ofek.dev`, version: `0.3.0`, gRPC socket: `unix:///csi/csi.sock`
I0424 16:54:18.030454       1 driver.go:59] Starting Google Cloud Storage CSI Driver - driver: `gcs.csi.ofek.dev`, version: `0.3.0`, gRPC socket: `unix:///csi/csi.sock`
I0424 16:54:17.363967       1 driver.go:59] Starting Google Cloud Storage CSI Driver - driver: `gcs.csi.ofek.dev`, version: `0.3.0`, gRPC socket: `unix:///csi/csi.sock`

@epalazuelosb
Copy link
Author

Found the mistake on implicitDirs flag:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: csi-gcs-pv
  annotations:
    pv.beta.kubernetes.io/gid: "1000"
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 50Gi
  persistentVolumeReclaimPolicy: Retain
  storageClassName: csi-gcs
  csi:
    driver: gcs.csi.ofek.dev
    volumeHandle: csi-gcs
    volumeAttributes:
      gid: "1000"
      uid: "1000"
      dirMode: "0775"
      fileMode: "0664"
      implicitDirs: "true"
    nodePublishSecretRef:
      name: csi-gcs-secret

Thank you for your help. Now it works correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants