Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document how to fix file permission errors while using odo with GKE/AKS/EKS #6840

Merged
merged 6 commits into from
May 30, 2023
Merged

Document how to fix file permission errors while using odo with GKE/AKS/EKS #6840

Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a63701d
Document how to fix storage permission issues with GKE/AKS/EKS
valaparthvi May 22, 2023
61f0033
Temp Change
valaparthvi May 24, 2023
3ede01e
First draft
valaparthvi May 24, 2023
03daf4d
Apply suggestions from code review
valaparthvi May 26, 2023
c672612
Remove sourceMapping documentation
valaparthvi May 26, 2023
d5eb9ba
Other code review changes
valaparthvi May 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
322 changes: 322 additions & 0 deletions docs/website/docs/user-guides/advanced/using-odo-with-other-clusters.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,322 @@
---
title: Troubleshoot Storage Permission issues on GKE/AKS/EKS cluster
valaparthvi marked this conversation as resolved.
Show resolved Hide resolved
sidebar_position: 9
---

Using odo to run an application on a GKE/AKS/EKS cluster does not always work out of the box, especially while using Devfiles from the [Devfile Registry](https://registry.devfile.io); users often encounter issues while syncing local files into the container due to insufficient permissions on mounted volumes.
valaparthvi marked this conversation as resolved.
Show resolved Hide resolved

<details>
<summary>For example, while running a Java Maven application using a <code>java-maven</code> devfile on an Amazon Elastic Kubernetes Service, a sample error may look like this.</summary>

```shell
$ odo dev
__
/ \__ Developing using the "java-springboot-starter" Devfile
\__/ \ Namespace: default
/ \__/ odo version: v3.6.0
\__/

↪ Running on the cluster in Dev mode
• Waiting for Kubernetes resources ...
✓ Added storage m2 to component
⚠ Pod is Pending
✓ Pod is Running
◑ Syncing files into the container ✗ Command 'tar xf - -C /projects --no-same-owner' in container failed.

✗ stdout:

✗ stderr: tar: src: Cannot mkdir: Permission denied
tar: src/main/resources/application.properties: Cannot open: No such file or directory
tar: HELP.md: Cannot open: Permission denied
tar: mvnw: Cannot open: Permission denied
tar: devfile.yaml: Cannot open: Permission denied
tar: mvnw.cmd: Cannot open: Permission denied
tar: pom.xml: Cannot open: Permission denied
tar: src: Cannot mkdir: Permission denied
tar: src/main/java/com/example/demo/DemoApplication.java: Cannot open: No such file or directory
tar: .gitignore: Cannot open: Permission denied
tar: src: Cannot mkdir: Permission denied
tar: src/test/java/com/example/demo/DemoApplicationTests.java: Cannot open: No such file or directory
tar: Exiting with failure status due to previous errors


✗ err: error while streaming command: command terminated with exit code 2

✗ Syncing files into the container [610ms]
Error occurred on Push - watch command was unable to push component: failed to sync to component with name java-springboot-starter: failed to sync to component with name java-springboot-starter: unable push files to pod: error while streaming command: command terminated with exit code 2

◐ Syncing files into the container ✗ Command 'tar xf - -C /projects --no-same-owner' in container failed.

✗ stdout:

✗ stderr: tar: src: Cannot mkdir: Permission denied
tar: src/main/resources/application.properties: Cannot open: No such file or directory
tar: src: Cannot mkdir: Permission denied
tar: src/test/java/com/example/demo/DemoApplicationTests.java: Cannot open: No such file or directory
tar: devfile.yaml: Cannot open: Permission denied
tar: src: Cannot mkdir: Permission denied
tar: src/main/java/com/example/demo/DemoApplication.java: Cannot open: No such file or directory
tar: pom.xml: Cannot open: Permission denied
tar: .gitignore: Cannot open: Permission denied
tar: mvnw.cmd: Cannot open: Permission denied
tar: HELP.md: Cannot open: Permission denied
tar: mvnw: Cannot open: Permission denied
tar: Exiting with failure status due to previous errors


✗ err: error while streaming command: command terminated with exit code 2
```

</details>

<details>
<summary>Or, while running a Go application with <code>go</code> devfile on an Azure Kubernetes Service may end up in an error like this.</summary>

```shell
$ odo dev
__
/ \__ Developing using the "places" Devfile
\__/ \ Namespace: default
/ \__/ odo version: v3.10.0
\__/

⚠ You are using "default" namespace, odo may not work as expected in the default namespace.
⚠ You may set a new namespace by running `odo create namespace <name>`, or set an existing one by running `odo set namespace <name>`

↪ Running on the cluster in Dev mode
• Waiting for Kubernetes resources ...
⚠ Pod is Pending
✓ Pod is Running
◐ Syncing files into the container ✗ Command 'tar xf - -C /projects --no-same-owner' in container failed.

✗ stdout:

✗ stderr: tar: main.go: Cannot open: Permission denied
tar: .gitignore: Cannot open: Permission denied
tar: README.md: Cannot open: Permission denied
tar: devfile.yaml: Cannot open: Permission denied
tar: go.mod: Cannot open: Permission denied
tar: Exiting with failure status due to previous errors


✗ err: error while streaming command: command terminated with exit code 2

✗ Syncing files into the container [4s]
Error occurred on Push - watch command was unable to push component: failed to sync to component with name places: failed to sync to component with name places: unable push files to pod: error while streaming command: command terminated with exit code 2


↪ Dev mode
Status:
Watching for changes in the current directory /tmp/go-app

Keyboard Commands:
[Ctrl+c] - Exit and delete resources from the cluster
[p] - Manually apply local changes to the application on the cluster
^CCleaning resources, please wait
✗ Dev mode interrupted by user
```

</details>

Various factors are responsible for this:
* Storage Provisioner used for the cluster
* User set by the container image
* Location on the container where the files are to be synced
* Using Ephemeral vs Non-Ephemeral Volumes

Users may encounter storage related permissions issues even while working on a standard Kubernetes or OpenShift cluster.

This guide will discuss some workarounds that can be used to fix these issues.

### Using Ephemeral Volumes
This is the simplest way to overcome this issue. There are 2 parts to this solution:
1. Set `odo` preference `Ephemeral` to _true_.

```shell
odo preference set Ephemeral true -f
```
2. If the Devfile contains a `volume` component, then set its `ephemeral` property to `true`.
rm3l marked this conversation as resolved.
Show resolved Hide resolved
The above configuration will use the [`emptyDir` Ephemeral volumes](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) instead of creating Persistent Volumes to mount the source files; it also ensures the current user can read/write to the directories.
valaparthvi marked this conversation as resolved.
Show resolved Hide resolved

### Define custom location to mount volumes and project source
All the Devfiles provided by the [Devfile Registry](https://registry.devfile.io) use container image set a non-root user, which is why we see these permission related issues; we usually do not see this for a root user.
valaparthvi marked this conversation as resolved.
Show resolved Hide resolved

When a user does not have access to all the locations in a system, we need to provide a way to use a location where the user can read/write. There are 2 parts to this solution as well:
1. Set `sourceMapping` of the Devfile `container` component to a location where the user has read/write access.
2. If a Devfile `volume` component is defined, then ensure that the `.volumeMounts.path` of the Devfile container component where the volume will be mount is in a location where the user has read/write access.
valaparthvi marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this solution, the files will be written in the container's filesystem, which is not optimal:

For example, when setting sourceMapping to java-maven-app as in the example below, this directory will be created in a directory outside of /projects (in the current directory specified in the container's image).

If you set sourceMapping to /projects/java-maven-app, you get the same permission problems.

This solution inhibits the use of a volume (either persistent or volatile) for the directory where the sources are placed, as we won't use this volume.


<details>
<summary>Example <code>java-maven</code> Devfile with custom <code>sourceMapping</code> and volume mount location.</summary>

```yaml showLineNumbers
commands:
- exec:
# highlight-next-line
commandLine: mvn -Dmaven.repo.local=${HOME}/.m2/repository package # <---- This is to ensure the correct m2 location is used
component: tools
group:
isDefault: true
kind: build
workingDir: ${PROJECT_SOURCE}
id: mvn-package
- exec:
commandLine: java -jar target/*.jar
component: tools
group:
isDefault: true
kind: run
workingDir: ${PROJECT_SOURCE}
id: run
- exec:
commandLine: java -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=${DEBUG_PORT},suspend=n
-jar target/*.jar
component: tools
group:
isDefault: true
kind: debug
workingDir: ${PROJECT_SOURCE}
id: debug
components:
- container:
command:
- tail
- -f
- /dev/null
endpoints:
- name: http-maven
targetPort: 8080
- exposure: none
name: debug
targetPort: 5858
env:
- name: DEBUG_PORT
value: "5858"
image: registry.access.redhat.com/ubi8/openjdk-11:latest
memoryLimit: 512Mi
mountSources: true
# highlight-next-line
sourceMapping: java-maven-app # <---- This will mount sources to <WORKDIR>/java-maven-app directory
volumeMounts:
- name: m2
# highlight-next-line
path: .m2 # <---- This will mount the volume to <WORKDIR>/.m2 directory
name: tools
- name: m2
volume: {}
metadata:
description: Java application based on Maven 3.6 and OpenJDK 11
displayName: Maven Java
icon: https://raw.githubusercontent.com/devfile-samples/devfile-stack-icons/main/java-maven.jpg
language: Java
name: jmaven-app
projectType: Maven
tags:
- Java
- Maven
version: 1.2.0
schemaVersion: 2.1.0
starterProjects:
- git:
remotes:
origin: https://github.com/odo-devfiles/springboot-ex.git
name: springbootproject
```
</details>

In the example above, we set `sourceMapping` to `java-maven-app`. Since the path provided is a relative one, this will sync all the files to `$HOME/java-maven-app`. If an absolute path was provided, it would use the path as it is.

We have also provided a relative path for `volumeMounts.path`, so the volume will be mounted to `${HOME}/.m2` location.
While using relative paths, the user must ensure to update all the usages of this location within the devfile. For example, we have modified the Devfile `build` command _mvn-package_ to use the `${HOME}/.m2/repository` location.


### Setting `fsGroup` to the PodSecurityContext
By setting `fsGroup` in the PodSecurityContext, all processes of the container are also made part of the supplementary group ID set in the field. The owner for volume mount location and any files created in that volume will be Group ID set in the field. This solution is quite common when looking for permission related issues on a mounted volume, [example](https://stackoverflow.com/questions/50156124/kubernetes-nfs-persistent-volumes-permission-denied#50187723).

This solution can be implemented by setting a `pod-overrides` attribute to the Devfile `container` component.
valaparthvi marked this conversation as resolved.
Show resolved Hide resolved

<details>
<summary>Example <code>java-maven</code> Devfile with a <code>fsGroup</code> set in PodSecurityContext.</summary>

```yaml showLineNumbers
commands:
- exec:
commandLine: mvn -Dmaven.repo.local=/home/user/.m2/repository package
component: tools
group:
isDefault: true
kind: build
workingDir: ${PROJECT_SOURCE}
id: mvn-package
- exec:
commandLine: java -jar target/*.jar
component: tools
group:
isDefault: true
kind: run
workingDir: ${PROJECT_SOURCE}
id: run
- exec:
commandLine: java -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=${DEBUG_PORT},suspend=n
-jar target/*.jar
component: tools
group:
isDefault: true
kind: debug
workingDir: ${PROJECT_SOURCE}
id: debug
components:
- container:
command:
- tail
- -f
- /dev/null
endpoints:
- name: http-maven
targetPort: 8080
- exposure: none
name: debug
targetPort: 5858
env:
- name: DEBUG_PORT
value: "5858"
image: registry.access.redhat.com/ubi8/openjdk-11:latest
memoryLimit: 512Mi
mountSources: true
volumeMounts:
- name: m2
path: /home/user/.m2
name: tools
# highlight-start
attributes:
pod-overrides:
spec:
securityContext:
fsGroup: 2000
# highlight-end
- name: m2
volume: {}
metadata:
description: Java application based on Maven 3.6 and OpenJDK 11
displayName: Maven Java
icon: https://raw.githubusercontent.com/devfile-samples/devfile-stack-icons/main/java-maven.jpg
language: Java
name: jmaven-app
projectType: Maven
tags:
- Java
- Maven
version: 1.2.0
schemaVersion: 2.1.0
starterProjects:
- git:
remotes:
origin: https://github.com/odo-devfiles/springboot-ex.git
name: springbootproject
```
</details>


But this solution may not always be feasible, especially while dealing with large filesystems.
>Be cautious with the use of fsGroup. The changing of group ownership of an entire volume can cause pod startup delays for slow and/or large filesystems. It can also be detrimental to other processes that share the same volume if their processes do not have access permissions to the new GID. For this reason, some providers for shared file systems such as NFS do not implement this functionality. These settings also do not affect ephemeral volumes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second part of the warning (sharing the volume with other processes) is not related to how odo uses the volume, as odo is creating a dedicated volume for the component, which won't be shared with others.

For the first part of the warning, we can explain that this will have an effect only at the "restart" of the pod (when the user modifies the container's information in the Devfile). In a normal use (without modifying the container in Devfile), there won't be any effect, as the Volume is empty when it is mounted the first time, and only populated with the sync.

These warnings are valid for containers that mount volumes with persistent data (imagine a Postgres image mounting a 1Tb volume with its data), but I think we can mitigate them in the case of odo.

>
> Read these articles by [Synk](https://snyk.io/blog/10-kubernetes-security-context-settings-you-should-understand/) and [Google Cloud](https://cloud.google.com/kubernetes-engine/docs/troubleshooting/troubleshooting-gke-storage#mounting_a_volume_stops_responding_due_to_the_fsgroup_setting) to learn more about it.