9p2000.L mount breaks symbolic link creation: Operation not permitted

[yinzara]

Minikube version: v1.2.0

Environment:

OS : MacOS v10.15.5
VM Driver: hyperkit
Docker version (e.g. docker -v): 18.09.2, build 6247962
Install tools: brew
What happened:
I received an error "ln: failed to create symbolic link '/test/bar': Operation not permitted"

What you expected to happen:
Create a successful symbolic link

How to reproduce it (as minimally and precisely as possible):

cd $HOME
touch foo
minikube start --vm-driver hyperkit --mount --mount-string $HOME:/test
minikube ssh -- ln -s /test/foo /test/bar

Anything else do we need to know:
This is important to me because I try and do things like npm install inside the container (for local development), and I'll get errors about it not being able to make symbolic links.

This reopens issue #890

This is happening with the Hyperkit driver but I suspect may happen with other drivers as well.

[yinzara]

Very interesting find seems to relate to:
#3796

The following works:

cd $HOME
touch foo
minikube start --vm-driver hyperkit
minikube mount --9p-version 9p2000.u $HOME:/test &
minikube ssh -- sudo ln -s /test/foo /test/bar

This means that there was some defect introduced with the switch to version L that prevents symbolic links from functioning.

[tstromberg]

I can confirm this on kvm2 & Linux. The current mount interface is driver agnostic.

Thanks for discovering a workaround! I'll add this to the long list of bugs related to go9p. We really need an alternative approach for mounting files.

[yinzara]

Yeah I would love to see an alternative. Unfortunately the work around above doesn't really help very much because the 9p2000.u version has issues with permissions, so it introduces a host of other problems.

[tstromberg]

Still an issue in v1.4.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[yinzara]

/remove-lifecycle stale

still a problem AFAIK

[hadrien-toma]

Also experiencing this on Linux (kvm2, none and virtualbox) 😢

kubectl version: 1.17.0
minikube version: 1.6.2

[medyagh]

@hadrien-toma do you mind trying with our latest release > 1.7.3 ?

[priyawadhwa]

@hadrien-toma were you able to upgrade and test this out?

[hadrien-toma]

Sorry for the delay, so from the test I just ran today:

Status

• kvm2 and virtualbox, issue is still there:

ln: failed to create symbolic link '/test/bar': Operation not permitted
ssh: exit status 1

• none, not tested because of:

  minikube v1.9.2 on Ubuntu 18.04
✨  Using the none driver based on user configuration

❗  'none' driver reported an issue: the 'none' driver must be run as the root user
💡  Suggestion: For non-root usage, try the newer 'docker' driver

💣  Sorry, Kubernetes v1.18.0 requires conntrack to be installed in root's path

Context

>  cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"

> docker --version
Docker version 19.03.8, build afacb8b7f0

> minikube version
minikube version: v1.9.2
commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393

> kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:50:46Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[sharifelgamal]

I suspect this is still an issue, I'm going to freeze this so it doesn't get closed.

[rnantes]

Im getting the same issue with minikube v1.16.0 on Darwin(macOS) 11.1 using the hyperkit driver

[jrgleason]

Getting similar issue in 1.14.2

2021-02-19 03:53:24.760 UTC [30] LOG:  could not link file "pg_wal/xlogtemp.30" to "pg_wal/000000010000000000000001": Operation not permitted
2021-02-19 03:53:24.761 UTC [30] FATAL:  could not open file "pg_wal/000000010000000000000001": No such file or directory

Darwin ....local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64 x86_64

More info here

https://stackoverflow.com/questions/66271590/how-do-i-cofigure-the-volume-mounts-for-a-postgres-kubernetes-container-osx

[Pictor13]

Any progress or idea on how to fix this issue?

I've been damned by this problem for the last year and half.
It creates issues that require lot of investment in debugging, to then figure out that it's a Minikube regression.
And there is not a clear solution; that makes it even more daunting :(

I am not sure what the parameter is about, but shouldn't it be at least mentioned somewhere in Minikube documentation, in case it is not going to be solved?

[sharifelgamal]

This seems to be an issue with 9p itself, upgrading the version broke symlinks. Our only real option is to change our mounting library, which would be a nontrivial undertaking.

[lubumbax]

This seems to be an issue with 9p itself, upgrading the version broke symlinks. Our only real option is to change our mounting library, which would be a nontrivial undertaking.

Somebody in this thread #4621 (comment) mentions having reproduced it with drivers kvm2 and virtualbox.
If that's so, that suggests that the issue is not only related to 9P in Hyperkit (though, personally I think it is really a 9P filesystem limitation).

[KatekovAnton]

Hi. I am trying to compile a simple c++ app in docker, but build is failing due to this issue. Log:

[1/369] Creating library symlink lib/libPocoFoundation.so
FAILED: lib/libPocoFoundation.so
/usr/bin/cmake -E cmake_symlink_library lib/libPocoFoundation.so.81 lib/libPocoFoundation.so.81 lib/libPocoFoundation.so && :
CMake Error: failed to create symbolic link 'lib/libPocoFoundation.so': operation not permitted
CMake Error: cmake_symlink_library: System Error: Operation not permitted
[2/369] Building CXX object third_party/poco/Encodings/CMakeFiles/Encodings.dir/src/Windows932Encoding.cpp.o
[3/369] Building CXX object third_party/poco/Encodings/CMakeFiles/Encodings.dir/src/Windows874Encoding.cpp.o
ninja: build stopped: subcommand failed.

Is there any workaround? Thank you.

[afbjorklund]

Is there any workaround? Thank you.

Use a directory on the VM, that is not mounted from remote (host). For instance, by copying the directory to a different location.