Files missing within minikube mounted volume

[georgecrawford]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Minikube version v0.19.0

Environment:

• OS (e.g. from /etc/os-release): MacOS Sierra 10.12.4
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): xhyve
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): minikube-v0.18.0.iso (NB: this contradicts this comment)
• Install tools:
• Others: docker-machine-driver-xhyve version 0.3.2

What happened:
On creating a fresh VM (minikube start --vm-driver=xhyve), I can see my /Users directory mounted at the same location on the VM (minikube ssh). However, I've encountered a directory where the file count is wrong:

ls -l /Users/george/[MYDIR]/ | wc && minikube ssh -- ls -l /Users/george/[MYDIR]/ | wc
      123     123    5620
      121     121    5547

I don't know how to explore this further. If there are fewer than 70 files in the directory, the numbers are the same. However, at some point over 70, the numbers diverge, as if the mount inside the minikube VM has simply not seen some of the files. They all have identical permissions.

What you expected to happen:
The directory count in minikube VM and my host Mac should always be the same

How to reproduce it (as minimally and precisely as possible):
I'm not sure. I've tried to reproduce it by creating empty files in a directory, but I can't recreate the bug. I've only seen it in one particular directory, but it doesn't seem to matter which of the files are in the directory - as soon as there are more than ~70, I see incorrect counts.

Anything else do we need to know:
I've cross-posted this here, as I wasn't sure if it was a problem with xhyve or minikube. Very happy to close one of them as a appropriate.

If it's important, the directory is mounted into a pod using a hostPath volume.

[georgecrawford]

NB: cross-posted here: machine-drivers/docker-machine-driver-xhyve#176

[georgecrawford]

I'm still suffering from this same problem with minikube ISO v0.20.0 and docker-machine-driver-xhyve v0.3.3. Is there any other information I can provide to get to the bottom of the issue?

[r2d4]

The default /Users mount is coming from xhyve. Can you try to see if the same problem exists with minikube mount? The 9p filesystem implementations we use are different, so its possible this bug doesn't exist in the implementation we use.

[georgecrawford]

mount doesn't seem to work at all for me

# OS X Host:
$ ls -l test | grep -v total | wc -l
126

# Minikube VM:
$ ls -l /Users/[path to...]/test/ | grep -v total | wc -l
124

# OS X Host
$ minikube mount $(pwd)/test:/test
Mounting /Users/[path to...]/test into /test on the minikube VM
This daemon process needs to stay alive for the mount to still be accessible...
ufs starting

# Minikube VM
$ ls -l /test | grep -v total | wc -l
ls: reading directory '/test': Unknown error 526
0

[georgecrawford]

Could be related to #1563 and #1562.

$ minikube mount $(pwd)/test:/test --v 10
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:52322
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .DriverName
Mounting /Users/[SNIP]/test into /test on the minikube VM
This daemon process needs to stay alive for the mount to still be accessible...
ufs starting
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:52327
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/george/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.35 -o IdentitiesOnly=yes -i /Users/george/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
sudo umount /test;
SSH cmd err, output: exit status 32: umount: /test: mountpoint not found

(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/george/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.35 -o IdentitiesOnly=yes -i /Users/george/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:

sudo mkdir -p /test || true;
sudo mount -t 9p -o trans=tcp -o port=52324 -o uid=1001 -o gid=1001 192.168.64.1 /test;
sudo chmod 775 /test;
2017/07/10 10:43:23 connected
2017/07/10 10:43:23 >>> 192.168.64.35:40334 Tversion tag 65535 msize 8192 version '9P2000.L'
2017/07/10 10:43:23 <<< 192.168.64.35:40334 Rversion tag 65535 msize 8192 version '9P2000'
2017/07/10 10:43:23 >>> 192.168.64.35:40334 Tattach tag 1 fid 0 afid 4294967295 uname 'nobody' nuname 0 aname ''
2017/07/10 10:43:23 <<< 192.168.64.35:40334 Rattach tag 1 aqid (1af9adb 1e4819e0 'd')
2017/07/10 10:43:23 >>> 192.168.64.35:40334 Tstat tag 1 fid 0
2017/07/10 10:43:23 <<< 192.168.64.35:40334 Rstat tag 1 st ('test' 'george' '20' '' q (1af9adb 1e4819e0 'd') m d775 at 0 mt 1499451628 l 4352 t 0 d 0 ext )
2017/07/10 10:43:23 >>> 192.168.64.35:40334 Tstat tag 1 fid 0
2017/07/10 10:43:23 <<< 192.168.64.35:40334 Rstat tag 1 st ('test' 'george' '20' '' q (1af9adb 1e4819e0 'd') m d775 at 0 mt 1499451628 l 4352 t 0 d 0 ext )
2017/07/10 10:43:23 >>> 192.168.64.35:40334 Twstat tag 1 fid 0 st ('' '' '' '' q (ffffffffffffffff ffffffff 'daAltL') m d775 at 4294967295 mt 4294967295 l 18446744073709551615 t 65535 d 4294967295 ext )
2017/07/10 10:43:23 <<< 192.168.64.35:40334 Rwstat tag 1
SSH cmd err, output: <nil>:

[antonmarin]

check free RAM on VPS. Helped for me

[flowstate]

So this issue, and the two linked above are all open. Is there a workaround? I understand if it's a tough solve, but without a workaround, this completely breaks our ability to set up local test environments. Any help would be appreciated.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close