spk dev fails with Input/output error on Ubuntu 16.04

[mokomull]

When running spk dev on Ubuntu 16.04, it fails with:

$ spk dev
App is now available from Sandstorm server. Ctrl+C to disconnect.
*** Uncaught exception ***
sandstorm/fuse.c++:369: failed: read(/dev/fuse): Input/output error
stack: 0x56ab9b 0x56865c 0x5e240b 0x5e0fdd 0x48b3a0 0x48aeb8

This occurs with the official Ubuntu kernel packages (tested with 4.4.0-36-generic), but does not occur with the package of 4.4.14-040414-generic from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.14-xenial/. This also does not appear to affect any previous versions of Ubuntu.

I tracked this down to an Ubuntu-specific patch. This patch adds a check at time-of-fd-use that the file descriptor for /dev/fuse has not crossed a pid namespace boundary; this interferes with the way Sandstorm mounts the root filesystem and sends the file descriptor via a SCM_RIGHTS message to the spk dev running as the developer. I have confirmed that spk dev functions correctly after reverting that individual commit (and fixing some conflicts).

[mokomull]

There was some discussion on IRC about this around 23rd and 24th of July, but I don't think a conclusion was reached as to the path forward.

[dckc]

Is there any workaround for this?

[mokomull]

The only workaround I know of is to use a mainline kernel built by the Canonical kernel team, as I don't see a way around this codepath in the generic kernel.

I'm actually attending Linux Plumbers Conference this week — if I run into someone who knows more about it than I do, I'll update this issue.

[mrdomino]

This is also happening in 16.10.

[mnutt]

The original issue I had on 4.4.0 also occurs on 4.8.0-41, unfortunately.

[urakagi]

This issue still seems to exist. Is downgrading to 15.10 or even 14.04 LTS a better idea?

[ocdtrekkie]

I've got someone who appears to be getting this error on Ubuntu 14.04.1 with kernel 4.4.0-87-generic.

[zenhack]

Has anyone tried reporting a bug to Ubuntu? From the commit message:

Since no use case currently exists for changing namespaces all translations are done relative to the pid namespace in use when /dev/fuse is opened. Mounting or /dev/fuse IO from another namespace will return errors.

Perhaps worth bringing our use case to their attention.

[ocdtrekkie]

I lack the Linux fu to make a competent report to them. The person I am working with successfully switched to a mainline kernel though and is able to proceed.

[xet7]

Yes, I did have to install that kernel to build Wekan spk on Ubuntu 14.04.5

[xet7]

@kubiko

Do you know is there already bug report about this in Ubuntu Launchpad ?

[kubiko]

@xet7 sorry but not really my domain, if this is something kernel has fixed best to search kernel bugs in launchpad

[zenhack]

Here's the launchpad bug that patch was intended to fix:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1605344

[xet7]

When running kernel 4.4.14 on Ubuntu 14.04 on VirtualBox, VirtualBox extensions do not work, screen stays in low resolution.

This can be fixed by:

1. Installing newer gcc version 4.9

sudo add-apt-repository ppa:ubuntu-toolchain-r/test

sudo apt-get update

sudo apt-get install gcc-4.9

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 20

2. Reinstalling VirtualBox extensions from VirtualBox Device menu.

Info source at VirtualBox forums

Info source at AskUbuntu

I also added info about this to Wekan Sandstorm spk page.

[zenhack]

It looks like that patch has been pushed into the upstream kernel:

torvalds/linux@0b6e9ea041e6c

I'd planned on trying to report the bug to ubuntu, but couldn't sort out the logic with what sandstorm is doing here well enough to feel like I could explain it coherently in a bug report. This being upstreamed makes it a bit more important though.

//cc @kentonv

[mokomull]

It's been quite a while since I've looked at what spk dev does behind-the-scenes, and I completely missed this patch reaching mainline — looks like it was merged into the v4.12 release.

If no one has better connections in the kernel community, I can try to get something written-up for LKML, but the last discussion I see about the relevant patch is from more than a year ago. We don't seem to be the only ones that saw a regression, but it doesn't look like anyone has published a patch for the agreed-upon workaround.

[zenhack]

@kentonv emailed the folks in the signed-off-by fields on that commit. They've acknowledged the regression, and the fuse maintainer sent us an (untested) patch from one of the developers. Hopefully sometime this week I'll have time to test it myself, but I'm going to attach it to this post in case someone else wants to take a whack at it. (Note that I've added a .txt extension, because github has a whitelist of extensions for some reason).

fuse-allow-server-to-run-in-different-pid-ns.patch.txt

[zenhack]

The relevant bits of conversation on IRC last night: https://botbot.me/freenode/sandstorm/2017-09-05/?msg=90688526&page=1

[mokomull]

I got my kernel test machine back up and running, and verified that spk dev fails with EIO on stock 4.12, and that this patch applies cleanly and successfully runs spk dev.

Tested-by: Matt Mullins <mmullins@mmlx.us>

[zenhack]

Quoting Matt Mullins (2017-09-06 23:44:40)

I got my kernel test machine back up and running, and verified that spk dev fails with EIO on stock 4.12, and that this patch applies cleanly and successfully runs spk dev.

Yep, I've just tested it myself, and can also confirm that it fixes the problem.

[zenhack]

This has now been merged into Linus's tree:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5d6d3a301c4e749e04be6fcdcf4cb1ffa8bae524

[kentonv]

Guess this is fixed, then!

[ocdtrekkie]

Any idea if/when any given Ubuntu version will have this fix?