Bazel sandboxing not supported WSL

[neilbalch]

Description of the problem / feature request:

I would like to build a codebase that uses sandboxing on the Windows Sybsystem for Linux.

Feature requests: what underlying problem are you trying to solve with this feature?

Whenever I try to build something, bazel throws the error: java.lang.IllegalStateException: com.google.devtools.build.lib.actions.UserExecException: Sandbox is not supported

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

1. Install the Ubuntu WSL on Windows 10
2. Download and install bazel
3. Try to build in a project that uses the sandboxing feature

What operating system are you running Bazel on?

Windows 10, 1709

What's the output of bazel info release?

201711232100+e0fe5d9

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://nbalch@.../gerrit/a/...
ecb118d249b64b785a04d44225a573207de6a30a
ecb118d249b64b785a04d44225a573207de6a30a

The two elipsis above are there to hide the server address. The proper remote shows up in reality.

Have you found anything relevant by searching the web?

I have tried to search other places on the web, but I haven't found anything like this yet.

[philwo]

I'm not surprised that the linux-sandbox strategy doesn't work on WSL, but Bazel should automatically fallback to a variant that works.

I can't see where this exception could come from in our code and can't find the string "Sandbox is not supported" either :( We'll have to try to reproduce it.

@neilbalch Could you try your build with bazel build --spawn_strategy=standalone --genrule_strategy=standalone //my:target and see if that works?

[neilbalch]

How interesting. When I get home today, I will try that command.

[neilbalch]

oops, didn't realize what the "Close and Comment" button did.

[neilbalch]

Nope, @philwo same error:

Unhandled exception thrown during build; message: com.google.devtools.build.lib.actions.UserExecException: Sandbox is not supported
INFO: Elapsed time: 4.813s
FAILED: Build did NOT complete successfully (1 packages loaded)
java.lang.IllegalStateException: com.google.devtools.build.lib.actions.UserExecException: Sandbox is not supported
at com.google.devtools.build.lib.sandbox.SandboxModule.executorInit(SandboxModule.java:70)
at com.google.devtools.build.lib.buildtool.ExecutionTool.(ExecutionTool.java:200)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:170)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:347)
at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:67)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:487)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:223)
at com.google.devtools.build.lib.runtime.CommandExecutor.exec(CommandExecutor.java:58)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:850)
at com.google.devtools.build.lib.server.GrpcServerImpl.access$2100(GrpcServerImpl.java:109)
at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:915)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.devtools.build.lib.actions.UserExecException: Sandbox is not supported
at com.google.devtools.build.lib.sandbox.SandboxActionContextProvider.create(SandboxActionContextProvider.java:69)
at com.google.devtools.build.lib.sandbox.SandboxModule.executorInit(SandboxModule.java:68)
... 13 more
java.lang.IllegalStateException: com.google.devtools.build.lib.actions.UserExecException: Sandbox is not supported
at com.google.devtools.build.lib.sandbox.SandboxModule.executorInit(SandboxModule.java:70)
at com.google.devtools.build.lib.buildtool.ExecutionTool.(ExecutionTool.java:200)
at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:170)
at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:347)
at com.google.devtools.build.lib.runtime.commands.BuildCommand.exec(BuildCommand.java:67)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:487)
at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:223)
at com.google.devtools.build.lib.runtime.CommandExecutor.exec(CommandExecutor.java:58)
at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:850)
at com.google.devtools.build.lib.server.GrpcServerImpl.access$2100(GrpcServerImpl.java:109)
at com.google.devtools.build.lib.server.GrpcServerImpl$2.lambda$run$0(GrpcServerImpl.java:915)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.devtools.build.lib.actions.UserExecException: Sandbox is not supported
at com.google.devtools.build.lib.sandbox.SandboxActionContextProvider.create(SandboxActionContextProvider.java:69)
at com.google.devtools.build.lib.sandbox.SandboxModule.executorInit(SandboxModule.java:68)
... 13 more

[neilbalch]

@philwo And you said you can't reproduce this error on another WSL installation?

[AustinSchuh]

@neilbalch This is a custom patch we have at robotics. If you pull the latest version, we have upgraded bazel and switched to using a hard-coded spawn strategy per @philwo 's advice in another ticket. The code isn't really set up to build outside the sandbox.

[AustinSchuh]

@neilbalch, How are you grabbing the latest version of Bazel and trying that? We've got a hermetic Bazel set up so it is versioned with the repo, not with your host.

[neilbalch]

@AustinSchuh I was trying both git pull origin master and git fetch origin master && git rebase -i origin/master. Both yielded the same result.

[AustinSchuh]

@neilbalch Looks like we haven't dropped that patch yet, but did also add Philipp's suggested workaround (--spawn_strategy=linux-sandbox). I'll ditch it next time we upgrade.

Please remove tools/bazel and try re-building with a more recent version of Bazel installed on your host. You'll likely need to fix some of the rules to fix any breakage. That should give Philipp a modern error.

[neilbalch]

@AustinSchuh Other than errors with the aos/downloader/downloader.bzl file that I am having a hard time resolving, bazel suggests I use the --ignore_unsupported_sandboxing flag that does dismiss the message about my platform not supporting sandboxing.

INFO: Reading 'startup' options from /home/neil/971-Robot-Code/tools/bazel.rc: --host_jvm_args=-Dbazel.DigestFunction=SHA1
WARNING: Sandboxed execution is not supported on your system and thus hermeticity of actions cannot be guaranteed. See http://bazel.io/docs/bazel-user-manual.html#sandboxing for more information. You can turn off this warning via --ignore_unsupported_sandboxing.
ERROR: /home/neil/971-Robot-Code/aos/downloader/downloader.bzl:37:13: function 'depset' does not exist.
ERROR: error loading package '@//y2017_bot3': Extension 'aos/downloader/downloader.bzl' has errors.
INFO: Elapsed time: 0.485s

The errors in downloader.bzl end compilation. I am not sure if there would be another error to encounter after the aforementioned, but it seems as if a non-modded bazel installation treats the lack of sandboxing support well.

[neilbalch]

After talking with @brian-peloton, it seems likely that the issues with WSL and sandboxing support stem from an improper implementation of unshare. Is this mendable by us or is this a limitation that must be resolved by the WSL team?

[philwo]

@philwo And you said you can't reproduce this error on another WSL installation?

No, I haven't tried yet to reproduce it myself. I just couldn't see where that error might come from, from looking through the sandboxing code. But @AustinSchuh said that you were running a custom version of Bazel, so that explains it :)

ERROR: /home/neil/971-Robot-Code/aos/downloader/downloader.bzl:37:13: function 'depset' does not exist.

depset was introduced in Bazel 0.7.0 (I think, at least its release notes mention it), so it looks like your Bazel version is too old. Can you try upgrading Bazel in your WSL?

After talking with @brian-peloton, it seems likely that the issues with WSL and sandboxing support stem from an improper implementation of unshare. Is this mendable by us or is this a limitation that must be resolved by the WSL team?

I was about to write a long reply regarding why this could never work in WSL and that unshare with all these namespace is such a complex feature that they probably would never get it to work, and then I found microsoft/WSL#2242 (comment). Wow. It looks like this might actually work (but require some development and testing effort on the linux-sandbox code to ensure we're only using namespace features that work).

I won't be able to work on this in the short-term, but that's pretty exciting! If someone wants to give it a try, I'm happy to review pull requests for this.

For now, I would recommend to try building without sandboxing (bazel build --spawn_strategy=standalone --genrule_strategy=standalone //my:target) or the lightweight sandbox variant which doesn't rely on any special kernel features: bazel build --spawn_strategy=processwrapper-sandbox --genrule_strategy=processwrapper-sandbox //my:target. Our official Bazel releases should automatically pick the processwrapper-sandbox strategy when linux-sandbox isn't available, so you can try running with no flags after upgrading your Bazel and it should just work.

[AustinSchuh]

@philwo We are passing in --spawn_strategy=linux-sandbox to explicitly disable that feature :P I don't really want to deal with code ending up on the robot built without sandboxing and any bugs that could come from that. @neilbalch, you can make a local modification to build, but please don't deploy to real hardware with that flag.

[bsilver8192]

That's interesting. My past experience with WSL was trying unshare on the command line and seeing lots of failures, but it sounds like they're working on that. Might be worth looking into in that case.

[philwo]

I'll close this, as I won't have any time to look into it in the next months. If someone wants to contribute patches that improve support for WSL, I'd be happy to review them, though.