support mTLS for authentication

[Mythra]

fixes #356

this adds in support for specifying a --tls_ca_file which when provided will validate client certificates against that particular authority.

I originally thought about splitting the --tls_ca_file option into two (e.g. having an: --mtls flag, that actually enforced the client certificates), but I figured if this is the only use case for specifying a --tls_ca_file (which it seems right now this is true for this project, though not in general) best to not add an extra option. If you'd like though I can always go back, and add it in.

This seems to pass all tests locally. I've also run just locally on my machine with self signed certs:

./bazel-remote --tls_ca_file ca.pem --tls_cert_file server.pem --tls_key_file server.key  --dir bazel-remote-tmp-cache/ --max_size 1

The Endpoint is properly authenticated:

# HTTP 1 gives a clearer error message
$ curl --httpv1.1 --cacert ca.pem -sSL -i https://localhost:8080/status
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0
$ curl --cacert ca.pem --cert basic-user.pem --key basic-user.key -sSL -i https://localhost:8080/status
HTTP/2 200 
content-type: application/json
content-length: 179
date: Tue, 29 Sep 2020 17:56:13 GMT

{
 "CurrSize": 377389094,
 "ReservedSize": 0,
 "MaxSize": 10737418240000,
 "NumFiles": 9296,
 "ServerTime": 1601402173,
 "GitCommit": "bfe94a99e2c765cffe6000a51443140f6534282d"
}

And with no ca certificate, but a cert/key it just accepts any user as it does now:

$ curl --cacert ca.pem -sSL -i https://localhost:8080/status
HTTP/2 200 content-type: application/json
content-length: 179
date: Tue, 29 Sep 2020 17:57:59 GMT

{
 "CurrSize": 377389094,
 "ReservedSize": 0,
 "MaxSize": 10737418240000,
 "NumFiles": 9296,
 "ServerTime": 1601402279,
 "GitCommit": "bfe94a99e2c765cffe6000a51443140f6534282d"
}

[mostynb]

Thanks for the PR- this looks good, with a few nitpicks.

Could you please also add some more documentation in the README.md file:

• An example in the config file section, after the other TLS options.
• Mention how to use this with bazel (which flags are required, etc).
• Regenerate the --help text.

[Mythra]

Thanks for the review! I'll take a bit later today to rework.

[Mythra]

Sorry that took a bit, was preoccupied. @mostynb I've opted for "mTLS (client certificate)" esque wording as I've actually never heard it called "client certificates" only "mTLS". I also believe that it can be helpful to understand that it's a mutual authentication, which I think makes the error message for specifying a TLS CA File but not cert/key more understandable. "if i'm just authenticating client certificates, why do I also need a key/cert?" is a question I imagine coming up.

Happy to change again, but let me know what you think, thanks!

[mostynb]

Looks good- thanks for contributing.

I also landed some related adjustments to README.md. It seems that mTLS/HTTPS doesn't work, so I skipped over that in the configuring bazel section of the docs for now.

[Mythra]

mTLS/HTTPS doesn't work, I'm confused what did you try? mTLS is https just with extra authentication.

[mostynb]

mTLS/HTTPS doesn't work, I'm confused what did you try? mTLS is https just with extra authentication.

In one terminal, I ran bazel-remote with --tls_ca_file, --tls_cert_file and --tls_key_file, so it's listening on port 8080 for https and 9092 for grpcs.

Then in another terminal, I tried building bazel remote using bazel 3.2.0, with the following flags:
--remote_cache=grpcs://localhost:9092, --tls_certificate, --tls_client_certificate and --tls_client_key set, which worked.

Then I changed the URL to https://localhost:8080, and tried again. On the bazel-remote side I get a bunch of http: TLS handshake error from 127.0.0.1:53642: EOF errors. On the bazel side I see a java stack trace:

WARNING: Reading from Remote Cache:
SSLEngine closed already
WARNING: Reading from Remote Cache:
ExtendedClosedChannelException
Internal error thrown during build. Printing stack trace: java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData{actionLookupKey=@zlib//:copy_public_headers BuildConfigurationValue.Key[7a889f35968a34de951cd99f682b84fbefe68e8879beff151d29b991f94fe405] false, actionIndex=0}' (requested by nodes 'ActionLookupData{actionLookupKey=@zlib//:zlib BuildConfigurationValue.Key[7a889f35968a34de951cd99f682b84fbefe68e8879beff151d29b991f94fe405] false, actionIndex=0}')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:513)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:398)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:677)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 more
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
	at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:674)
	at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:577)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1141)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1256)
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:212)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1330)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
	... 20 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
	at java.base/sun.security.validator.Validator.validate(Unknown Source)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
	at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:242)
	at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:670)
	... 30 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
	at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
	... 38 more

[Mythra]

Interesting! Based on my testing above with cURL it should just work, this honestly seems like a problem with bazel not using the client certificate for HTTPS. Will dig in and make any changes here if necessary though. Thanks!

[mostynb]

bazel seems to have tests for mTLS with both https and grpcs:
bazelbuild/bazel@0a75645