Docs for sccache, enable tests

.travis.yml

@@ -29,13 +29,16 @@ matrix:
     - rust: nightly
 
 before_script:
-  - if [[ "${TRAVIS_RUST_VERSION}" = "nightly" ]]; then export EXTRA_FEATURES=unstable; fi
-  - if [[ "${DIST_SCCACHE}" = "1" ]]; then export EXTRA_FEATURES="dist-client dist-server"; fi
+  - export EXTRA_FEATURES=
+  - if [[ "${TRAVIS_RUST_VERSION}" = "nightly" ]]; then export EXTRA_FEATURES="$EXTRA_FEATURES unstable"; fi
+  - if [[ "${DIST_SCCACHE}" = "1" ]]; then export EXTRA_FEATURES="$EXTRA_FEATURES dist-client dist-server"; fi
 
 script:
   - cargo build --verbose --features="all ${EXTRA_FEATURES}"
   - RUST_BACKTRACE=1 cargo test --all --verbose --no-default-features --features="${EXTRA_FEATURES}"
   - RUST_BACKTRACE=1 cargo test --all --verbose --features="all ${EXTRA_FEATURES}"
+  # Requires PR #321
+  #- if [[ "${DIST_SCCACHE}" = "1" ]]; then RUST_BACKTRACE=1 cargo test --all --verbose --features="all dist-tests ${EXTRA_FEATURES}" test_dist_ -- --test-threads 1; fi
 
 before_deploy:
   - name="sccache-$TRAVIS_TAG-$TARGET"

docs/Distributed.md

@@ -39,9 +39,13 @@ The HTTP implementation of sccache has the following API:
    - `POST alloc_job`
       - Called by a client to submit a compilation request.
       - Returns information on where the job is allocated it should run.
+   - `GET server_certificate`
+      - Called by a client to retrieve the (dynamically created) HTTPS
+        certificate for a server, for use in communication with that server.
+      - Returns a digest and PEM for the temporary server HTTPS certificate.
    - `POST heartbeat_server`
       - Called (repeatedly) by servers to register as available for jobs.
-   - `POST state`
+   - `POST job_state`
       - Called by servers to inform the scheduler of the state of the job.
    - `GET status`
       - Returns information about the scheduler.
@@ -55,16 +59,17 @@ The HTTP implementation of sccache has the following API:
       - Called by the client to run a job.
       - Returns the compilation stdout along with files created.
 
-There are two axes of security in this setup:
+There are three axes of security in this setup:
 
 1. Can the scheduler trust the servers?
 2. Is the client permitted to submit and run jobs?
+3. Can third parties see and/or modify traffic?
 
 ### Server Trust
 
 If a server is malicious, they can return malicious compilation output to a user.
-To protect against this, servers must be authenticated. You have three means for
-doing this, and the scheduler and all servers must use the same mechanism.
+To protect against this, servers must be authenticated to the scheduler. You have three
+means for doing this, and the scheduler and all servers must use the same mechanism.
 
 Once a server has registered itself using the selected authentication, the scheduler
 will trust the registered server address and use it for builds.
@@ -87,8 +92,8 @@ follows:
 server_auth = { type = "jwt_hs256", secret_key = "YOUR_KEY_HERE" }
 ```
 
-Now generate a token for the server, giving the IP and port the scheduler can
-connect to the server on (IP `192.168.1.10` and port `10501` here):
+Now generate a token for the server, giving the IP and port the scheduler and clients can
+connect to the server on (address `192.168.1.10:10501` here):
 
 ```
 sccache-dist auth generate-jwt-hs256-server-token \
@@ -155,10 +160,9 @@ Done!
 
 If a client is malicious, they can cause a DoS of distributed sccache servers or
 explore ways to escape the build sandbox. To protect against this, clients must
-be authenticated. You have two means for doing this, and the scheduler and all
-clients must use the same mechanism.
+be authenticated.
 
-Each client will use the authentication for the initial job allocation request
+Each client will use an authentication token for the initial job allocation request
 to the scheduler. A successful allocation will return a job token that is used
 to authorise requests to the appropriate server for that specific job.
 
@@ -168,6 +172,49 @@ the scheduler during registration. This means that the server can verify users
 without either a) adding client authentication to every server or b) needing
 secret transfer between scheduler and server on every job allocation.
 
+#### OAuth2
+
+This is a group of similar methods for achieving the same thing - the client
+retrieves a token from an OAuth2 service, and then submits it to the scheduler
+which has a few different options for performing validation on that token.
+
+*To use it*:
+
+Put one of the following settings in your scheduleer config file to determine how
+the scheduler will validate tokens from the client:
+
+```
+# Use the known settings for Mozilla OAuth2 token validation
+client_auth = { type = "mozilla" }
+
+# Will forward the valid JWT token onto another URL in the `Bearer` header, with a
+# success response indicating the token is valid. Optional `cache_secs` how long
+# to cache successful authentication for.
+client_auth = { type = "proxy_token", url = "...", cache_secs = 60 }
+```
+
+Additionally, each client should set up an OAuth2 configuration in the with one of
+the following settings (as appropriate for your OAuth service):
+
+```
+# Use the known settings for Mozilla OAuth2 authentication
+auth = { type = "mozilla" }
+
+# Use the Authorization Code with PKCE flow. This requires a client id,
+# an initial authorize URL (which may have parameters like 'audience' depending
+# on your service) and the URL for retrieving a token after the browser flow.
+auth = { type = "oauth2_code_grant_pkce", client_id = "...", auth_url = "...", token_url = "..." }
+
+# Use the Implicit flow (typically not recommended due to security issues). This requires
+# a client id and an authorize URL (which may have parameters like 'audience' depending
+# on your service).
+auth = { type = "oauth2_implicit", client_id = "...", auth_url = "..." }
+```
+
+The client should then run `sccache --dist-auth` and follow the instructions to retrieve
+a token. This will be automatically cached locally for the token expiry period (manual
+revalidatation will be necessary after expiry).
+
 #### Token
 
 This method simply shares a token between the scheduler and all clients. A token
@@ -207,6 +254,23 @@ client_auth = { type = "DANGEROUSLY_INSECURE" }
 ```
 
 Remove any `auth =` setting under the `[dist]` heading in your client config file
-(it will default to insecure).
+(it will default to this insecure mode).
 
 Done!
+
+### Eavesdropping and Tampering Protection
+
+If third parties can see traffic to the servers, source code can be leaked. If third
+parties can modify traffic to and from the servers or the scheduler, they can cause
+the client to receive malicious compiled objects.
+
+Securing communication with the scheduler is the responsibility of the sccache cluster
+administrator - it is recommended to put a webserver with a HTTPS certificate in front
+of the scheduler and instruct clients to configure their `scheduler_url` with the
+appropriate `https://` address.
+
+Securing communication with the server is performed automatically - HTTPS certificates
+are generated dynamically on server startup and communicated to the scheduler during
+the heartbeat. If a client does not have the appropriate certificate for communicating
+securely with a server (after receiving a job allocation from the scheduler), the
+certificate will be requested from the scheduler.

docs/DistributedQuickstart.md

@@ -4,7 +4,7 @@ sccache distributed compilation quickstart
 This is a quick start guide to getting distributed compilation working with sccache. This guide currently only covers Linux clients, although macOS and Windows clients are supported.
 
 Get sccache binaries
-====================
+--------------------
 
 Either download pre-built sccache binaries (not currently available), or build sccache locally with the `dist-client` and `dist-server` features enabled:
 ```
@@ -14,12 +14,16 @@ cargo build --release --features="dist-client dist-server"
 The `target/release/sccache` binary will be used on the client, and the `target/release/sccache-dist` binary will be used on the scheduler and build server.
 
 Configure a scheduler
-=====================
+---------------------
 
 The scheduler is a daemon that manages compile request from clients and parcels them out to build servers. You only need one of these per sccache setup. Currently only Linux is supported for running the scheduler.
 
 Create a scheduler.conf file to configure client/server authentication. A minimal example looks like:
 ```toml
+# The socket address the scheduler will listen on. It's strongly recommended to listen
+# on localhost and put a HTTPS server in front of it.
+public_addr = "127.0.0.1:10600"
+
 [client_auth]
 type = "token"
 token = "my client token"
@@ -37,7 +41,7 @@ sccache-dist scheduler --config scheduler.conf
 If the scheduler fails to start you may need to set `RUST_LOG=trace` when starting it to get useful diagnostics.
 
 Configure a build server
-========================
+------------------------
 
 A build server communicates with the scheduler and executes compiles requested by clients. Only Linux is supported for running a build server, but executing cross-compile requests from macOS/Windows clients is supported.
 
@@ -50,11 +54,11 @@ cache_dir = "/tmp/toolchains"
 # The maximum size of the toolchain cache, in bytes.
 # If unspecified the default is 10GB.
 # toolchain_cache_size = 10737418240
-# An IP address and port on which clients can connect to this builder.
-# NOTE: you must use port 10501 here!
+# A public IP address and port that clients will use to connect to this builder.
 public_addr = "192.168.1.1:10501"
-# The IP address of the scheduler.
-scheduler_addr = "192.168.1.1"
+# The URL used to connect to the scheduler (should use https, given an ideal
+# setup of a HTTPS server in front of the scheduler)
+scheduler_url = "https://192.168.1.1"
 
 [builder]
 type = "overlay"
@@ -77,7 +81,7 @@ sudo sccache-dist server --config server.conf
 As with the scheduler, if the build server fails to start you may need to set `RUST_LOG=trace` to get useful diagnostics.
 
 Configure a client
-==================
+------------------
 
 A client uses `sccache` to wrap compile commands, communicates with the scheduler to find available build servers, and communicates with build servers to execute the compiles and receive the results.
 
@@ -86,10 +90,9 @@ Clients require the `icecc-create-env` script, which is part of `icecream` for p
 Create a client config file in `~/.config/sccache/config`. A minimal example looks like:
 ```toml
 [dist]
-# The IP address of the scheduler.
-scheduler_addr = "192.168.1.1"
-# A directory in which toolchain information will be cached.
-cache_dir = "/tmp/toolchains"
+# The URL used to connect to the scheduler (should use https, given an ideal
+# setup of a HTTPS server in front of the scheduler)
+scheduler_url = "https://192.168.1.1"
 # Used for mapping local toolchains to remote cross-compile toolchains. Empty in
 # this example where the client and build server are both Linux.
 toolchains = []
@@ -101,3 +104,77 @@ type = "token"
 # This should match the `client_auth` section of the scheduler config.
 token = "my client token"
 ```
+
+Configuring for a Mozilla build servers
+---------------------------------------
+
+Mozilla build servers will typically require clients to be authenticated with the
+[Mozilla identity system](https://github.com/mozilla-iam/mozilla-iam).
+
+To configure for scheduler for this, the `client_auth` section should be as follows
+so any client tokens are validated with the Mozilla service:
+
+```
+[client_auth]
+type = "mozilla"
+```
+
+Clients should configure their `dist.auth` section as follows:
+
+```
+[dist.auth]
+type = "mozilla"
+```
+
+And retrieve a token from the Mozilla identity service by running `sccache --dist-auth`
+and following the instructions. Completing this process will retrieve and cache a token
+valid for 7 days.
+
+Using custom toolchains
+-----------------------
+
+Since Windows and OSX cannot automatically package toolchains, it is important to be
+able to manually specify toolchains for distribution. This functionality is also available
+on Linux.
+
+Using custom toolchains involves adding a `dist.toolchains` section to your client config
+file (you can add it multiple times to specify multiple toolchains).
+
+On Linux and OSX:
+
+```
+[[dist.toolchains]]
+type = "path_override"
+compiler_executable = "/usr/bin/gcc-5"
+archive = "/home/me/toolchains/gcc-5-38505675dd9514438ed26497fceb0fe0.tar.gz"
+archive_compiler_executable = "/usr/bin/gcc"
+```
+
+On Windows:
+
+```
+[[dist.toolchains]]
+type = "path_override"
+compiler_executable = "C:/clang/bin\\clang-cl.exe"
+archive = "C:/toolchains/gcc-5-38505675dd9514438ed26497fceb0fe0.tar.gz"
+archive_compiler_executable = "/usr/bin/gcc"
+```
+
+Where:
+ - `compiler_executable` identifies the path that sccache will match against to activate
+   this configuration (you need to be careful on Windows - paths can have slashes in both
+   directions, and you may need to escape backslashes, as in the example)
+ - `archive` is the compressed tar archive containing the compiler toolchain to distribute
+   when `compiler_executable` is matched
+ - `archive_compiler_executable` is the path within the archive the distributed
+   compilation should invoke
+
+A toolchain archive should be a Gzip compressed TAR archive, containing a filesystem
+sufficient to run the compiler without relying on any external files. If you have archives
+compatible with icecream (created with `icecc-create-env`, like
+[these ones](https://github.com/jyavenard/mozilla-icecream) for OSX), they should also work
+with sccache. To create a Windows toolchain, it is recommended that you download the [Clang
+binaries for Ubuntu 16.04](http://releases.llvm.org/download.html) and extract them,
+package up the toolchain using the extracted `bin/clang` file (requires
+[PR #321](https://github.com/mozilla/sccache/pull/321)) and then insert `bin/clang-cl` at
+the appropriate path as a symlink to the `bin/clang` binary.

src/bin/sccache-dist/token_check.rs

@@ -247,15 +247,7 @@ impl ProxyTokenCheck {
     }
 
     fn check_token_with_forwarding(&self, token: &str) -> Result<()> {
-        #[derive(Deserialize)]
-        struct Token {
-            exp: u64,
-        }
-        let unsafe_token = jwt::dangerous_unsafe_decode::<Token>(token).chain_err(|| "Unable to decode jwt")?;
         trace!("Validating token by forwarding to {}", self.url);
-        if UNIX_EPOCH + Duration::from_secs(unsafe_token.claims.exp) < SystemTime::now() {
-            bail!("JWT expired")
-        }
         // If the token is cached and not cache has not expired, return it
         if let Some(ref auth_cache) = self.maybe_auth_cache {
             let mut auth_cache = auth_cache.lock().unwrap();
@@ -272,7 +264,7 @@ impl ProxyTokenCheck {
         let res = self.client.get(&self.url).set_header(header).send()
             .chain_err(|| "Failed to make request to proxying url")?;
         if !res.status().is_success() {
-            bail!("JWT forwarded to {} returned {}", self.url, res.status());
+            bail!("Token forwarded to {} returned {}", self.url, res.status());
         }
         // Cache the token
         if let Some(ref auth_cache) = self.maybe_auth_cache {