Add Rust to build process

[samstokes]

As a first step toward the Rust sidecar (codename: stroller) for pushing traces to the editor, this adds a "hello world" Rust web server and extends the Docker-based build process to compile it.

Build process

On a standard scripts/builder invocation, this does:

• add a Rust toolchain (1.30.1 is the latest stable release atm) to the Docker image
• download and compile stroller's dependencies specified via Cargo.toml on first build, and store them in a Docker volume, so that subsequent builds don't have to redo it
• compile stroller

This doesn't yet:

• install the stroller binary to any particular location inside the container, or attempt to run it. It should probably do the latter - I could use a pointer to the best place to fork the process, I'm guessing one of the Python scripts?
• cache the Rust dependencies across CI builds (I think it will cache the Rust toolchain though, by virtue of that being part of the base Docker image?)
• merge cleanly with master :( I'll bring it up to date but don't anticipate too many changes.
• pass CI (possibly because it doesn't merge?) - I don't have access to the Circle account yet so I can't see why it failed, but will fix before merging.

I haven't been able to test if this plays nicely with the --watch mode of scripts/builder since I've had trouble getting that to work correctly on my machine, although in principle I'd expect it not to conflict.

stroller

I had to pick a name for the sidecar project directory (and binary), so I called it stroller, since it's there for the main server process to push. That's probably too clever :) Happy to rename to "pusher-proxy" or something similar.

The Rust code isn't particularly interesting yet - this PR is just to get the build process working. It listens on port 3000 and sends hello-world HTTP responses.

Rust version

Following

Pin versions as closely as possible. You should get exactly the version you expect.

from the Engineering Guide, this pins to a specific stable release of Rust (1.30.1), and uses SHAs to verify integrity. (This is lifted directly from the official docker-rust Docker image.)

[IanConnolly]

Couple of small changes request, but this looks good!

Some future things that I'd like to see added when you start in earnest:

• failing CI if rustfmt produces any diff (ie. force rustfmt formatted code)
• integrating clippy into test suite + CI

[IanConnolly]

Have you given any thought to using async/await? No need to make a decision now -- but if you are I'm totally okay with pinning a nightly that we have to bump every couple of weeks to make that happen.

[samstokes]

I've not looked into Rust's async/await much (although I'm familiar with similar features in other languages e.g. JS), so I don't have a strong opinion. Do you know what's the state of library support for async/await at the moment? It looks like Tokio's support is "experimental" and Hyper's is a WIP, but I'm not clear how much that's about improving the ergonomics vs just having it work at all.

That said, I'm not imagining that the initial scope of stroller will require much interesting control flow or concurrency handling - for now it's pretty much going to be "on an HTTP POST from the backend, send an event to Pusher, then log the result somewhere" - so we might not gain much from async/await at this stage.

Once we add direct websocket connections into the mix, that'll replace the middle step with "look up the right websocket connection and send an event to it", which isn't much more complex.

That said, both approaches have questions like "what happens if the client hasn't opened a Pusher channel / websocket connection yet", "what happens if we fail to push the event". We could have stroller implement store-and-forward in those cases, which would complicate the concurrency model. However, I suspect we won't want to do that, because there's one stroller per backend instance and we could get into a mess with duplicate or dropped pushes. Instead we may want to have some sort of less-frequent "catchup" poll for lost events, or always send the most recent events on websocket connection establishment, or similar. (Or maybe we don't care enough about occasional lost pushes to bother with any of that.)

[IanConnolly]

I had hoped tokio+hyper would be further along than when I last looked -- having given it a quick glance over I think swerving it for now is probably best.

Agreed with your thoughts re: initial scope.

For managing connections/store+forward/pusher replacement in the future, I'm imagining another service that stroller pushes to. Out of scope for now obvs, and I agree with punting on lost pushes/store+forward til then.

[IanConnolly]

Also can you add a test? Just to get the harness up and running.

[samstokes]

Also can you add a test? Just to get the harness up and running.

Good call. d7ff3db adds a test for the placeholder code and extends the build script to run cargo test. (I think that's sufficient to have the CI builds also run the tests, right?)

[IanConnolly]

Yep, CI should run the files because the relevant files are in the 'initial set'. This lgtm now and can ship whenever you rebase + deal with the conflicts. Feel free to ping me if rebased changes have any tricky/potentially controversial resolutions you want me to take a look at