Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Rust to build process #420

Merged
merged 5 commits into from
Dec 1, 2018
+831 −2
Merged

Add Rust to build process #420

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8377f7d
Add 'stroller' dir containing skeletal Rust web server
samstokes Nov 27, 2018
16f3f6a
Add Rust stable to Docker image and build stroller
samstokes Nov 27, 2018
5ff06e7
Rebuild stroller when source files change
samstokes Nov 30, 2018
33314b8
Add Rust test run to build script
samstokes Nov 30, 2018
696f994
fix perms after rebase
samstokes Nov 30, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,36 @@ ENV TERM=xterm-256color
RUN opam install -y \
ocamlformat

########################
# Install Rust toolchain
########################

USER root

ENV RUSTUP_HOME=/usr/local/rustup \
CARGO_HOME=/usr/local/cargo \
PATH=/usr/local/cargo/bin:$PATH \
RUST_VERSION=1.30.1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you given any thought to using async/await? No need to make a decision now -- but if you are I'm totally okay with pinning a nightly that we have to bump every couple of weeks to make that happen.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've not looked into Rust's async/await much (although I'm familiar with similar features in other languages e.g. JS), so I don't have a strong opinion. Do you know what's the state of library support for async/await at the moment? It looks like Tokio's support is "experimental" and Hyper's is a WIP, but I'm not clear how much that's about improving the ergonomics vs just having it work at all.

That said, I'm not imagining that the initial scope of stroller will require much interesting control flow or concurrency handling - for now it's pretty much going to be "on an HTTP POST from the backend, send an event to Pusher, then log the result somewhere" - so we might not gain much from async/await at this stage.

Once we add direct websocket connections into the mix, that'll replace the middle step with "look up the right websocket connection and send an event to it", which isn't much more complex.

That said, both approaches have questions like "what happens if the client hasn't opened a Pusher channel / websocket connection yet", "what happens if we fail to push the event". We could have stroller implement store-and-forward in those cases, which would complicate the concurrency model. However, I suspect we won't want to do that, because there's one stroller per backend instance and we could get into a mess with duplicate or dropped pushes. Instead we may want to have some sort of less-frequent "catchup" poll for lost events, or always send the most recent events on websocket connection establishment, or similar. (Or maybe we don't care enough about occasional lost pushes to bother with any of that.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had hoped tokio+hyper would be further along than when I last looked -- having given it a quick glance over I think swerving it for now is probably best.

Agreed with your thoughts re: initial scope.

For managing connections/store+forward/pusher replacement in the future, I'm imagining another service that stroller pushes to. Out of scope for now obvs, and I agree with punting on lost pushes/store+forward til then.


RUN dpkgArch="$(dpkg --print-architecture)"; \
case "${dpkgArch##*-}" in \
amd64) rustArch='x86_64-unknown-linux-gnu'; rustupSha256='0077ff9c19f722e2be202698c037413099e1188c0c233c12a2297bf18e9ff6e7' ;; \
armhf) rustArch='armv7-unknown-linux-gnueabihf'; rustupSha256='f139e5be4ea2db7ff151c122f5d24af3c587c4fc74a7414e262cb34403278ad3' ;; \
arm64) rustArch='aarch64-unknown-linux-gnu'; rustupSha256='c7d5471e71a315134e7499af75eb177d1f574858f1c6b8e61b436702d671a4e2' ;; \
i386) rustArch='i686-unknown-linux-gnu'; rustupSha256='909ce4e2d0c9bf60ba5a85426c38cceb5ae77979ab2b1e354e76b9851b5ec5ed' ;; \
*) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \
esac; \
url="https://static.rust-lang.org/rustup/archive/1.14.0/${rustArch}/rustup-init"; \
wget "$url"; \
echo "${rustupSha256} *rustup-init" | sha256sum -c -; \
chmod +x rustup-init; \
./rustup-init -y --no-modify-path --default-toolchain $RUST_VERSION; \
rm rustup-init; \
chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
rustup --version; \
cargo --version; \
rustc --version;

############################
# Finish
############################
Expand Down
2 changes: 2 additions & 0 deletions scripts/builder
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ else
MOUNTS+=" --mount type=volume,src=dark_server_build,dst=/home/dark/app/server/_build"
MOUNTS+=" --mount type=volume,src=dark_client2_node_modules,dst=/home/dark/app/client2/node_modules"
MOUNTS+=" --mount type=volume,src=dark_client2_lib,dst=/home/dark/app/client2/lib"
MOUNTS+=" --mount type=volume,src=dark_stroller_target,dst=/home/dark/app/stroller/target"
MOUNTS+=" --mount type=volume,src=dark_rust_cargo,dst=/usr/local/cargo"

if [[ -e "../conduit-frontend" ]]; then
MOUNTS="$MOUNTS --mount type=bind,src=$PWD/../conduit-frontend,dst=/home/dark/conduit-frontend"
Expand Down
4 changes: 3 additions & 1 deletion scripts/clear-builder-volumes
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@ set -euo pipefail
docker volume rm \
dark_server_build \
dark_client2_node_modules \
dark_client2_lib
dark_client2_lib \
dark_stroller_target \
dark_rust_cargo
2 changes: 2 additions & 0 deletions scripts/support/build-server
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ def initial_compile():
, "client2/package.json"
, "client2/src/main.ml"
, "client2/__tests__/ast_test.ml"
, "stroller/Cargo.toml"
]
exit = compile(files)

Expand Down Expand Up @@ -157,6 +158,7 @@ def main():
run_or_fail("sudo chown dark:dark server/_build || true")
run_or_fail("sudo chown dark:dark client2/node_modules || true")
run_or_fail("sudo chown dark:dark client2/lib || true")
run_or_fail("sudo chown dark:dark stroller/target || true")
run_or_fail("scripts/support/write-config-file")
run_or_fail("scripts/support/allow-docker-access")
create_app_dirs()
Expand Down
37 changes: 36 additions & 1 deletion scripts/support/compile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,28 @@ def server_test():
+ ci
+ " 2>&1")

def stroller_build():
start = time.time()
global profile

if os.environ.get("CI"):
build_flags = " --release"
pass
else:
build_flags = ""
pass

build = "cd stroller && unbuffer cargo build" + build_flags
if profile:
return run_backend(start, landmarks + build)
else:
return run_backend(start, build)

def stroller_test():
start = time.time()
return run_test(start,
"cd stroller && unbuffer cargo test")

def reload_server():
start = time.time()
global profile
Expand All @@ -160,6 +182,8 @@ class Should:
def __init__(self):
self.server_build = False
self.server_test = False
self.stroller_build = False
self.stroller_test = False
self.npm_install = False
self.client2_build = False
self.bundlejs_build = False
Expand All @@ -174,6 +198,11 @@ def execute(should):
sb = should.server_build and server_build()
should.reload_server |= sb

stb = should.stroller_build and stroller_build()

should.stroller_test |= stb
stt = should.stroller_test and stroller_test()

ni = should.npm_install and npm_install ()
should.client2_build |= ni
should.darkex_build |= ni
Expand All @@ -199,7 +228,7 @@ def execute(should):
should.server_test |= sb
ot = should.server_test and server_test()

return sb and ni and c2b and bjsb and deb and c2t and ge and rs and rb and ot
return sb and stb and stt and ni and c2b and bjsb and deb and c2t and ge and rs and rb and ot



Expand All @@ -221,6 +250,12 @@ def mark(should, f):
elif "scripts/support/runserver" in f:
should.reload_server = True

# Rust
elif ("stroller/" in f) \
and (("Cargo.toml" in f) \
or (".rs" in f)):
should.stroller_build = True

# JS
elif "client2/package.json" in f:
should.npm_install = True
Expand Down
Loading