behind.md

Behind the Scenes

---

---

More boringly...

• node.js
• Amazon Web Services
• CloudFront / Load Balancers
• 3-5 EC2 instances
• EFS / S3 / DynamoDb

---

---

CE stats

---

CE stats

---

CE stats

• 1,000,000 compiles per week
• 1.5/sec average
• 4/sec peak
• 80,000 executions per week
• 3000 short URLs per week

---

Compiler stats

• 400+ compilers
• 250+ GB

Ada

Analysis

Assembly

C

C++

Clean

Cppx

CUDA

D

Fortran

Go

Haskell

ispc

LLVM IR

OCaml

Pascal

Rust

Swift

Zig

---

Maintaining

admin-node ~> ce --env prod instances list
Address          State      Type       ELB     Service  Version       
34.226.244.207   running    t3.medium  healthy running  3965 (master) 
3.91.14.221      running    c5.large   healthy running  3965 (master) 

admin-node ~> ce --env prod builds list
Live  Branch     Version    Size       Hash          
 -->  master     3965       58.2MiB    969925..8b69c5
      master     3979       58.2MiB    9410c2..fbc044
      policy-... 3983       58.2MiB    27eccb..62da61

admin-node ~> ce --env prod builds set_current 3979
admin-node ~> ce --env prod instances restart

---

Compilers

• Built with custom docker containers
• Daily process
• World's worst CI
• Takes 4+hr/day on fast machine (c5d.4xlarge)

---

Security

• Compilers
• User execution
• firejail
• nsjail

---

Monitoring

• papertrail - live event monitoring
• AWS Dashboard
• stathat - aggregate statistics
• Google Analytics
• sentry - javascript error aggregation
• StatusCake - uptime monitoring

---

When things go wrong

---

When things go wrong

• Bad config (August 2018)

• EFS transfer limit (May 2017)

  

• S3 outage (Feb 2017)

• EFS/NFS/bind mount/systemd catastrophe (September 2019)