Build out Pipeline for Whistium Builds

[rpadaki]

Copied from Slack

Curious people's thoughts on a development setup for Whistium (our server Chromium-based fork). Right now it's going to be as simple as:

1. Log on to roshan-builds-chromium EC2 instance
2. Make your changes
3. Build
4. Upload to S3
5. Rebuild your browsers/whistium mandelbox to pull the new version
   Meanwhile, deployment instances will also just pull from the S3 bucket. This is fine for now because we don't need to make any more changes for the time being.

Eventually, though, we need a more robust setup if we're going to make significant changes. This will consist of two components:
CI for dev, staging, and prod, which applies our Whistium patches to Chromium or Brave, builds it, and uploads the .deb output to S3.
A development build setup
The CI is self-explanatory. The development build setup needs to enable us to modify and test Chromium during development, or to build mandelboxes in general even if not modifying Whistium. I see a few options:

1. Require all developer EC2 instances to be capable of building Chromium. This will either be expensive or lead to extremely time-consuming builds. Probably both.
2. Pull Whistium from the dev CI S3 bucket. If a developer needs to modify Whistium, they should have a separate beefy instance capable of building it, and then they can manually load it into their mandelbox. But pulling from dev as a default can make everyone else's lives easier.
3. Cache on a hash of the patch (a mouthful, I know) in S3. That is, have an S3 bucket where we keep Whistium builds indexed by a hash of the diff.
   When we ./build.sh browsers/whistium, check if hash_contents("whistium.patch") is a folder in the S3 bucket.
   • If it is, pull that whistium.deb and use it for the mandelbox.
   • If isn't, use the gh CLI to trigger a manual workflow run on a beefy EC2 runner and wait for it to finish. This workflow takes the submitted whistium.patch, applies it to Chromium, and uploads the result to the hash_contents("whistium.patch") folder in S3.
   • The great thing about this approach is that we don't need a separate pipeline for CI versus development. Dev/staging/prod CI can use the exact same logic above, allowing them to skip unnecessarily building Chromium if a developer already triggered a build when they merged the most recent whistium.patch PR. Also no need to rebuild after promotions.
   • To avoid the S3 bucket becoming massive, we could add very simple logic to evict any builds that are not dev/staging/prod and are over a week old.
4. Don't cache on a hash of the patch. Similar to Catch staging up with master #3, but instead of this shared S3 bucket, have the workflow directly send the Chromium build back to the developer's EC2 instance. While this may be conceptually simpler, it will lead to extraneous work.
5. Consider Catch staging up with master #3 and Set up monorepo (into master) #4 using GitHub Actions artifacts instead of an S3 bucket. Then we can take advantage of GitHub's default retention policy as well.