[RFC] Support for horizontal scaling Sourcebot

[brendan-kellam]

This discussion tracks adding support for horizontal scaling to Sourcebot.

TL;DR

• I think the existing single, self-contained docker container is great since it's super easy to use and is radically simple.
• But it's probably not practical in larger deployments, especially ones with thousands of repositories.
• We've seen instances where when a machine is under heavy indexing load, the webapp can become unresponsive or even OOM errors can occur.
• I think the first logical step here is to distribute our services as separate images (i.e., ghcr.io/sourcebot-dev/(webapp|worker|zoekt))
• Upstream dependencies like Postgres, and Redis would be separate.
• More images = more complexity, so I think the best mechanism to manage this complexity is moving towards declarative configs. We could do the following:
  • For local deployments, we can define a docker-container.yml at the root of the repository. Deploying will now involve 1) Cloning the repo, and 2) running docker compose up.
  • For production deployments, we can use technologies such as Helm, Terraform, Railway, and NixOS
• I think it will be important to have sensible defaults (s.t., it's very easy to deploy), while having deep configuration options that are well documented (s.t., it fits the user's deployment story).

Open questions / investigations

• What will we need to do to make each one of our services (webapp, worker, zoekt) to support multiple replicas at the same time?

Is this Overkill?

I'm trying not to go too overboard with the design here. DX should not be sacrificed with this change: ideally it should improve. Sourcebot should be just as easy to deploy as a vertically scaling service, both locally and in production. Deploying it as a horizontally scaling service in production should also be easy, while including configuration depth to support a wide range of deployment scenarios.

Locally, using Docker Compose will streamline deployment since a) it's a single command, docker compose up, and b) it's declarative, so all of the options (w/ sensible defaults, or commented out) can be included inside the docker-compose.yml.

In production, using Helm, Terraform, NixOS, etc. will help get people setup quickly. Again, sensible defaults here are key.

We also should continue supporting the self-contained image since to avoid introducing a breaking change here. We can evaluate if it's still worth shipping whenever we do a v5.

[abraverm]

The more deployment types you would support the more issues would be reported and divarage your attention from the core. The nature with Open Source project is that there would be many deployment types and I believe it is better for the project to be the gathering place for them. That said I think a better startucture would be having a seperate repository for each deployment type, but centralized integration test suite. The project could commit to two type of deployments, lets say Docker (single container) and Kubernetes (Helm), and to the test suite repository. Maturity and stability of other deployment repositories would depend on which version of Sourcebot and test suite it passed. That said, I believe the critical point for horizonal scaling of Sourcebot is the integration test suite, there needs to be an easy path for one to report and test issues with any type of deployment.

[kquinsland]

The more deployment types you would support the more issues would be reported and divarage your attention from the core.

My thoughts, more or less exactly.

I can say precisely zero nice or polite things about helm and many, many, many many things about all of helm's many faults and poor design decisions that have inspired countless chart maintainers to come up with bespoke/hacky workarounds.

For essentially that reason, i'd suggest not bothering and letting $otherPeople maintain the seemingly unbounded number of ways to deploy $something into k8s.

Publish well tested docker containers and a robust docker-compose file to show how each container interacts with the other(s) and comment it with good rules of thumb for scaling. With a high-quality lowest-common-denominator, people are then free to make their own kustomize stack(s) or helm chart(s) or even just run that docker compose file through tools like kompose as per their individual needs and constraints.