Deploying Turborepo with Docker and CI/CD documentation

[hsellik]

Summary

I recently tried breaking up a frontend application to a monorepo with both Nx and Turborepo. I found most friction with setting up the monorepo with both CI/CD and Docker. The official Turborepo examples are only for with CI and without Docker or with Docker and without CI.

Similar discussion is raised for TurboRepo and Google Cloud with no answers. There is also a very similar discussion for Nx with no conclusion. However, the description of the problem is exactly the same:

Having a read though the documentation there seems to be a real lack of examples and best practices on how to deploy your shiny new project and searching around for articles is a sea of outdated or hacky looking workarounds.

I would like to discuss how the community is deploying their repos and some best practice input from the Dev team and then either help write some docs or at least curate them into an article. I am really enjoying developing with nx but I find the docs lacking.

A secondary thing would be some examples of CI/CD configs for the main providers for different configs.

I feel that most of the first time implementors face similar challenges and most CI/CD setups have the following goal: only run the deploy steps for the affected applications and at the same time display the steps in the CI/CD pipeline.

I would like to discuss with the community on the best practices for deploying a Turborepo with Docker with CI/CD.

Removing this friction would make Turborepo a superior choice for most monorepo setups. Perhaps a solution here could help the Nx community as well.

[anthonyshew]

Would love for folks to share their learnings!

For context, the reason you don't see a highly bespoke setup in examples or documentation is that almost no one builds the same CI. The examples and docs you see are meant to be focused on one tool at a time, and we trust developers to bring together all of the pieces they need for their specific case. The examples/docs walk right up to the point where the spread of questions people ask isn't something we can answer to in a quick, generalized way.

As an example, "how to Docker + CI" instantly widens out to a vast matrix of factors like:

• CI vendor
• Docker registry being used (if at all?)
• Infrastructure vendor
• Tools being used in the container itself ("How do I use Prisma with this now?", "That's not the package manager I'm using, I need an example with my package manager now", etc. )
• Platform (Windows, macOs, Linux, FreeBSD, etc.)
• ...on and on...

We can provide two examples - one for Docker, one for CI - and let you bring those two ideas together in a way that's meaningful for you. The goal isn't necessary to create code that you can copy and paste together, but to show the foundations and concepts so that you can build off of the code we do provide.

Additionally, each example/doc adds maintenance burden and we struggle to keep up with the ecosystem with what we have now. There's constant innovation out there, and we try to do our best to keep what we have updated, but we depend on the community for much of that maintenance. The more examples, docs, and other tools that we document, the more we get taken away from our core mission.

I can understand how that might not be the answer you're looking for, and I'd be interested in finding out if there's a way that this answer can be different in the future through a feature set in Turborepo that's so outrageously robust that we can do all the work of gluing all of your tools together. However, that day isn't today and we're focused on making sure we do our core job spectacularly first.

[hsellik]

Thank you for taking time to answer on the matter. If you are ever planning to do such a guide to reduce developer friction, I would be perfectly happy with an example for any package manager with any CI vendor. It would be much easier to get a general idea on how Turborepo should be used. I do understand that this would probably lead people to want versions for their specific vendors.

From what I understand, the way one builds applications with CI and Docker is:

Only deploy affected apps:

1. Run install -> build -> test as in GitLab guide
2. Run npx turbo-ignore <app> || exit_code=$? && echo "BUILD_<APP>=$exit_code" >> turbo_ignore_variables.env for each app to figure out if it's affected. Store it to env variable file or directly to env variable if CI allows. Some script could be used to dynamically figure out the apps or a new line needs to be added manually for each app.
3. For each app, trigger Docker build separately according to the Docker guide. Env variables created in step 1 can be used to skip unchanged apps.
   3.1) Push the docker images to the registry
4. According to turbo-ignore variables, only deploy the ones that are affected. Also requiring check for each application separately.

Should we trigger these build and deploy steps manually/with script in CI based on outputs of turbo-ignore or should we somehow be able to run these steps with a Turborepo command? Some command perhaps which would trigger build and deployment only for affected apps?

Deploy all apps:

1. Run install -> build -> test as in GitLab guide
2. For each app, trigger Docker build separately according to the Docker guide
   2.1) Push the docker images to the registry
3. Run deploy for all the applications

Most of the steps are probably cached, but building and deployment still happens. Depending on the circumstances, the newly deployed images may or may be actually deployed again.

[anthonyshew]

There will be a --affected flag in the next release to use: #8884