Monorepos and Workspaces

[adam-beck]

One thing I haven't quite figured out how to do properly is running a development environment -- with Docker Compose -- for a monorepo that is using workspaces. It would be awesome if anybody had advice (or if this sort of setup is possible). There seems to be very little literature online about this topic. In Bret's Udemy course, there was mention of monorepos in the Q/A, but it's 3 years old and doesn't seem to solve the problem the "docker way". I would love to provide an example in this repo as well but, again, I'm not sure where to even begin.

I've been oblivious to some of the more recent changes to Docker in the last couple of years. So part of it may just be my ignorance. Lately, I've just been developing outside of containers, which never sat well with me. I'm hoping there has been some new feature that I'm missing to make this easier than it sounds like it would be.

Monorepos seem to diverge from the purpose of what a container is/does. That is, each project should be isolated. But also, for development, there needs to be a way to house the "root" workspace and inject the "packages" as well. Almost seems like it wouldn't work.

[BretFisher]

It might help to describe specific issues that a monorepo presents for docker.

Compose is still the right tool I think, if you want to dev in Docker. I would use the watch feature for sure, and maybe the new includes feature. Tilt (acquired by Docker) would be an option if wanting to dev in k8s. You may use Turborepo or Nx for package management.

This could be a good thread to document common monorepo issues and docker-ish solutions.

[adam-beck]

So I'll be honest: I haven't tried much yet. The problem I'm anticipating is handling workspaces. I know each tool (npm, yarn, pnpm, bun) all work a little different, so I'll just start with npm as I'm assuming it's the most commonly used.

I have the following app structure

./docker-npm-workspace
├── compose.yml
├── package.json
├── package-lock.json
├── node_modules/
│   ├── workspace-app (symlinked to ../workspace-app)
│   ├── workspace-lib (symlinked to ../workspace-lib)
│   ├── ... all* packages from all workspace projects ...
├── workspace-app
│   ├── index.js
│   └── package.json
└── workspace-lib
    ├── index.js
    └── package.json

* if there are different versions, then potentially a project will have its own node_modules where that module would live. This wouldn't be common, but it is possible.

workspace-app is an express server. workspace-lib is just a createLogger function.

I'm hoping to spend time on this over the weekend. But what I don't understand is how to best handle the "root" files (specifically node_modules). Each service has its own Dockerfile which works great for production builds. However, on my host machine for development, there is no node_modules within each service. So I don't know how to translate this into Docker Compose. Even if I were to use npm install for each service, I would lose the benefits of workspaces and would have to rebuild the image whenever a dependent project changes. That's not the case without Docker Compose. When I make a change to workspace-lib, I want to restart workspace-app and see the changes.

I'm probably describing the issue poorly and I realize it's a little unfair to have others look at this when I myself haven't really tried anything. Like I said, I'm hoping to take a look at this over the weekend.

[adam-beck]

Not sure if I'm stuck in analysis paralysis or what. I really only had 1 idea and it didn't pan out (probably because I'm using volumes incorrectly).

Basically, what I tried was having the "root" of the project be its own service container. It wasn't a running server or anything, but the Dockerfile copied the entire tree (COPY . .) and RUN npm install. This got me my monorepo in a single container.

workspace-lib has no build step and doesn't need to be running, so it doesn't have a Dockerfile. workspace-app was a little trickier because I want its Dockerfile to be self-contained, and it needs the package-lock.json from the parent. I found this article where the author just manually copies it -- temporarily -- for the sake of building. But that article doesn't deal with inter-workspace dependencies. I don't want to lose that benefit. That's the whole point. Regardless, I want to focus on development and not building at this point.

I'm still giving this some thought. It looks like my option is going to be to have one Dockerfile for all my workspace projects. I can use the node_modules from the root and place them in the sub-images. I still haven't determined how I will get the "update workspace-lib and be able to restart workspace-app to see the changes".

[adam-beck]

This repo seems to get you part of the way there: https://github.com/moofoo/turbo-docker-monorepo.

There is an associated blog post by the author: https://dev.to/moofoo/creating-a-development-dockerfile-and-docker-composeyml-for-yarn-122-monorepos-using-turborepo-896

But unfortunatley, there is still no way to see the changes in the shared library without rebuilding.

[adam-beck]

Okay I got something that is close to what I was envisioning. I'm still trying to improve it. My hope is that there are some existing tools (like turborepo) that can simplify some of it.

I wanted a monorepo while still maintaining individual services. This bends the rules of what a monorepo is in a way. Because when you develop a monorepo locally, it just expects all the code to be accessible. When you have individual services, you want to isolate the code to only whats needed. Now, from a build perspective that makes sense. But I was after a "as-close-to-prod-as-possible" setup. So here is my compose file:

services:
  
  monorepo:
    build:
      context: .
    volumes:
      - monorepo:/monorepo
    command: []

  server:
    build:
      context: .
      dockerfile: apps/server/Dockerfile
    depends_on:
      - monorepo
    init: true
    ports:
      - "3000:3000"
    command: /monorepo/node_modules/.bin/nodemon -w /monorepo/packages -w /monorepo/apps/server /monorepo/apps/server/index.js
    develop:
      watch:
        - action: sync
          path: ./apps/server
          target: /monorepo/apps/server
    volumes:
      - monorepo:/monorepo
        
  greet-service:
    depends_on:
      - monorepo
    build:
      context: .
      dockerfile: apps/greet-service/Dockerfile
    init: true
    ports:
      - "3001:3000"
    command: /monorepo/node_modules/.bin/nodemon -w /monorepo/packages -w /monorepo/apps/greet-service /monorepo/apps/greet-service/index.js
    develop:
      watch:
        - action: sync
          path: ./apps/server
          target: /monorepo/apps/server
    volumes:
      - monorepo:/monorepo

volumes:
  monorepo:

The monorepo service is an ephemeral service that just gets all the code into a shared volume. From there, the other services (node servers) "attach" into that volume. It's all pretty ugly. It's not like the services can be completely oblivious when working as part of something bigger. I'm just not used to this with Docker.

Anyway, here is the entire project. The README is just my "steps" along the way. I did manage to put the entire monorepo into a single service (as can be seen on the main branch) and that worked great. But then I miss out on some of the other stuff Docker provides.

https://github.com/adam-beck/my-monorepo-attempt/tree/can-i-have-individual-services-for-servers

[BretFisher]

Pinned. Good walkthrough of your thought process and trials!