support docker-compose configs from multiple directories

[landism]

Describe the Feature You Want

The ability to compose multiple docker-compose files from different projects, e.g. docker_compose(['proj1/docker-compose.yaml', 'proj2/docker-compose.yaml'])

Current Behavior

We infer the DC project directory from the first config provided and then use that for all other DC configs.
This means, e.g., paths in proj2 are interpreted relative to proj1, even though they are likely written to be interpreted relative to proj2.

Why Do You Want This?

Onboarding to Tilt and wanting to run multiple DC projects at once

Notes

• This actually doesn't work in DC - When using multiple docker-compose.yml files from different directories, local paths are not followed correctly. docker/compose#3874
• Maybe this should actually be a bug instead of a feature request? Except we're matching the underlying DC behavior, so a user migrating from DC couldn't have had this working in the first place.

[nicks]

re: "Maybe this should actually be a bug instead of a feature request? Except we're matching the underlying DC behavior,"

ya, there was a fairly long discussion on this when we first added support for docker_compose, and we ended up matching the behavior in the spec:

https://github.com/compose-spec/compose-spec/blob/master/spec.md#compose-file

Multiple Compose files can be combined together to define the application model. The combination of YAML files MUST be implemented by appending/overriding YAML elements based on Compose file order set by the user. Simple attributes and maps get overridden by the highest order Compose file, lists get merged by appending. Relative paths MUST be resolved based on the first Compose file's parent folder, whenever complimentary files being merged are hosted in other folders.

Seems reasonable to add support for independent docker-compose apps; it's just something we punted on

[rwoll]

We'd find this feature super useful as we have several docker compose projects we want to manage at once with Tilt, but they have clashing configs (default network name overrides) that cannot be combined.

This is what I wanted to write:

docker_compose(["/path/to/internal/backend/docker-compose.yml"])
docker_compose(["/path/to/secondary/infra/docker-compose.yml"])

Without Tilt, these must be run as:

$ cd /path/to/internal/backend
$ docker-compose up

$ cd /path/to/secondary/infra
$ docker-compose up

but it looks like Tilt is trying to do docker-compose -f /path/to/internal/backend/docker-compose.yml -f /path/to/secondary/infra/docker-compose.yml.

[mcreenan]

Has anyone found a decent workaround for this?

Would it be possible to implement a custom docker_compose function in tilt/starlink that would support this by not combining all docker-compose files, even when called in separate docker_compose calls?

[nicks]

Currently the backend supports this. i think it just needs to be threaded through the tiltfile api.

I think part of the problem here is that, in the interviews with compose users that I've done, MOST people don't deeply understand how compose-file overrides work. Some people don't use them at all, some people use them a lot, and some people use them accidentally (when what they really want is multiple compose projects).

One of the downsides of the current tiltfile API for docker-compose is that it's hard to come up with an API that fits what all 3 groups expect.

One API I considered was something like:

docker_compose('a.yaml', overrides=[...])
docker_compose('b.yaml', overrides=[])

where the presence of the overrides argument means "yes, I know what overrides are, and i want this to be a new project". But from an API standpoint, it's odd that overrides=[] behaves differently than when the argument is omitted.

alternatively, something like:

docker_compose(project='a.yaml', overrides=[])

where the presence of project= triggers "this is a new project"

i could also imagine trying to infer from the compose file itself whether it's an override file or a project file. That might be a can of worms though, maybe @milas or @nicksieger have thoughts on this idea (since they are now compose maintainers)

[mcreenan]

Thanks for the info.

I really like the idea of project being the indicator, especially since I already use that in my use case.

Alternatively, a backwards compatible way would be a new parameter that tells docker_compose to invoke the docker-compose files independently, that would default to False. That way, projects would have to opt-in to that behavior.

For anyone else that's run into this, I did start work on a tilt extension to create an alternate to docker_compose that runs each call separately. My testing so far indicates it works but I haven't done really thorough testing yet. https://github.com/mcreenan/tilt-extensions

[nicksieger]

I think trying to infer overrides by looking at files is going to be tricky without knowing how the developer wants the files to be applied. The fact that Tilt lumps multiple files into a single project regardless of the number of docker_compose statements might contribute to the confusion by breaking peoples' tendency to think that one docker_compose == one docker compose up.

I do like the idea of introducing a project= or project_name= optional argument as a way to support multiple projects that could be compatible with the current implementation.

@mcreenan can you say a little more about your setup where you do want separated compose projects (and everything that goes with them, notably, separate networks to which the project containers are attached)?

[mcreenan]

@nicksieger Right now we just use the default network for each docker compose projects. I will be looking into making them share a network, but that's much less important.

The short of it is:

• We have a main monolith app with a frontend and backend/api in one repo, then some microservices in their own repos with a frontend and backend/api.
• The monolith app doesn't directly communicate with the microservice apps, only indirectly via the browser (redirects/form posts)
• The microservice apps do communicate with the monolith app, calling the monolith app's api from the backend
• The monolith app doesn't currently run inside docker containers locally
• The microservice apps do run inside docker containers

So we want to be able to run the monolith app and multiple microservice apps in parallel with one control plane.

[nicksieger]

Ok, that setup makes sense. So what is it about Tilt's current behavior that merges multiple docker compose statements and/or files into a single project that's problematic? Is it the use of relative paths in each compose file that gets broken when merging?

[mcreenan]

@nicksieger Tilt won't even allow it, as it throws an error when you try to use docker_compose in separate Tiltfiles as it uses the path of each Tiltfile as the working directory. So if they are in separate directories, it errors.

See

tilt/internal/tiltfile/docker_compose.go

Lines 68 to 73 in 5d444b8

	currentTiltfilePath := starkit.CurrentExecPath(thread)
	if dc.tiltfilePath != "" && dc.tiltfilePath != currentTiltfilePath {
	return starlark.None, fmt.Errorf("Cannot load docker-compose files from two different Tiltfiles.\n"+
	"docker-compose must have a single working directory:\n"+
	"(%s, %s)", dc.tiltfilePath, currentTiltfilePath)
	}

[ElementalWarrior]

I think this mostly is solved. One small issue I've encountered so far with microservices via docker-compose is that you can have containers with the same name in different repos. But you will get an error for resource already existing.

Error in docker_compose: dc_resource named "db" already exists

[nicksieger]

There's a new argument to dc_resource called new_name that will help with this issue. You will need to change the name of the first resource before the second docker_compose call is made.

https://docs.tilt.dev/api.html#api.dc_resource

[Wulfheart]

@nicksieger how would this look like?

[nicksieger]

@Wulfheart for example, if you have two compose files a.yaml and b.yaml that both define a service named foo, you'd write in your Tiltfile

docker_compose('a.yaml', project_name='a')
dc_resource('foo', project_name='a', new_name='a-foo')
docker_compose('b.yaml', project_name='b')
# This second one is not necessary for disambiguation, it just adds the prefix for consistency
dc_resource('foo', project_name='b', new_name='b-foo')