Base image resource does not get deployed due to missing image

[samuliy]

Expected Behavior

Running two resources, first one using a Dockerfile extending from the second, second using Dockerfile that is the base for the first one. Both resources, the one using the base image and the one extending the base image should deploy without an error.

Current Behavior

1. First resource builds the base image

STEP 1/5 — Building Dockerfile: [x/tilt-deployment-base]
Building Dockerfile for platform linux/amd64:
...

2. First resource skips pushing the base image to repository:

STEP 2/5 — Pushing x/tilt-deployment-base:tilt-c284e86bc661040f
     Skipping push: base image does not need deploy

3. The resource using the base image fails due to missing image (masked the registry and namespace from the log output):

Tracking new pod rollout (tilt-deployment-base-6dc4c5d649-pjncq):
     ┊ Scheduled       - <1s
     ┊ Initialized     - <1s
     ┃ Not Ready       - (ContainersNotReady): containers with unready status: [tilt-deployment-base]
[event: pod x/tilt-deployment-base-6dc4c5d649-pjncq] Pulling image "xxx/tilt-deployment-base:tilt-c284e86bc661040f"
[event: pod x/tilt-deployment-base-6dc4c5d649-pjncq] Failed to pull image "xxx/tilt-deployment-base:tilt-c284e86bc661040f": rpc error: code = NotFound desc = failed to pull and unpack image "xxx/tilt-deployment-base:tilt-c284e86bc661040f": failed to resolve reference "xxx/tilt-deployment-base:tilt-c284e86bc661040f": xxx/tilt-deployment-base:tilt-c284e86bc661040f: not found

Steps to Reproduce

This issue can be somewhat hard to reproduce, as it feels like it's related to some race-condition.
I can consistently reproduce it with the following steps, by adding a local_resource with a sleep command and setting the resource_deps in a way that the base-image resource builds later.

1. Files to reproduce the issue with: test.tar.gz

2. Run tilt up

Context

tilt doctor Output

$ tilt doctor
Tilt: v0.33.22, built 2025-01-03
System: linux-amd64
---
Docker
- Host: unix:///var/run/docker.sock
- Server Version: 27.4.1
- API Version: 1.47
- Builder: 2
- Compose Version: v2.28.1
---
Kubernetes
- Env: kind
- Context: kind-kind
- Cluster Name: kind-kind
- Namespace: default
- Container Runtime: containerd
- Version: v1.32.0
- Cluster Local Registry: &RegistryHosting{Host:localhost:5001,HostFromClusterNetwork:,HostFromContainerRuntime:,Help:https://kind.sigs.k8s.io/docs/user/local-registry/,SingleName:,}
---
Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗

The info below helps us understand how you're using Tilt so we can improve,
but is not required to ask for help.
---
Analytics Settings
--> (These results reflect your personal opt in/out status and may be overridden by an `analytics_settings` call in your Tiltfile)
- User Mode: opt-out
- Machine: 8a6cac578b8f7d68ec08cb2f39a8a246
- Repo: IdDstfTxPu7tcDfMm9XZvA==

About Your Use Case

Use case is the ability to run the base image and see it live, while also being able to extend from it and have the changes into base image propagate into child-resources that use it.

During Tilt startup, this can be mitigated by defining resource_deps in a way that the images build in correct order, but when making changes to the images and updating the resources live, it's possible the build order is incorrect.

[nicks]

thanks for the report! as a temporary workaround, i think if you set up the resource_deps so that the manifest with the base image gets deployed first, i think it will work ok.

[samuliy]

Thanks for the reply! Setting resource_deps does indeed help when running tilt up, but doesn't help when the resources are already up, and we make changes to files that cause image rebuilds.

[samuliy]

I built Tilt with the new merged changes and did not run into the issue, at least in the couple of hours I worked on my projects. However, one time I managed to get one of my deployments into a state, where I assume the base image was built with an older set of files than what was on my disk. I'm not sure how to reproduce this or if it was something that I messed up. If this doesn't ring any bells right now, I can try to check if I can make a reproducible scenario and try to figure it out later.

Thanks for working on this!