Two issues I'm noticing with using the docker-container driver to work around the caching issue:

[Volato999]

          Two issues I'm noticing with using the docker-container driver to work around the caching issue:

1. It adds some export/import steps
2. docker push seems to be pushing all layers?

With the default driver, rebuilds of code-only changes take ~1 minute (when I get proper caching of the expensive layers in my image).
With the docker-container driver, these 2 factors mean rebuilds after code-only changes take ~4-5 minutes.

export/import steps

#25 exporting to oci image format
#25 exporting layers done
#25 exporting manifest sha256:01230f6377dec5a6988c924373bb62afe2837d3afa7bb0e84e98a016481c1c81 done
#25 exporting config sha256:4f48d81bc559f074600e3088949591f885d4ef3c74b8d833408864b6bd013df4 done
#25 sending tarball
#25 ...

#26 importing to docker
#26 DONE 32.1s

#25 exporting to oci image format
#25 sending tarball 43.0s done
#25 DONE 43.0s

This seems to add an extra minute to the build. I'm working with large images (~3.5gb from various scientific Python libraries), which I'm guessing exacerbates this issue.

docker push issue

Pushing my 3.5gb image takes ~3 minutes.

It seems that with the docker-container driver, docker push isn't able to see that the expensive layers are shared, and it's pushing all the layers instead of only pushing the new layers? I'm guessing this based on the output from docker push not saying "Layer already exists":

6474dc186dfd: Preparing
2d80b2e557e9: Preparing
59149f33a870: Preparing
ed04f21afbe5: Preparing
c9ec67fe6421: Preparing
e42dc4266416: Preparing
a55e5a0e7c4a: Preparing
aef13dfbb6f9: Preparing
1e602bec2da5: Preparing
b1c4e3f331ea: Preparing
3fdf9f44ae06: Preparing
78ce42cd87aa: Preparing
82e21ae59256: Preparing
02c055ef67f5: Preparing
e42dc4266416: Waiting
3fdf9f44ae06: Waiting
a55e5a0e7c4a: Waiting
78ce42cd87aa: Waiting
aef13dfbb6f9: Waiting
1e602bec2da5: Waiting
82e21ae59256: Waiting
b1c4e3f331ea: Waiting
02c055ef67f5: Waiting
ed04f21afbe5: Pushed
59149f33a870: Pushed
c9ec67fe6421: Pushed
2d80b2e557e9: Pushed
aef13dfbb6f9: Pushed
1e602bec2da5: Pushed
b1c4e3f331ea: Pushed
3fdf9f44ae06: Pushed
6474dc186dfd: Pushed
82e21ae59256: Pushed
78ce42cd87aa: Pushed
02c055ef67f5: Pushed
e42dc4266416: Pushed
a55e5a0e7c4a: Pushed

I push several tags. The first push takes 3 minutes, and the rest of the tags finish quickly as they all say "Layer already exists" for all the layers.

Originally posted by @jli in moby/buildkit#2274 (comment)