incremental building: docker:build-watch for detecting new SNAPSHOT jars in flat classloaders that go into /maven

[jstrachan]

when working with Apache Karaf there's a really handy command dev:watch *. You provision your bundles to karaf then if you rebuild any jar/bundle thats a SNAPSHOT via "mvn install", this command will watch your local maven repo for any SNAPSHOT jars that are deployed that get rebuilt. If they do it redeploys those bundles into the Karaf container. (For more detail see the watch section at the bottom of this page.

As a developer you just run dev:watch * then any mvn install in any other maven project for a SNAPSHOT jar will redeploy your stuff! Its a very neat way to reduce the compile/build/deploy time.

It'd be cool to have something similar for folks developing flat class loader docker images (e.g. camel boot / spring boot / dropwizard / wildfly swarm).

Imagine if we could do something like:

mvn docker:build-watch -Dpost-command="fabric8:delete-pods"

This would then look at all the jars in the /maven folder for the docker image (using the local target/docker/...maven folder that the mvn docker:build created locally), find all the SNAPSHOT jars/wars and watch them in the ~/.m2/repository local repository until a user hits a key to break the watch loop.

For any jar/war thats changed, we know that this jar needs to go into /maven in the docker image; so we create a new image, based on the last image created, copying this new jar into the place we know it goes.

i.e. we'd get development time fast docker builds; only the changed jars need to be POSTed (or ADD'ed to the Dockerfile) so it should be really fast.

i.e. we assume a full valid build is done first via mvn docker:build; then mvn docker:build-watch is an incremental build from then on watching to see if any artifact included in the docker image changes; we're then doing the absolute minimum to create a new image based on the last one, only copying changed jars into the new image.

This is only for incremental development mind you; we'd use the real mvn docker:build for releases. It'd just really cut down the amount of time developers spend typing mvn commands and building

[rhuss]

If I understand you right, you want to create a new image with a FROM on the just running container's image, with overwriting the changed jars in maven, then stop the old container and create a new container on the new image (or simply kill the old container when running in a kube env) ?

What should happen with the repo name of the old image ? It should probably moved to the new image (to the top layer).

[jstrachan]

@rhuss yeah thats right. The watching part could use volumes like this https://github.com/rhuss/docker-maven-plugin/issues/119 - or could create incremental images.

Yes, we should move the new incremental image to the new image name (e.g. 1.0-SNAPSHOT) (maybe label the old one as 1.0-SNAPSHOT.old.X' or something?

[rhuss]

Another idea for cleaining up image: Always create imags with a tag suffix like -inc (or whatever) which are treated specially. So there can be the real image (withou -inc) and the inc image. inc-images can be easily cleaned up (there's already a 'docker:remove' which can be extended), and there is no danger that they will be pushed accidentally.

One could even go so far to cleanup automatically with a ShutdownHook when using docker:start -Ddocker.mode=inc

[jstrachan]

we can add multiple labels to an image can't we; so we could use some 'incremental tag' prefix/suffix when we create the image; then when we label it with the project version (1.0-SNAPSHOT say) we can have 2 labels? So that the latest image (via mvn docker:build or mvn docker:build-watch would always be 1.0-SNAPSHOT so no need to update docker compose / kubernetes json/yaml. But all incremental images could have an extra label on them used for GC?

[rhuss]

I see your point with references in deployment descriptors which would then be different.

My point is, that although you can have multiple labels you can have only one unique tag. If we promote (and maybe even verify) that 'development'/'incremental' image may have only -SNAPSHOT or latest I would be happy, since this clearly indicates that its about a non-production image.

But maybe we should start small and simply recreate the whole image, maybe with (flag-enabled0 removing the old image after building it so that we keep a clean repo. Might be not so fast, but its much easier than to create a custom image layer on top (frankly, I dont know yet how to do this, keeping all the metadata like commands etc. on the top layer)

[jstrachan]

Yeah. Maybe in incremental/continuous build mode we should just use latest as the docker image label anyway? Then it's clear from the image name in the kubernetes JSON that we're using the latest snapshot?

Might be nice in general in the docker plugin to default the image tag to latest if the project version ends in SNAPSHOT too? In fabric8 we tend to use ${project.version} in the docker.image property - maybe the docker plugin should generate its own property folks can use - say - ${docker.image.tag} which if not defined it's set to latest for snapshots or use ${project.version}?

[rhuss]

latest as a default might be a good idea except that it also often means the latest, stable version. Especially when you forget to add a tag to an image (e.g. docker pull fabric8/fabric8) it will look for latest and if latest is then the bleeding edge, super duper buggy development image, I'm not sure if that is what the user expects. I'm still puzzled whether 'latest' should be interpreted as the latest cutting edge image or the latest stable release.

---

You can omit the tag when specifying the image name, that implicitly creates a latest in the plugin.

Also, you can always add more tags with the <tags> configuration (or with the docker.tags.1 etc. properties) by your own (which e.g. can be combined nicely with some dist profiles etc.)

[rhuss]

Good news, I found a rather easy way to find the list of source files in an assembly (using a TrackArchiver to create a dummy archive) so detecting when something changes shouldn't be hard anymore.

Next step is to detangle the BuildMojo and StartMojo since both are needed for a proper watch: Building new images and restarting them. I consider to introduce a WatchMojo and extract common functionality to BuildMojo and StartMojo into base classes which in turn is again surprisingly involved (plexus components vs. maven parameters etc., you know, all this ancient spaghetti stuff).

When it comes to rebuilding images during watching, my plan is to support the following features:

• Complete rebuild of the original image
• Incremental build with overlaying only the changed files (i.e. a Dockerfile only with a single FROM pointing to the current image and one or more ADD directives)
• Incremental with periodical complete rebuild (and removing the incremental layers up to this point in time). This should be a good compromise between the extra space and FS overhead with incremental builds and the increased build time when using complete images.

Options which could influence this behaviour:

• watchInterval
• mode :
  • build --> rebuild containers
  • run --> start/stop containers
  • both --> ...
• goalHook : Provide a dedicated maven goal to call after a file change has been detected (using mojo-executor --> useful for e.g. calling Kubernetes API to delete a pod so that it can refetch a freshly build image

There are still some interesting open questions:

• What about initially dynamically assigned ports which are probably internally used via Maven props ? Should they be kept stable ? (e.g. for linke containers ?) Or should new dynamically ports be assigned and linked containers which are using these ports should be restarted, too ?
• What about exposed volume ? Should running container which has those volumes be mounted also be restarted ?
• When to do the cleanup of the incremental images ?

My roadmap for this is:

• Implement the full image rebuild first, without taking care about any dependencies
• Next, incremental images
• Next, periodic cleanup by interspersed full builds
• Then think about port dynamic port mapping and container dependencies
• The 'goalHook' for calling out after a rebuild

Hmm, quite some stuff. But I think its worth to go in this direction since it would make dev's life a lot easier.

What's your opinion on this ?

[rhuss]

These are the use case scenarios I have in mind:

• Local docker developer, starting multiple linked containers on her local machine (or boot2docker VM) with mvn docker:watch in a terminal. This calls blocks forever until CTL-C is called. She continues to develop, calling mvn install in subprojects or even in other projects. As soon as these activities touches on of the artefacts in one of the images started with docker:watch a rebuild (incremental or full) kicks in for the affected images. Then the container are restarted as well as any containers which depend on these changed images (link, volumes). And all in the proper order.
• Kubernetes developer, creating pods with fabric8:apply will also start mvn docker:watch but with -Ddocker.watchMode=build -Ddocker.goalHook=fabric8:delete-pods so that after the images are rebuild and then the affected pods are moved so that Kubernetes will fetch the freshly created image to rebuild the pods.

Any other use cases ?

[rhuss]

Mostly implemented, the rest has been distributed to new issues. Closing that one.