Latest images on docker hub don't contain latest ROS packages

[kartikmohta]

Docker hub (https://hub.docker.com/r/osrf/ros/builds/) shows that the latest builds were created 23 days ago (Dec 3) but after launching the indigo-desktop-full image and running apt-get update && apt-get dist-upgrade, a lot of ROS packages are marked for upgrade, for example:

Preparing to unpack .../ros-indigo-catkin_0.6.16-0trusty-20151110-2013-+0000_amd64.deb ...
Unpacking ros-indigo-catkin (0.6.16-0trusty-20151110-2013-+0000) over (0.6.14-0trusty-20150424-0354-+0000) ...
Preparing to unpack .../ros-indigo-rosbuild_1.11.11-0trusty-20151110-2358-+0000_amd64.deb ...
Unpacking ros-indigo-rosbuild (1.11.11-0trusty-20151110-2358-+0000) over (1.11.8-0trusty-20150522-1131-+0000) ...
Preparing to unpack .../ros-indigo-rosconsole_1.11.16-0trusty-20151111-0003-+0000_amd64.deb ...
Unpacking ros-indigo-rosconsole (1.11.16-0trusty-20151111-0003-+0000) over (1.11.13-0trusty-20150522-1135-+0000) ...
Preparing to unpack .../ros-indigo-rosgraph-msgs_1.11.1-0trusty-20151111-0016-+0000_amd64.deb ...
Unpacking ros-indigo-rosgraph-msgs (1.11.1-0trusty-20151111-0016-+0000) over (1.11.1-0trusty-20150522-1154-+0000) ...
Preparing to unpack .../ros-indigo-xmlrpcpp_1.11.16-0trusty-20151110-2344-+0000_amd64.deb ...
Unpacking ros-indigo-xmlrpcpp (1.11.16-0trusty-20151110-2344-+0000) over (1.11.13-0trusty-20150522-0939-+0000) ...
Preparing to unpack .../ros-indigo-roscpp_1.11.16-0trusty-20151111-0411-+0000_amd64.deb ...
Unpacking ros-indigo-roscpp (1.11.16-0trusty-20151111-0411-+0000) over (1.11.13-0trusty-20150522-1157-+0000) ...

I would have expected that the latest images on docker hub would contain latest versions of all the included ROS packages (as of the image date), is that not correct?

[tfoote]

There are new releases of packages every couple of hours in the ROS ecosystem. Since ROS is a distributed project the standard docker hub tools for monitoring repositories to trigger rebuilds doesn't work. The builds on docker hub are just periodic snapshots.

[kartikmohta]

It's just that some of the package versions are more than 6 months old! For example, roscpp version in the latest indigo-desktop-full docker image is 1.11.13 which was released in May, whereas the latest roscpp version is 1.11.16 released in Nov. I don't know whether that is expected or not, but believe that it shouldn't be the case for docker images built less than a month ago.

[ruffsl]

@kartikmohta is this something that would be resolved with the meta package version bump, like specified here? I made a small PR #26 to automate the maintenance. I just checked it again just know (it checks online for current ros package versions) and I didn't observe any version changes.

[kartikmohta]

Actually the problem is that the ros-base images are part of the docker official images which are not regenerated very often (I think). ros-*-desktop images depend on the ros-base images, so they get the old versions of packages in ros-base images. Also, on the docker hub page for the official ROS images (https://hub.docker.com/_/ros/), it doesn't show when they were built, so there's no way to know how new they are.

I pulled ros-indigo-ros-base right now and seems like the image was updated recently since it has updated ROS packages in there.

[ruffsl]

Well, looking at the header info for both repos lists that they "Last pushed: 2 days ago". The base image that the official ros repo depends on, ubuntu:14.04, it updated rather regularly (almost weekly).

Although docker history on the ros repo shows up as 8 days; not sure why the discrepancy.

$ docker pull ros
Using default tag: latest
latest: Pulling from library/ros
Digest: sha256:d93437d7749e6dc396f8ce948eee1b737be6c1d1e5869d2d4c3bad9919c27dc4
Status: Image is up to date for ros:latest

$ docker history ros
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
24ebf5532804        8 days ago          /bin/sh -c apt-get update && apt-get install    14.46 MB            
ab31c0fa91fd        8 days ago          /bin/sh -c #(nop) MAINTAINER Dirk Thomas dtho   0 B                 
2dd56b814a75        8 days ago          /bin/sh -c #(nop) CMD ["bash"]                  0 B                 
daa6642c85a7        8 days ago          /bin/sh -c #(nop) ENTRYPOINT &{["/ros_entrypo   0 B                 
6ca21ec334c4        8 days ago          /bin/sh -c #(nop) COPY file:5e239e5268e921076   95 B                
80f5f1bda9dc        8 days ago          /bin/sh -c apt-get update && apt-get install    531.2 MB            
175c23c23c9a        8 days ago          /bin/sh -c rosdep init     && rosdep update     1.627 MB            
ffd62193e6a0        8 days ago          /bin/sh -c apt-get update && apt-get install    89.89 MB            
497eaee16f64        8 days ago          /bin/sh -c #(nop) ENV ROS_DISTRO=indigo         0 B                 
17d6beedb4e1        8 days ago          /bin/sh -c echo "deb http://packages.ros.org/   51 B                
d8addcf153a4        8 days ago          /bin/sh -c apt-key adv --keyserver ha.pool.sk   25.83 kB            
07c80fb1909a        8 days ago          /bin/sh -c #(nop) ENV LANG=en_US.UTF-8          0 B                 
e19b67a7b142        8 days ago          /bin/sh -c locale-gen en_US.UTF-8               1.621 MB            
6d9a22b8f5d2        8 days ago          /bin/sh -c #(nop) MAINTAINER Dirk Thomas dtho   0 B                 
c4bea91afef3        8 days ago          /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B                 
d9e545b90db8        8 days ago          /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/   1.895 kB            
4cdc0cbc1936        8 days ago          /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   194.5 kB            
fcee8bcfe180        8 days ago          /bin/sh -c #(nop) ADD file:482777a8ad62044151   187.7 MB            

There is a cascading problem (that the official docker repos do not suffer from, but non official automated repos do) that results in child tags being built from parent tags from the previous generation, e.g. the ros*desktop tags in our case is always one build behind. But like I mentioned, the official repo churns over rather frequently, so for the unofficial release under osrf profile for desktop support it should be ok.

If you really need up-to-date desktop images, you could make your own automated repo that points to the official ros repo directly, and do a chain of triggering automated repos for any rigorous hierarchy. It just gets tricky to get away with hierarchy of tags like in the official repos. Short of setting of a third party CLI to build and push the images, I'm not sure how to resolve this with docker hub just yet. I'll search for a ticket, and submit one if I don't already find a similar dockerhub feature request. We revisited this over in related issue NVIDIA/nvidia-docker#18. If you having thoughts on it, let me know.

[ruffsl]

I've added a "Compare Images" link to each repo listed in the readme, here is the one for ros and for osrf/ros. The official ros repo looks in good order, but osrf/ros has a peculiar divide in it's lineage thanks to what I mentioned above.

ros	osrf/ros

But anyhow, @kartikmohta are you still experiencing old packages in fresh pulls of osrf/ros?

[kartikmohta]

I've set up our CI system to run a apt-get dist-upgrade in the image in order to avoid this problem, so haven't checked recently. Can check again later today.

[kartikmohta]

@ruffsl It seems that it's still an issue, I guess the problem right now is that the ubuntu repo hasn't rebuilt in a while:

$ docker history ubuntu
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
3876b81b5a81        4 weeks ago         /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
<missing>           4 weeks ago         /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/   1.895 kB
<missing>           4 weeks ago         /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   194.5 kB
<missing>           4 weeks ago         /bin/sh -c #(nop) ADD file:7ce20ce3daa6af21db   187.7 MB

leading to the ros repos not rebuilding and so on...

I guess I'll keep the apt-get dist-upgrade for now, until I get to making our own triggered repo.

[ruffsl]

I guess I could see why they might like to keep the official ubuntu image calm for a while unless a change is required, otherwise more than a half of Docker Hub would be thrown into a rebuild, a non-trivial load and not to be triggered lightly.

My suggestion would be in perhaps use the ros_core image, as so any exotic dependency would be freshly installed. I'm not aware of how active the packages under ros_core meta-package are but hopefully they not as critical to be bleeding edge current.

If you roping in a number of packages that are basically as large as desktop-full, I might also recommend using squid-in-a-can as a simple proxy cache while builing containers, somthing tfoote showed me last summer. Might also want to modify the compose file to enable a persistent volume for your squid cache.

[kartikmohta]

Well I created the issue since the version of roscpp was old in the docker image so using the ros_core image wouldn't have helped. Anyway, I think I'll keep the apt-get dist-upgrade for now. Thanks for the sharing the details of the process! You can close the issue if you think that is ok.

[ruffsl]

Ok @kartikmohta , thanks for the feedback! When Docker Hup improves the automate build process, I'll try and fix lagging osrf/ros tags as well.