docker release for 5.26.0

[cyril23]

Hey guys,

thanks for the release of 5.26.0 here. Great to see the OSRM project is still going strong!

Would you be so kind and make a docker release version of 5.26.0, too? Currently it's stuck at 5.25.0

(If anyone shows me how to do it, I could do the job next time, too.)

If I am reading https://github.com/Project-OSRM/osrm-backend-docker correctly, then the whole process is automated and I just need to be a little bit more patient:

The OSRM demo server periodically uses build-master.sh to grab the latest master code and publish a new image to DockerHub.

But does this work for the releases (in docker: tags), too? https://hub.docker.com/r/osrm/osrm-backend/tags

[danpat]

@cyril23 Docker has changed their terms of service - honestly, I haven't read the details, but it seems that our automated builds aren't working.

Building docker images locally should be trivial though - just docker build . in the root of the code repo and you'll shortly have a local image to use.

[hoerup]

I'm thinking that something is missing in the migration from travis to github actions ??

[mjjbell]

IIUC, there's a Docker Hub build hook that's no longer triggering. Docker Hub documentation says this now requires a subscription.

We can apply to re-enable the build pipeline by joining the open source program.

[hoerup]

I assume that the container image build could be done as a GH action then pushed to either docker hub or ghcr.io

[nilsnolde]

probably worth migrating to github packages instead of docker (until gh packages pulls the same move;)). we did the same for our images, from what I heard the "open source program" is pretty useless, @jcoupey tried it once IIRC

[nilsnolde]

if you don't have a crack at it I might @hoerup . for reference, I use this to build and push valhalla: https://github.com/gis-ops/docker-valhalla/blob/master/.github/workflows/publish.yml. I omitted dockerhub for now, but could/should be done as well, if only for convenience reason (docker pull ghcr.io/... is somewhat more work on the command line;))

[jcoupey]

@jcoupey tried it once IIRC

That was the Travis "open-source program", not Docker, but yeah it's been a rather frustrating experience. This kind of program seems to be just in place for marketing purposes ("look, we still support open-source") but if you dive into the details there are usually enough restrictions so that most projects don't fit in. I would not waste too much time on this kind of "program" if there's a viable alternative.

[cyril23]

@cyril23 Docker has changed their terms of service - honestly, I haven't read the details, but it seems that our automated builds aren't working.

Building docker images locally should be trivial though - just docker build . in the root of the code repo and you'll shortly have a local image to use.

If I try it in the root of the https://github.com/Project-OSRM/osrm-backend.git repo, I get the error

freeuser@freeuser-VirtualBox:~/osrm-backend$ docker build .
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /home/freeuser/osrm-backend/Dockerfile: no such file or directory
freeuser@freeuser-VirtualBox:~/osrm-backend$ 

So I did a cd docker first and then tried again, as described here:

# fresh Ubuntu 18 LTS install, all updates
# once neccessary
sudo apt-get install git
git clone https://github.com/Project-OSRM/osrm-backend.git
cd osrm-backend

# for updating to a specific version
git checkout v5.26.0
cd docker
sudo docker build .

But got an error

Building OSRM 
fatal: Not a git repository (or any of the parent directories): .git
Building OSRM gitsha 
Building Release with ENABLE_ASSERTIONS=Off BUILD_TOOLS=Off
CMake Error: The source directory "/src" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
The command '/bin/sh -c NPROC=${BUILD_CONCURRENCY:-$(grep -c ^processor /proc/cpuinfo 2>/dev/null || 1)} &&     apt-get update &&     apt-get -y --no-install-recommends install cmake make git gcc g++ libbz2-dev libxml2-dev     libzip-dev libboost1.62-all-dev lua5.2 liblua5.2-dev libtbb-dev -o APT::Install-Suggests=0 -o APT::Install-Recommends=0 &&     echo "Building OSRM ${DOCKER_TAG}" &&     git show --format="%H" | head -n1 > /opt/OSRM_GITSHA &&     echo "Building OSRM gitsha $(cat /opt/OSRM_GITSHA)" &&     mkdir -p build &&     cd build &&     BUILD_TYPE="Release" &&     ENABLE_ASSERTIONS="Off" &&     BUILD_TOOLS="Off" &&     case ${DOCKER_TAG} in *"-debug"*) BUILD_TYPE="Debug";; esac &&     case ${DOCKER_TAG} in *"-assertions"*) BUILD_TYPE="RelWithDebInfo" && ENABLE_ASSERTIONS="On" && BUILD_TOOLS="On";; esac &&     echo "Building ${BUILD_TYPE} with ENABLE_ASSERTIONS=${ENABLE_ASSERTIONS} BUILD_TOOLS=${BUILD_TOOLS}" &&     cmake .. -DCMAKE_BUILD_TYPE=${BUILD_TYPE} -DENABLE_ASSERTIONS=${ENABLE_ASSERTIONS} -DBUILD_TOOLS=${BUILD_TOOLS} -DENABLE_LTO=On &&     make -j${NPROC} install &&     cd ../profiles &&     cp -r * /opt &&     strip /usr/local/bin/* &&     rm -rf /src /usr/local/lib/libosrm*' returned a non-zero code: 1

Full output: docker_build_problem.txt

What am I doing wrong here?

[hoerup]

You need something like
docker build . -f docker/Dockerfile

[cyril23]

docker build . -f docker/Dockerfile

So I tried:

# fresh Ubuntu 18 LTS, all updates, docker and git already installed

# only once
git clone https://github.com/Project-OSRM/osrm-backend.git
cd osrm-backend

# for updating to a specific version
git checkout tags/v5.26.0
sudo docker build . -f docker/Dockerfile

It ran longer, but resulted in various errors, see docker_build_errors.txt

[hoerup]

Looks like a OOM kill of a gcc process. Check memory consumption

[cyril23]

Looks like a OOM kill of a gcc process. Check memory consumption

Thanks, yes, that was the case. Now the build was completed successfully. log: docker_build_ok.txt

So how do I use my locally built docker image instead of automatically downloading and using the dockerhub one? instead of osrm/osrm-backend:v5.26.0 can I somehow use the path to my locally built image?

# fresh Ubuntu 18 LTS, all updates, docker and git already installed

# only once
git clone https://github.com/Project-OSRM/osrm-backend.git
cd osrm-backend

# for updating to a specific version
git checkout tags/v5.26.0
sudo docker build . -f docker/Dockerfile

# download current osm data
cd ~
mkdir osm && cd osm
wget http://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf

# trying to run the osrm extract from locally built docker image
sudo docker run -t -v "${PWD}:/data" osrm/osrm-backend:v5.26.0 osrm-extract -p /opt/car.lua /data/berlin-latest.osm.pbf

# output is an error unfortunately
Unable to find image 'osrm/osrm-backend:v5.26.0' locally
docker: Error response from daemon: manifest for osrm/osrm-backend:v5.26.0 not found: manifest unknown: manifest unknown.
See 'docker run --help'.

[danpat]

One of the last outputs from the docker build command would be something like:

 => => writing image sha256:be233d8d32bb38ab109112dfc16a678f6922f416c36d089ec775ea0c906c3185                                                                                         0.0s

That sha256 value is the id of the image. You can do:

docker run -t -i be233d8d32bb38ab109112dfc16a678f6922f416c36d089ec775ea0c906c3185

to run it, instead of using osrm/osrm-backend:v5.26.0.

Alternatively, you can supply a "tag" to the docker build command, like:

docker build . -f docker/Dockerfile -t my_osrm_build

then later:

docker run -t -v "${PWD}:/data" my_osrm_build osrm-extract .....

[cyril23]

Alternatively, you can supply a "tag" to the docker build command, like:

That sounds like the solution I need. I put it all together:

# fresh Ubuntu 18 LTS, all updates, docker and git already installed

# only once
git clone https://github.com/Project-OSRM/osrm-backend.git
cd osrm-backend

# for updating to a specific version
git checkout tags/v5.26.0
# tag the docker build to make it easier to access later on
# info: uses up to 8.4 GiB memory on a Ubuntu 18 LTS desktop .. 12 GB memory for the VM was more than enough
sudo docker build . -f docker/Dockerfile -t osrmv5.26.0

# download current osm data
cd ~
mkdir osm && cd osm
wget http://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf

# using the locally built docker image which was tagged "osrmv5.26.0":
# preprocessing, here: CH pipeline
sudo docker run -t -v "${PWD}:/data" osrmv5.26.0 osrm-extract -p /opt/car.lua /data/berlin-latest.osm.pbf
sudo docker run -t -v "${PWD}:/data" osrmv5.26.0 osrm-contract /data/berlin-latest.osm.pbf
# start osrm
sudo docker run -p 5000:5000 -t -v "${PWD}:/data" osrmv5.26.0 osrm-routed --algorithm ch /data/berlin-latest.osm.pbf

This worked perfectly for me. Here is an example request being served:

[info] starting up engines, v5.26.0
[info] Threads: 8
[info] IP address: 0.0.0.0
[info] IP port: 5000
[info] http 1.1 compression handled by zlib version 1.2.8
[info] Listening on: 0.0.0.0:5000
[info] running and waiting for requests
[info] 28-09-2021 09:11:18 1.48104ms 172.17.0.1 http://127.0.0.1:9966/ Mozilla/5.0(X11;Ubuntu;Linuxx86_64;rv:92.0)Gecko/20100101Firefox/92.0 200 /route/v1/driving/13.399023642365556,52.52840475;13.411228335288781,52.521025949999995?overview=false&alternatives=true&steps=true&hints=;

Thanks a lot for the information, to all of you!

By the way, indeed the Github Actions are unlimited and free for public (open source) projects: https://de.github.com/features/actions#pricing and https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#about-billing-for-github-actions and https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#about-billing-for-github-actions

GitHub Actions usage is free for both public repositories and self-hosted runners.

[hoerup]

Next tag will be automatically build by github actions. See #6138

[cyril23]

Next tag will be automatically build by github actions. See #6138

with the documentation to manually build it here #6134 (comment) and the next tag being build automatically by github actions, this issue can be closed now probably