Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix: Dockerfile RUN cache #559

Merged
merged 4 commits into from
Apr 19, 2024
Merged

Bugfix: Dockerfile RUN cache #559

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a714cf5
Dockerfile: use private RUN cache to avoid conflicting builds
carlomion Apr 18, 2024
bf7f3a0
ci: use docker/metadata-action to set tags and labels
carlomion Apr 18, 2024
882bfc9
Dockerfile: avoid removing specific packages after compilation, inste…
carlomion Apr 18, 2024
101a702
Fix generation of custom date tag
carlomion Apr 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 27 additions & 12 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,25 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v4


# Compute the Docker tags and labels to apply to the built image
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
# List of Docker images to use as base name for tags
images: |
mediagis/nominatim

# Disable generation of latest tag
flavor: |
latest=false

# Add simple tag with nominatim version + tag with custom date format
tags: |
type=raw,value=${{ matrix.nominatim.version }}
type=raw,value=${{ matrix.nominatim.version }}-{{date 'YYYY-MM-DDTHH-mm'}}

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

Expand All @@ -274,14 +292,11 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Set container date tag
run: |
echo "DATE_TAG=$(date +%Y-%m-%dT%H-%M)" >> $GITHUB_ENV

- name: Build and push production docker image to Dockerhub
if: ${{ github.ref == 'refs/heads/master' && github.repository_owner == 'mediagis' }}
run: |-
docker buildx build --platform linux/amd64,linux/arm64 --push \
-t mediagis/nominatim:${{ matrix.nominatim.version }} \
-t mediagis/nominatim:${{ matrix.nominatim.version }}-${DATE_TAG} .
working-directory: ${{ matrix.nominatim.version }}
- name: Build and push
uses: docker/build-push-action@v5
with:
context: ${{ matrix.nominatim.version }}
platforms: linux/amd64,linux/arm64
push: ${{ github.ref == 'refs/heads/master' && github.repository_owner == 'mediagis' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
24 changes: 11 additions & 13 deletions 4.4/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,13 @@ ENV LANG=C.UTF-8

WORKDIR /app

RUN \
--mount=type=cache,target=/var/cache/apt \
# Inspired by https://github.com/reproducible-containers/buildkit-cache-dance?tab=readme-ov-file#apt-get-github-actions
RUN \
--mount=type=cache,target=/var/cache/apt,sharing=locked \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused why concurrent access was even possible. I was under the impression that these are two totally different machines/vms and don't share anything, let alone disk. Where is this cache being saved to?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As explained here https://docs.docker.com/reference/dockerfile/#run---mounttypecache by default the parameter RUN --cache sharing is shared, meaning that the arm64 and amd64 builds, running in parallel inside the same GitHub runner (here is a quick overview of the multi-platform build done by Docker) will conflict with each other.

--mount=type=cache,target=/var/lib/apt,sharing=locked \
# Keep downloaded APT packages in the docker build cache
rm -f /etc/apt/apt.conf.d/docker-clean && \
echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' >/etc/apt/apt.conf.d/keep-cache && \
# Do not start daemons after installation.
echo '#!/bin/sh\nexit 101' > /usr/sbin/policy-rc.d \
&& chmod +x /usr/sbin/policy-rc.d \
Expand All @@ -24,7 +29,6 @@ RUN \
-o APT::Install-Suggests="false" \
# Build tools from sources.
build-essential \
g++ \
cmake \
libpq-dev \
zlib1g-dev \
Expand Down Expand Up @@ -76,7 +80,7 @@ RUN true \
&& echo "listen_addresses='*'" >> /etc/postgresql/14/main/postgresql.conf

# Osmium install to run continuous updates.
RUN --mount=type=cache,target=/root/.cache/pip \
RUN --mount=type=cache,target=/root/.cache/pip,sharing=locked \
pip3 install osmium

# Nominatim install.
Expand All @@ -94,18 +98,13 @@ RUN true \

RUN true \
# Remove development and unused packages.
&& apt-get -y remove --purge \
cpp-9 \
gcc-9* \
g++ \
&& apt-get -y remove --purge --auto-remove \
build-essential \
cmake \
git \
make \
cmake* \
llvm-10* \
libc6-dev \
linux-libc-dev \
libclang-*-dev \
build-essential \
liblua*-dev \
postgresql-server-dev-14 \
nlohmann-json3-dev \
Expand All @@ -114,7 +113,6 @@ RUN true \
/tmp/* \
/var/tmp/* \
/app/src/.git \
/var/lib/apt/lists/* \
# Remove nominatim source and build directories
&& rm /app/*.tar.bz2 \
&& rm -rf /app/build \
Expand Down