Official Docker image?

[DarrienG]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Issue

System information. For client/server mode post info for both machines.

N/A

Your borg version (borg -V).

N/A

Operating system (distribution) and version.

N/A

Hardware / network configuration, and filesystems used.

N/A

There are a couple of unofficial Docker images for borg, but there's interest in an official Docker image. Especially after the runc vulnerability that was reported the other day, an officially supported image would inspire more trust in it.

I know I could always build one of my own, but the world is moving towards containerized services, and it would likely drive up usage.

I don't mind making a Docker related PR if desired :)

[ThomasWaldmann]

I personally don't use docker.

For which of the supported platforms would you like to add docker images?
What exactly would such an image offer?

[DarrienG]

Docker runs on most platforms, so any OS is fine.

Most "single binary" images tend to be based on alpine linux, but Debian, Ubuntu, Fedora, or the like is a fine base image, I don't think anyone would care much.

We're thinking about using Borg at big company XYZ and we have a huge Kubernetes cluster. We would like to use Borg for automating backups and all that jazz.

The image would be used as a sidecar container to run alongside another container in our cluster, and back up any contents that are important.

Containers are supposed to be single purpose, single process, so installing a cron and the borg binary alongside say our Jenkins container isn't much of an option.

As I said before, we can make our own, but an image with official support that is updated as borg is updated, and changed as borg has different dependencies would be great.

[pierreozoux]

I have 2 comments here :) I came to open the same issue ;)

Yes, an official image would be amazing, I already created this one and I'd be happy to PR upstream if it is something you want to have.
(It does a bit more than just borg, but well.. we can figure that out during the PR)

We also have 2 docker-compose that goes with that:
https://github.com/libresh/compose-borg-server
https://github.com/libresh/compose-borg-client

@DarrienG for kubernetes you are actually looking at https://appscode.com/products/stash/ ;) I know it is based on restic, but it is so cool ;) (And I'd love to have that with borg, restic doesn't have compression)

[ThomasWaldmann]

Considering the linux fat binaries are built on debian, debian 64bit would be a good base. In master branch we use stretch currently. Have a look at the Vagrantfile "stretch64" machine.

If this shall have a chance to become part of release process, it must be fully automated.

Also, there is a unfinished ticket of specifying a standard client/server setup, so guess the docker images could be based on that. #1553

[DarrienG]

At work we use a modified version of docker-library, which is likely overkill.

It looks like the docker foundation has their own GitHub integration which will probably do this in just a few clicks.

Set it up to only build when a new tag is added, and you should be good to go.

[ThomasWaldmann]

I'ld like to build binaries on machines / OS I control, not in the "cloud".

[DarrienG]

If you're ok with Travis (which isn't on your machine, but it does look like your build system) you can build through that on a tag release: https://docs.travis-ci.com/user/docker/ and https://docs.travis-ci.com/user/deployment/releases/

Otherwise you'll want to add a webhook that fires a POST or whatever to a server you own.

Building the image will look like this:

# With an authenticated docker client, see https://docs.docker.com/v17.12/docker-cloud/builds/push-images/
LATEST="borg_collective/borg-backup:latest"
DOCKER_IMAGE="borg_collective/borg-backup:$GIT_TAG_FROM_WEBHOOK"

git clone https://github.com/borgbackup/borg
cd borgbackup
git checkout $GIT_TAG

docker build -t "$DOCKER_IMAGE" .
docker tag "$DOCKER_IMAGE" "$LATEST"

docker push "$DOCKER_IMAGE"
docker push "$LATEST"

[DarrienG]

FWIW - I'm not entirely certain how the release process works for borgbackup. If you on the other hand make tags on your local machine and then push the tag up, you could make a hook on your git client itself which will in turn run the above commands quietly in the background.

https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks

[pierreozoux]

Where is the pipeline to push to pip?
The easiest is really to let https://hub.docker.com/ do the build and host the docker image.

Do you have some infrastructure to host a docker registry and run a pipeline?

I do plan to build one here if you need. (there is no runner nor registry yet, but I'll work on that on the coming weeks).

[ThomasWaldmann]

The stuff needs to be build in the vagrant VM, see Vagrantfile "stretch64" (as I already said above).

No travis, no github, no AWS, ...

I'ld accept a PR that is tested and modifies the Vagrantfile (master branch) so it builds 2(?) docker images.

But as I don't use / don't need docker, I won't implement this myself.

[pierreozoux]

Ok, I understand, I took a look at the vagrant, it will be easy.
For the 2 docker image, do you mean:

• a simple one with just Borg bin
• a complexe one with my client/server code to implement create specs for backup service? #1553

This would be cool indeed and cover most of what people need :)

I'll modify my PR later with that.

@DarrienG you'll have to help me maintainer that, if we share the work, it should be easier :)

@ThomasWaldmann I have more questions:

How do you push to pip? After reading the vagrant file, it was not clear to me.
In the Dockerfile, should I use the local bin, or the pip version (like in the PR) (in the second case, I have to make sure the build happens after the pip push).
Is it ok for you to host the docker image on hub.docker.com or would you prefer your own registry?
In the complexe Docker image, is it ok to ship Prometheus metrics like I currently do, or should I separate this logic? (I think separating would make more sense, but well, up to you :) )

Thanks for your time!

[ThomasWaldmann]

I thought you wanted to make one for the client and one for the server. But just do what makes sense. :)

The source dist (sdist, then uploaded to pypi after signing) currently comes from my development machine, the fat binaries are built and tested in the (self-installed, if possible) vagrant VMs you see in the Vagrantfile and fetched from there to dev machine via vagrant scp. I'ld fetch the docker image from there in the same way.

Inside the docker image, I guess working from source (via the sdist, python setup.py sdist, no need to wait for pypi or downloading it from there) is better than using the fat binary (which is a rather huge wrapper, but that is not needed as that is solved by docker, AFAIK).

I don't know any docker stuff, but an obvious requirement is that users can verify authenticity of any downloaded file via a gpg signature before running it as root and letting it access all their files. Another requirement is not "polluting" the vagrant machines with low-trust stuff (e.g. binary downloads from 3rd party sources, not debian).

Prometheus stuff sounds rather specific, so I guess without makes more sense for broad use.

[DarrienG]

@pierreozoux Happy to help get this going and keep it up to snuff. We're using a company image in some alpha services right now and would definitely like official support. Since you already have a Dockerfile and all that, feel free to add me to the review you make.

I personally don't think Prometheus metrics should be P1/built in.

[DannyBen]

I am also very much in favor of an official image with the borg binary installed.

Seeing the discussion veered a little off course in my opinion, I think that:

1. This should be something simple. No need for any of the dependencies that were mentioned above.
2. Unless there is a complication with borg on Alpine, this most definitely should be an Alpine based image.
3. Testing of the build should be minimal - as you are not testing borg itself, but just the fact that it was built properly and executes (even borg --version could be a sufficient test).
4. There should be no need for a separate server and client images - in the same way there is only one binary.
5. Instead (of 4), the README of this Docker image should simply include instructions (docker run ...) on how to start a server and how to start a client - including all volumes needed etc.
6. I believe it should be acceptable (and even desired) to use a completely separate repository for just this purpose.
7. Tagging of such image on DockerHub should include the borg version.

I also might be able to assist in that area if it is desired. I am a heavy docker user, but unfortunately, a borg noob....

EDIT

Is there any reason not to simply rely on the official alpine package?

FROM alpine
RUN apk --no-cache add borgbackup
ENTRYPOINT ["borg"]
CMD []

Then, all that is left to do is create the README, perhaps add a couple more directives, configure automated build and be done with it. No?

[ThomasWaldmann]

See #4538 with some alpine msgpack issue. Guess until that is solved and also borgbackup updated to latest stable, this is no good base.

[DannyBen]

Personally, I think we can proceed and make preparations for an official docker image, based on the assumption that the alpine package will be fixed soon.

For reference, here is the issue tracker for the msgpack warning:
https://bugs.alpinelinux.org/issues/10401

I intend to prepare such an image for my own use, and to follow the principles I mentioned earlier. If anyone might be interested in putting this under borg's official umbrella, I am happy to do so and contribute what I can.

[DannyBen]

If anyone is interested:

1. I have created my take on a borg client image.
2. I also have a version of the server, but I have decided to split these guys for now, since the server requires sshd, and slightly different examples - so I figured I should probably keep these concerns separate.
3. I hope I mounted all of the folders needed for proper operation of borg from the client side (e.g. cache, config, private key)

So, I am putting this here for a couple of reasons:

1. If anyone wants to use it / contribute to it - by all means.
2. If anyone from the borg team wants to use it as a base for an official image - also by all means.

In any case, I hope an official image comes out, and I hope it is not a lot more complicated than this Dockerfile.

[funkyfuture]

I hope it is not a lot more complicated than …

i agree, the use-cases are very different, so a base image should really just provide the software and its dependencies, but not even define an ENTRYPOINT or CMD.

[euh2]

I very much like @DannyBen take on this.

For quite some time I used the following line in my Dockerfile to have the latest Borg version (should be extended by some tests):

RUN curl --silent "https://api.github.com/repos/borgbackup/borg/releases/latest" | jq -r '.assets[] | select(.name | endswith("linux64")).browser_download_url' | wget -nv --show-progress --progress=bar:force:noscroll -i - \
    && mv borg* /usr/local/bin/borg && chown root:root /usr/local/bin/borg && chmod 755 /usr/local/bin/borg \
    && /usr/local/bin/borg -V

[staticvoidmaine]

Not sure if there's been any more movement on this, but FWIW I also agree with @DannyBen and @funkyfuture in that the base image should be generic and not preclude some people from using it due to it being opinionated.

Additional functionality and/or opinions on its use should be specified in people's own images derived from the official base image.

[alexxxnf]

Hi guys,

This issue has not been updated for a quite a while. What's the current status?

[ThomasWaldmann]

There's PR #4372 open since quite a while.

While I don't personally use docker and thus have no clue about it, I feel it is too complex and should be way simpler (IIRC there were also some comments along these lines).

There is also the maintenance issue: if we merge some "official Docker support", we need to maintain it. I obviously won't do it and so there is some fear that the more complex it is, the less maintainable it will be.

So, blockers: a simple thing and maintainers for it.

[EthraZa]

Why are people making it complex?
You just need to choose a official base image (ie. debian:stable-slim) and add something as simple as one line to the Dockerfile to add the borg binary and that's it, everything else is aleready there.

Example:

FROM debian:stable-slim
ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get -y update && apt-get -y install --no-install-recommends \
        fuse \
        openssh-server \
        wget && \
        apt-get -y clean && \
        apt-get -y autoclean && \
        apt-get -y autoremove && \
        rm -rf /var/lib/apt/lists/*

RUN groupadd -g 1000 borg && \
        groupadd -g 500 core && \
        groupadd -g 233 docker

RUN useradd -m -u 1000 -s /bin/bash -c "Stevedore" -g borg borg -G adm,backup,core,disk,docker,mail,nogroup,operator,root,staff,sudo,tape,tty,users,www-data

RUN wget https://github.com/borgbackup/borg/releases/latest/download/borg-linux64 -O /usr/bin/borg && \
        chmod 755 /usr/bin/borg

ENTRYPOINT /usr/bin/borg

Did not tested this Dockerfile, but if it's not this, it's very close to this.
Borg already use env variables that can be passed by environment flags in Docker, but, anyways, it's just borg usage, not exactly Docker specific.

[EthraZa]

Done, Just made my own Backup oriented Docker image:

https://github.com/EthraZa/BackupOps

[yash-fn]

Very nice thank you!

[hatl]

Maybe this can serve as the "official" image?
https://github.com/borgmatic-collective/docker-borgmatic

[stuaxo]

I had a look at the Alpine related issues above, and they have all been fixed, so having an Alpine based docker image should not be an issue any more.

[azlux]

I also have my own since there are no official ones with borg and tool for database backup. (https://github.com/azlux/borgbackup-docker)
There are many way to create a image. Do we want cron, tool to export, borgmatic, how many env variables, DB backup before borg...
Maybe having 2-3 images for differences used. Or just one with the borgbackup tool only.

[RonnyPfannschmidt]

A single image whose entrypoint is a dumb init wrapped call to borg

Stuff like cron in a container is a no go

[enpaul]

Hi all, I'm new to Borg but have a lot of experience with Docker and am looking at bringing Borg into my container infrastructure. If people are still looking for a solution here, I'd be happy to look at resurrecting @pierreozoux 's PR (#4372) for an official image. I can also commit to maintaining the container in the future.

@ThomasWaldmann are there docs somewhere on the release process I could read so I can make sure the docker implementation can integrate with your existing process?

[ThomasWaldmann]

There is a "how to release" section in the development section of the docs.

And as I don't use docker, docker is not part of that process.

[MartinX3]

@enpaul I use borgmatic as client and borg-backup as server just fine in rootless podman containers in arch linux.

[AnotherStranger]

Hello there,
I just created a docker container to act as a borgbackup server:
https://github.com/AnotherStranger/docker-borg-backup
I forked it from https://github.com/tgbyte/docker-borg-backup.

Maybe this is useful for some looking here.
Feedback is appreciated :-)

[MartinX3]

@AnotherStranger is there a changelog to see what you optimized without analyzing every commit?
And will you PR your changes into the upstream repo? :)

[AnotherStranger]

@MartinX3

The main differences are:

1. I switched away from the custom baseimage to the fedora docker image. However, I am planning to change it to alpine.
2. The container adds borg serve with path restrictions to all authorized keys. Further options can be specified via an env variable.
3. Other changes are build-related. I added GitHub actions to build and publish the image automatically. Releases and versioning is done via semantic-release. I added pre-commit hooks to ensure that the Dockerfile passes linting using hadolint.
4. The image is built for all archs which are supported by the fedora baseimage.

I doubt that upstreaming is something tgbyte would desire, as I changed the baseimage. Maybe the borg serve with path restriction is worth upstreaming.

[MartinX3]

@AnotherStranger

That sounds nice.
Also Alpine / Arch Linux for a smaller container image is nice.

Also I hope the fork won't get abandoned and receive an upgrade guide for us end users. :)

[AnotherStranger]

@MartinX3
I don't intend to abandon it anytime soon, as I not only use it privately, but also at work.
I also intend to write some easy to understand docs. I can ping you if you want (and have time) to review them.

[adeverteuil]

Hi everyone,
Thanks @AnotherStranger, I tested your docker-borg-backup image and it works well!

I went with another route though. I tried deploying AnotherStranger/docker-borg-backup on TrueNAS SCALE, but the Environment variables only accept values up to 1024 characters :( so I couldn't pass a value with multiple SSH keys.

I also wanted more flexibility in the authorized_keys, such as a different path for each user/client machine.

My approach is to use a generic OpenSSH image linuxserver/openssh-server instead of looking for a Borg-specific image. I found out about LinuxServer.io, which is a collection of community-maintained images. They are all built with s6-overlay which makes them very easy to customize and extend. They also have design standards, such as storing all persistent configuration in /config, which also makes it really easy to manage.

Here is my full solution:
https://alexandre.deverteuil.net/post/borg-backup-server-on-truenas-scale/
My post is about hosting on TrueNAS SCALE, but it can be used with any Docker environment.

[eric-j-ason]

@adeverteuil

Here is my full solution: https://alexandre.deverteuil.net/post/borg-backup-server-on-truenas-scale/ My post is about hosting on TrueNAS SCALE, but it can be used with any Docker environment.

Do you know if this works also on TrueNAS SCALE 23.10?

Any opinion on whether a similar approach be suitable if instead one wanted to run Borg as a client on TrueNAS SCALE?

[AnotherStranger]

Hi everyone, Thanks @AnotherStranger, I tested your docker-borg-backup image and it works well!

I went with another route though. I tried deploying AnotherStranger/docker-borg-backup on TrueNAS SCALE, but the Environment variables only accept values up to 1024 characters :( so I couldn't pass a value with multiple SSH keys.

I also wanted more flexibility in the authorized_keys, such as a different path for each user/client machine.

My approach is to use a generic OpenSSH image linuxserver/openssh-server instead of looking for a Borg-specific image. I found out about LinuxServer.io, which is a collection of community-maintained images. They are all built with s6-overlay which makes them very easy to customize and extend. They also have design standards, such as storing all persistent configuration in /config, which also makes it really easy to manage.

Here is my full solution: https://alexandre.deverteuil.net/post/borg-backup-server-on-truenas-scale/ My post is about hosting on TrueNAS SCALE, but it can be used with any Docker environment.

Hey @adeverteuil,
Sorry for not getting back to you earlier!
I read your blog entry, and it was a very nice read!

Regarding the Environment variable size: The container now supports mounting your authorized_keys file as an alternative to the Environment variable. So for anyone who is still looking for this feature, you can give my container image a try here: https://github.com/AnotherStranger/docker-borg-backup/issues.

Regarding TrueNas: There is a chart to install the borg-server image to your TrueNas Scale installation available here. However, I don't have a TrueNas server, so I can't say anything about this chart.

If anyone here has any Feedback or feature requests for the server image, feel free to open an Issue in the docker-borg-backup repo.

[rugk]

BTW in case one does not know it yet, https://borgwarehouse.com/ looks like a very promising candidate, if you want a little more aka a (web) GUI to manage the repositories of your backup (server).

I have not practically tested it yet, but it looks very great from what I can see and read there!

[ToeiRei]

I have not practically tested it yet, but it looks very great from what I can see and read there!

I did try on TrueNAS Scale using the docker-compose.yaml
Let's just say, not a big fan of it anymore the more I look into it.