Cannot backup a docker image through /proc/<pid>/root/

[unilynx]

I like to backup docker containers with their full filesystem as they appear to the container itself so I don't miss any changes made to the overlayfs filesystems if we incorrectly set up the volume mounts.

On linux, every docker container's filesystem can be accessed through /proc/<pid>/root/, and I can point rsync at this to create a full backup. This has the added advantage that I don't have to add rsync to all container images I use.

I'd like to do this with borg too, but it gets confused when running inside this filesystem. But:

# Session 1
docker run --name test-borg --rm -ti alpine:latest sleep 365d

# Session 2
DOCKERPID=$(docker inspect -f '{{.State.Pid}}' test-borg)
mkdir /tmp/borgrepo 
borg init -e none /tmp/borgrepo
cd /proc/$DOCKERPID/root/
borg create /tmp/borgrepo::testarchive .

fails with

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4253, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4185, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 152, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 546, in do_create
    log_json=args.log_json)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 292, in __init__
    self.cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 11, in <module>
    load_entry_point('borgbackup==1.1.5', 'console_scripts', 'borg')()
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4275, in main
    tb = '%s\n%s' % (traceback.format_exc(), sysinfo())
  File "/usr/lib/python3/dist-packages/borg/helpers.py", line 1631, in sysinfo
    info.append('PID: %d  CWD: %s' % (os.getpid(), os.getcwd()))
FileNotFoundError: [Errno 2] No such file or directory
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 67, in apport_excepthook
    binary = os.path.realpath(os.path.join(os.getcwd(), sys.argv[0]))
FileNotFoundError: [Errno 2] No such file or directory

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4253, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4185, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 152, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 546, in do_create
    log_json=args.log_json)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 292, in __init__
    self.cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/borg", line 11, in <module>
    load_entry_point('borgbackup==1.1.5', 'console_scripts', 'borg')()
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4275, in main
    tb = '%s\n%s' % (traceback.format_exc(), sysinfo())
  File "/usr/lib/python3/dist-packages/borg/helpers.py", line 1631, in sysinfo
    info.append('PID: %d  CWD: %s' % (os.getpid(), os.getcwd()))
FileNotFoundError: [Errno 2] No such file or directory

I can't point borg to the absolute path /proc/$DOCKERPID/root/ either - it will just create an empty archive then

I currently just backup the volumes, but it would be nice if this were possible, or if there is a workaround that permits this.

[ThomasWaldmann]

There seems to be a problem below borg.

os.getcwd() is just trying to get the current working directory and that is expected to work, but it fails for you.

[robaato]

I was curious as this is a great idea, so I checked. Definitely not borg's issue.

This fails too with ENOENT:
#include <stdio.h>
#include <unistd.h>
#include <errno.h>

int main() {
char buffer[4096];
char *c=getcwd(buffer,4096);
if (!c) printf("Error %d\n",errno);
else printf("%s\n",buffer);
return 0;
}

[unilynx]

I was curious as this is a great idea, so I checked. Definitely not borg's issue.

This fails too with ENOENT:

Same here, but pwd which I would expect to simply be a wrapper around getcwd properly reports /proc/PID/root ...
and running strace over your code doesn't show any system call returning -ENOENT.

It looks like glibc is doing this (yep, https://github.com/lattera/glibc/blob/master/sysdeps/posix/getcwd.c#L395 ) but pwd somehow compensates.

But I can imagine fixing os.getcwd() to do whatever pwd does instead of relying on the glibc implementation would be outside Borg's scope. Thanks, at least the issue has been pinpointed, that might help finding a workaround..

EDIT: It's bash. /bin/pwd fails too, pwd apparently remembers the last cd command to figure out the path.

[robaato]

Not exactly (https://github.com/coreutils/coreutils/blob/master/src/pwd.c):

root@****:/proc/3804/root# /bin/pwd -L
/proc/3804/root
root@****:/proc/3804/root# /bin/pwd -P
/bin/pwd: couldn't find directory entry in ‘..’ with matching i-node

This is because inode for .. is special here (matches .):

root@****:/proc/3804/root# ls -ali
total 96
5242981 drwxr-xr-x   1 231072 231072 4096 mar 26  2020 .
5242981 drwxr-xr-x   1 231072 231072 4096 mar 26  2020 ..

EDIT: pwd -L reads $PWD if exists, that's the magic

[unilynx]

I've done some more testing - by adding a parameter to explicitly set the working directory I get usable archives out of borg.

With these changes to replace getcwd() with a --working-directory parameter if specified: 2851a84...unilynx:workingdirectory

I can do:

cd /
borg create --list --exclude=proc --exclude=sys --working-directory /proc/10560/root/ /tmp/borgrepo::testarchive .

And I have an extractable archive. A bit comparable to tar's -C option

This still leaves some corner cases open (I haven't checked whether the working-directory path is absolute and probably should, and borg still doesn't like being executed in /proc/10560/root/ unless --working-directory is set) but would a change like this be an option for borg upstream?

[ThomasWaldmann]

Not sure this should be worked around in borg. Isn't this rather a bug in docker (or in your usage of docker)?

[ThomasWaldmann]

OTOH, such an option could be useful to get rid of leading path segments, like replacing:

cd /home
borg create repo::archive someuser otheruser

[ThomasWaldmann]

From tar, we could use the same option names maybe (check if -C is still available)?

  -C, --directory=DIR        change to directory DIR

[ThomasWaldmann]

Maybe some other people using borg with docker can comment here whether they also see this issue or how they have it working.

[alexconstsh]

OTOH, such an option could be useful to get rid of leading path segments, like replacing:

cd /home
borg create repo::archive someuser otheruser

Is there an issue for this? I think I saw it some time, but I can't find it now. EDIT: found it: #1783
I would love to see this option in future releases as I prefer all my settings to be in a single configuration file (I use borgmatic which reads yaml configs and passes options from them to borg). Right now my backup script looks like this and I don't really like it:

( cd /mnt/data          && borgmatic -c ~/data/backup-info/configs/data.yaml     -C --progress )
( cd "$HOME"            && borgmatic -c ~/data/backup-info/configs/home.yaml     -C --progress )
( cd /mnt/data-ext      && borgmatic -c ~/data/backup-info/configs/data-ext.yaml -C --progress )

[alexconstsh]

From tar, we could use the same option names maybe (check if -C is still available)?

  -C, --directory=DIR        change to directory DIR

-C is a synonym for --compression

[ThomasWaldmann]

Closing this in favour of #1783. We have a link to the above mentioned code there also.