borg 1.4

[ThomasWaldmann]

borg 1.4 as upcoming near-future stable release series

Considering borg2 will take quite a while still, borg 1.4 will contain some more conservative changes.

Your help is needed!

But some of these changes will be still big enough to require some alpha/beta/rc testing before they can be considered stable and production-ready again.

We have automated tests, but usually some issues are only found when practically using borg. Of course, you would run additional backups with the pre-release code, not your important production backups (use 1.2.x for that).

I will add stuff to the 1.4.0* milestones there: https://github.com/borgbackup/borg/milestones

Code is in the 1.4-maint branch. https://github.com/borgbackup/borg/tree/1.4-maint

Soon there will be some alpha/beta releases on pypi.org: https://pypi.org/project/borgbackup/#history

Why not 1.3.x!?

The reason why it is 1.4 and not 1.3 is because we had some 1.3 development versions (kind of very early versions of what will be borg2 later), so I skipped that number to avoid confusion.

Stuff that should be tested

• see change log, everything related to the changes listed there
• BORG_EXIT_CODES=modern borg ... commands - they should give correct error / warning / success exit codes as documented in internals/frontends.
• BORG_EXIT_CODES=legacy borg ... commands - they should give correct error / warning / success exit codes as documented in internals/frontends. This is currently the default mode, even if BORG_EXIT_CODES is not set. Should behave mostly as borg 1.1/1.2.
• Internally, borg < 1.4 often used returning an exit code from a function call to signal an error. Now, in borg 1.4, many places raise an exception to signal an error. This could also mean semantic differences in how the code works, so it should be checked that nothing unexpected happens.
• Empty PATH arguments ("") are rejected. Usually, this should not happen in interactive use, but scripts and wrappers should be checked whether they accidentally give empty paths to borg. In <1.4, this was ignored, but lead to unexpected behaviour, in >= 1.4 this will terminate with an error.
• testing, packaging, installing it and related scripting - there were quite some changes, e.g. setup.py -> pyproject.toml.
• some requirements changed/increased: python, cython, msgpack - and also the OS and pyinstaller release we use to build binaries.

[ThomasWaldmann]

BORG_EXIT_CODES
When set to “modern”, the borg process will return more specific exit codes (rc).
Default is “legacy” and returns rc 2 for all errors, 1 for all warnings, 0 for success.

https://borgbackup.readthedocs.io/en/1.4.0a1/usage/general.html#return-codes
https://borgbackup.readthedocs.io/en/1.4.0a1/internals/frontends.html#message-ids

[witten]

I did some basic testing with BORG_EXIT_CODES=modern on Borg 1.4.0a1, and I can confirm that causing various warning conditions like changing a file while it's being backed up or giving Borg a non-existent source path causes the appropriate exit code (100 and 107 respectively). I've also managed to get different error codes like 13 (repository does not exist) and -9 (killed by signal 9). I assume that exit code -9 is the same as 137.

[ThomasWaldmann]

Hmm, does the shell use some unusual integer conversion? uint8 137 != int8 -9

[ThomasWaldmann]

https://docs.python.org/3/library/subprocess.html#subprocess.Popen.returncode

negative return codes (-N) for signal exits seem to be specific to Python's subprocess, the shell uses 128+N for signal N.

[witten]

Yeah, exactly. Here's a minimal repro assuming an existing repository called repo and a borg1.4.0a1 binary in the system path:

python -c "import subprocess; print('return code:', subprocess.call(['borg1.4.0a1', 'create', '--read-special', 'repo::archive', '/dev/zero']))" &
killall -9 borg1.4.0a1

This prints:

return code: -9

[Ravinou]

Hi! I can't necessarily test everything you ask, but I did take the time to test this version in BorgWarehouse, which I'm developing, and everything works. Thanks for your work 😉

[ThomasWaldmann]

New 1.4 feature: the slashdot hack: #8060

The slashdot hack in paths (recursion roots) is triggered by using /./: /this/gets/stripped/./this/gets/archived means to process that fs object, but strip the prefix on the left side of ./ from the archived items (in this case, this/gets/archived will be the path in the archived item).

Please review / comment.

[Panzer1119]

How about a system like Proxmox Backup Server uses it?

Define a path you want to backup and a name for it, e.g. user:/home/user etc:/etc etc2:/dir/subdir/etc.

Or even better, what about Docker style with a path mapping for in- and outside the archive: /dir/on/host:/path/inside/archive /etc:/etc /dir/subdir/etc:/etc2.

And then the /dir/subdir/etc would be backed up as /etc2?

[ThomasWaldmann]

I see what you mean. Problem is that about all bytes (chars) except 0x00 are valid in unix paths (with "/", "." and ".." having a special meaning), so ":" is a valid char in file names.

[rovo89]

I suggested this approach because it has been implemented in rsync for almost 20 years, refer to their man page:

It is also possible to limit the amount of path information that is sent as implied directories for each path you specify. With a modern rsync on the sending side (beginning with 2.6.7), you can insert a dot and a slash into the source path, like this:

rsync -avR /foo/./bar/baz.c remote:/tmp/

That would create /tmp/bar/baz.c on the remote machine. (Note that the dot must be followed by a slash, so "/foo/." would not be abbreviated.)

I would really like to have some way to backup multiple path without projecting my current file system structure into the archive, and this is a clever hack. It also integrates nicely with existing wrappers such as borgmatic. The chances that someone passes a path literally containing /./ should be very low, so I wouldn't worry about it too much (besides documenting it of course).

The name clash could happen with any implementation, whether it's this way, via parameter or some other mapping. Even more likely when removing path components from the middle, just think multiple backups. This has to be addressed of course. Maybe last one wins or an error, at least it must be ensured that the archive doesn't become unreadable.

The same issue might appear for .tar archives which can also contain multiple contents for the same filename. Haven't checked how that is handled by borg.

[rovo89]

I see what you mean. Problem is that about all bytes (chars) except 0x00 are valid in unix paths (with "/", "." and ".." having a special meaning), so ":" is a valid char in file names.

If you prefer a map like suggested, you could of course split at /./ and instead of stripping the prefix left of it, use anything to the left as source and everything to the right as target in the archive. Strip the source path, prepend the target path.

Should work as well, also I like that the current implementation behaves like the standard tool and that it's still a valid path to exactly what's being backed up. /dir/on/host/./path/inside/archive /etc/./etc /dir/subdir/etc/./etc2 could be processed, but is a but hard to read.

[ThomasWaldmann]

Guess it is better to continue discussion about this in #4685.

[ThomasWaldmann]

Need some feedback about #8094 - please read and give feedback there!