please add something like --existing-ok to init command

[rpodgorny]

i think it would be pretty nice (and quite useful for my case) to have an option to make borg not fail with exception and traceback on 'borg init' when the repository already exists.

that way, i could set my backup script (which basically does init+create+prune) to exit on failure (set -e).

[ThomasWaldmann]

considering you need init only once (and it even needs to be interactive when using encryption), what's your use case precisely?

[anarcat]

i understand the use case: just have a wrapper that creates the repo if missing, and not fail otherwise. it could check if the directory exist, but that's an extra step... borg could just avoid failing...

[ThomasWaldmann]

borg init is not automatic for every usecase. If you enable encryption, it will ask you for a passphrase.
there might be also a security question the first time one accesses a previously unknown repo.

Also, wouldn't you rather want to know if a repo that should be at some specific target location suddenly vanishes and needs to get recreated than just silently recreating it?

[rpodgorny]

fair questions! (and i'm currently reworking my script) but still, borg - as a backup solution - should be as non-interactive friendly as possible (imho) to be made run easily from cron etc...

[anarcat]

@rpodgorny agreed.

[ThomasWaldmann]

I still think init should be manual (as far as borg is concerned).

For the same reasons why you maybe do not want to automate fdisk/mkfs/labelling backup disk preparation steps.

[olivier-monaco]

Hi,

I'm updating an old issue with a wider request.

I'm working to a wrapper script as others and the interactive nature of borg can be a problem. What @rpodgorny speak about here is just one use case. Environment variables are a good "workaround" but not a "standard" way of think, even when using manually. Think about many command line tool which use switch, good example is "rm -f".

My "wrapper script" run borg remotely on many servers from one central server, some sort of scheduler. I rely on SSH and it's not so easy to pass environment variable (I need to add a helper script on all servers). Having command line switch would be easer ;).

But, my main problem is the interactive stuff. I would like borg to use the default answer when the environment variable is not set. Instead, it what for answer on stdin and this can block my script (workaround in progress). It would be very usefull to have some --non-interactive option at least.

[David-5-1]

Hi,

I was looking for a way to ensure some borg repository is initialized and ended up here. My goal is to have one central backup server with one borg repository for each server and manage everything with ansible (i.e. have both server and backup configuration automated).

The least dirty workaround I can think of would be to use borg check to check if the repository is created but this would likely end up in ansible runs taking up a long time :/ and the only realistic solution I could think of is to check that borg init fails with return code 2 and a message like "Repository .* already exists." in stderr…

Are there good reasons not to do what I try to do ? (I know having multiple repositories will use more disk space.)

[mlbarrow]

Unless I don't understand your question, couldn't you just look at the top-level of the directory for these contents: README config data/ hints.1 index.1 integrity.1 *michael at barrow dot me* +1.541.600.2027 "Do not anticipate trouble, or worry about what may never happen. Keep in the sunlight." -- B. Franklin

…

On Mon, Apr 2, 2018 at 1:08 PM, David-5-1 ***@***.***> wrote: Hi, I was looking for a way to ensure some borg repository is initialized and ended up here. My goal is to have one central backup server with one borg repository for each server and manage everything with ansible (i.e. have both server and backup configuration automated). The least dirty workaround I can think of would be to use borg check to check if the repository is created but this would likely end up in ansible runs taking up a long time :/ and the only realistic solution I could think of is to check that borg init fails with return code 2 and a message like "Repository .* already exists." in stderr… Are there good reasons not to do what I try to do ? (I know having multiple repositories will use more disk space.) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#271 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABzVssUAmM0NxCxeejbjV77lshnVQKSYks5tkoVFgaJpZM4GNzoc> .

[ThomasWaldmann]

$ mkdir invalid
$ borg list invalid
/home/tw/invalid is not a valid repository. Check repo config.
$ echo $?
2

$ borg list doesntexist
Repository /home/tw/doesntexist does not exist.
$ echo $?
2

$ borg list repo
Enter passphrase for key /home/tw/repo: 
arch                                 Tue, 2018-03-27 00:58:06 [9db...c05]
...
$ echo $?
0

[paolobenve]

I'm having the same issue, because my script runs borg for a subdirs set, and a new subdir may appear.

I use

borg init --encryption=none $BACKUPS_DIR 2> /dev/null

before every borg create command.

[hi2u]

I've written multiple wrapper systems for both attic + borg in multiple languages, and having to check if the repo exists and init only when it doesn't is the most complicated part. These scripts could be much simpler otherwise. Everyone else using scripts around borg, including all the Ansible configs etc have to write this logic and execute multiple commands for themselves too. It would be nice if it were just built in (and optional to use obviously).

I really wish there were a combined init+create command where I just put all the settings into a single command and borg will auto-init only if needed. It doesn't need to be the default behaviour or anything, but there are a lot of us who would use this.... basically anyone doing any automated setups with borg... which is quite a lot of us now.

So maybe rather than a new flag on the init command as suggested in this issue, it would be even simpler as a new flag --auto-init on the create command? Or perhaps instead even a new separate initcreate command to clearly separate it from the regular imperative init + create commands?

As for the 'needs to ask for passphrase' thing, that is already handled by either environment variables and/or using keyfile repos. Everybody that wants this feature is already scripting their own non-interactive inits.

For the people that don't want auto-init / want to know when the repo doesn't exist... obviously they just wouldn't use this feature. For me personally, I would use the feature on most of my repos, but not all them.

This feature would bring Borg more in line with more modern declarative over imperative practises of systems administration in general, and reduce a lot of the redundant (and bug prone) scripting that users of borg need to do for themselves for this fairly common use case.

[fantasya-pbem]

Current behaviour of borg init is to exit with code 2 if repository does exist. Think of a new option like --warn-if-exists that makes Borg exit with code 1 if repo does exist (instead of error code 2).

Would this be a solution? Or do you require exit code 0?

[callowaylc]

@fantasya-pbem exit 1 is a generic error in shell, which makes that result less useful and still just as annoying.

$ docker run -it --rm alpine sh
/ # apk add -q moreutils
/ # errno 1
EPERM 1 Operation not permitted

Most shell scripts (should) have exit early flag, so the strategy for conditional checks is to bail on any statement that returns an exit code != 0.

As others have suggested before me, i'd like to automate this behavior without having to worry whether the repository exists, because from a declarative perspective, my only concern is that it does exist, not whether it did or did not exist previously.

[MEschenbacher]

Let me tell you my story:

I'm using borg init within an ansible playbook and am missing a idempotent way of making sure that the repo exists and was inited before continuing/completing the play. In my case, the client inits the repo and uses ssh as transport.

During the process, I adopted the exit status as indication by allowing

• 0 for successful creation
• 2 for an already existing repo

This worked well until I realized that a unsuccessful ssh connection to the borg master (e.g. because of a blocking network ACL or firewall) results in borg existing with the same known-to-be-good exit status of 2.

Back to quare one. I believe there is still need for a idempotent way to make sure the repo exists.

[ThomasWaldmann]

2 just means something like "sudden, fatal error". 1 is some warning (needs manual log checking). 0 is success.

[MEschenbacher]

Yep thanks for confirming.

I realized back in 2015 you argued that borg init was supposed to be a manual step, however, it is perfectly possible to fully automate repo creation using BORG_PASSPHRASE or similar. Is this design -- or rather -- usage choice currently still valid?

Do you acknowledge the need for such a flag e.g. borg init --existing-ok?

[ThomasWaldmann]

i still think this is complex, unclear and can have / cause lots of issues.

so if somebody wants to put a lot of work into this, make a PR (after making a good plan considering all the potential cases) and I'll review it.

[MEschenbacher]

We will have to figure out the exact goal, scope and behaviour of --existing-ok before beginng the implementation.

I imagine --existing-ok to behave exactly as a regular borg init except that it executes a check beforehand to see if repository path exists. If true, it must not perform any write operations and exit 0. This ensures that no data is ever overwritten accidentally in the case of an existing repository.

The feature is fully backward compatible as it does not change existing behavour.

I'm eager to hear feedback on the proposal as well as your insight and consideration into the complexity and problems it may cause.

[David-5-1]

I think the ideal for provisioning tools like ansible would be to have a command that ensures the path is a working Borg repository. But I guess this is simply not reasonably possible (i.e. without simply dropping the directory in case everything is not OK).

The suggestion MEschenbacher detailed would IMO be practical to simplify normal situation handling through ansible and the like. However, I think the name should be clearer about the fact is does nothing if the directory exists and is not empty (it could be --skip-nonempty-repository).

[SchizoDuckie]

Can we please decide on a solution for this since it's a 6 year old issue that has some real valid use cases?
I'm running into the same issues as above (initializing repositories automagically) and it seems so trivial to me to provide an argument for the simple use case of automation.

[ThomasWaldmann]

We recently automated some borg stuff and I ran into issues with the borg list based idea (see above).
It does not work if the repo already exists AND a backup is currently running into it, locking the repo.
--bypass-lock is also not helpful to work around this case.

The current ansible solution we use is like this:

- name: Init Borg repo
  ansible.builtin.command: borg init --encryption repokey-blake2
  environment:
    BORG_REPO: '{{ borgbackup_location }}'
    BORG_PASSPHRASE: '{{ borgbackup_passphrase }}'
  when: result.rc == 2
  register: result
  changed_when: result.rc == 0
  failed_when: result.rc == 2 and 'A repository already exists at' not in result.stderr or result.rc != 2 and result.rc != 0

[dami0]

Could a different error code be used for when the entire directory of the repo doesn't exist or would that complicate the code too much?

[ThomasWaldmann]

@dami0 do you mean the parent directory of the repo directory?

[dami0]

No, the actual directory of the repository itself, just to simplify the error handling so we can distinguish between the repository not existing and problems accessing like when the lock is present.

[ThomasWaldmann]

@dami0 still working on improvements in #7976 ...

From frontends.rst docs (the rc exit codes given will be used when "modern" exit codes are enabled):

    Repository.AlreadyExists rc: 10 traceback: no
        A repository already exists at {}.

    Repository.DoesNotExist rc: 13 traceback: no
        Repository {} does not exist.

    Repository.InvalidRepository rc: 15 traceback: no
        {} is not a valid repository. Check repo config.

    Repository.InvalidRepositoryConfig rc: 16 traceback: no
        {} does not have a valid configuration. Check repo config [{}].

    Repository.ParentPathDoesNotExist rc: 18 traceback: no
        The parent path of the repo directory [{}] does not exist.

    Repository.PathAlreadyExists rc: 19 traceback: no
        There is already something at {}.

    PathNotAllowed rc: 83 traceback: no
        Repository path not allowed: {}

So, guess this is enough for scripts to decide what to do if borg exits with one of these codes.

Especially borg init terminating with rc 10 should be useful.

[ThomasWaldmann]

In borg 1.4-maint and master branches, borg init will now fail with a very specific return code if the repo already exists (for compatibility reasons this needs BORG_EXIT_CODES=modern at least in 1.4).

I think this is the better solution than adding an option to ignore this and the return code can be checked in the wrapper and it can react accordingly, depending on whether that is expected or not (usually one runs borg init only once and not onto existing repos).