Exclude patterns GUI improvements

[m3nu]

Just noticed that Nextcloud (desktop client) comes with a nice list of default exclusions. Maybe something for Vorta to assimilate.

I agree that one simple kind of patterns would be enough.

Originally posted by @m3nu in #838 (comment)

[erAck]

BackInTime also has a sensible set of default excludes:

• .gvfs
• .cache/*
• .thumbnails*
• [Tt]rash*
• .backup
• *~
• .dropbox*
• /proc/*
• /sys/*
• /dev/*
• /run/*
• /etc/mtab
• /var/cache/apt/archives/*.deb
• lost+found/*
• /tmp/*
• /var/tmp/*
• /var/backups/*
• .Private

[m3nu]

We discussed adding defaults before and the feedback wasn't good due to the large variety of OS and use cases.

So if we add defaults, they will be OS-specific and disabled by default.

[m3nu]

Suggested by @Wikinaut in #970

I cannot find that borg option --exclude-caches to be used by vorta. Was that intended?

See https://borgbackup.readthedocs.io/en/stable/usage/create.html?highlight=exclude%20caches

I suggest to add this as a vorta option which is active by default.

[m3nu]

Getting close to starting this feature. Was thinking about this a bit more and would want those features:

• Maintain templates of exclusions and enable them for selected profiles. Like "Dev Cache Files" or "App Cache Files".
• Currently I use 2 groups of exclusions: specific paths (e.g. ~/Library/BlahBlah/Cache and patterns that should be matched everywhere (e.g. */.git)
• Maybe we also want "Exclude folders with this file"
• Add description to exclusion
• Import and export exclusions as CSV? (pattern, desc)

Anything to add? @Hofer-Julian @bastiencyr ?

[Hofer-Julian]

Sounds good.
One idea:

• Add description to exclusion
• Import and export exclusions as CSV? (pattern, desc)

How about allowing comments with #?
In that case you don't need an extra field for descriptions and the export format would be a list of patterns and comments instead of a csv.

[m3nu]

I like the idea of making the import/export file a valid patterns file.

In the UI, I'd prefer to have pattern and commend separate. Else we literally have a free text box again.

[bastiencyr]

Souds good too.

Maintain templates of exclusions and enable them for selected profiles. Like "Dev Cache Files" or "App Cache Files".

For naming, maybe some more meaningful names for users like "User Temporary Files", "Dev Irrelevant Files". Its more user orientated and may not know what a cache is.

[m3nu]

macOS via arkTanlis on Reddit.

(Some wildcards used here are not appropriate for the default fnmatch patterns we use.)

~/.Trash
**.DS_Store
~/Library/Caches
~/Library/Logs
**.cache
~/.npm
~/Library/Containers
**.git
**node_modules

[m3nu]

Adding my own excludes:

*/.DS_Store
*/._sync_*
*.pyc
*.mp4
*.mkv
*/.~*
*-journal
*/.owncloudsync.log
*/.cache
*/__pycache__
~/Pictures/20*
~/Pictures/Phone

[m3nu]

Some more inspiration from Back in Time. They probably keep a list for Linux.

[real-yfprojects]

I sketched a first GUI mock-up. What do you think?

[m3nu]

I think it's great. 🙌 Not much to add.

So the data model is like description, pattern, OS, on/off? Some of those are built-in and some are supplied by the user? We should put the built-in patterns in a reusable format, since they may be relevant for other projects, if we end up collecting enough.

[Hofer-Julian]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

[real-yfprojects]

So the data model is like description, pattern, OS, on/off? Some of those are built-in and some are supplied by the user? We should put the built-in patterns in a reusable format, since they may be relevant for other projects, if we end up collecting enough.

There patterns that can be grouped together with a name. They can be toggled through the GUI or by commenting them out in raw text.
The predefined pattern groups that have tags (if they are app, os specific,...) and possibly a description. There is a search bar for searching through the patterns/groups by group name and tags as well as pattern contents.

We can integrate online sources for patterns like gitignore.io or github/gitignore for excluding different development files.

[real-yfprojects]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

I had a look at Pika Backup. It seems like they only allow excluding existing files/folders that can be added to the exclusion list through a file chooser.

[m3nu]

That's not very flexible. E.g. to exclude a type of files.

[Hofer-Julian]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

I had a look at Pika Backup. It seems like they only allow excluding existing files/folders that can be added to the exclusion list through a file chooser.

That specific feature is not released yet.
But that screenshot should give you an idea: https://gitlab.gnome.org/World/pika-backup/-/issues/94#note_1504221

[m3nu]

Yeah, having multiple patterns in a group makes sense too.

So the data struct would be like: group description, [list of patterns], OS (as per Python sys.platform), on/off

[real-yfprojects]

Yeah, having multiple patterns in a group makes sense too.

So the data struct would be like: group description, [list of patterns], OS (as per Python sys.platform), on/off

Yes, that would be a good start. However the user should be able to toggle single patterns, edit the predefined patterns and add own patterns. This could be realised in two database tables:
PatternGroup and Pattern. However my idea was getting the data from a raw text file/entry the user can edit.

That specific feature is not released yet. But that screenshot should give you an idea: https://gitlab.gnome.org/World/pika-backup/-/issues/94#note_1504221

Thanks. I like that the mockup shows the size of the excluded files.

[fooness]

Hello everyone. Could you please point me to the current list of different exclude defaults/suggestions? Maybe I missed the link here, or did search for the wrong keywords. I thought I could look the exclusions up here in the repo, without installing Vorta (as I’m basically using borg without GUI for my macOS backups).

---

Here’s what I came up with for now; it’s a mixture of Arq’s exclusion list as well as the suggestions from the previously linked reddit thread and from the posts in this very GitHub issue … comments and feedback very much appreciated; this is macOS-specific.

**/.*.swp
**/.DS_Store
**/.VolumeIcon.icns

# not quite sure what the following two exclusions are for
**/.file
**/.vol

**/.DocumentRevisions-V100/
**/.MobileBackups/
**/.MobileBackups.trash/
**/.Spotlight-V100/
**/.TemporaryItems/
**/.Trash/
**/.Trashes/

**/.cache/
**/.fseventsd/
**/.npm/

**/Library/**/Cache/
**/Library/**/Caches/
**/Library/**/Containers/
**/Library/**/CrashReporter/
**/Library/**/DerivedData/
**/Library/**/Developer/
**/Library/**/Metadata/
**/Library/**/MobileSync/
**/Library/**/Logs/

**/iTunes/

**/build/
**/dist/
**/node_modules/
**/tmp/
**/vendor/

/System/
/Users/Shared/
/Volumes/
/bin/
/cores/
/dev/
/etc/
/home/
/macOS Install Data/
#/nix/
#/opt/
/private/
/sbin/
/tmp/
#/usr/
/var/

# comment: /nix/ + /opt/ + /usr/ maybe should not be excluded due to e.g. homebrew …

[m3nu]

Thanks for sharing! We don't ship a default list yet. Still work in progress. Based on the current plan, we may do different groups that can be toggled. Since exclusions are really OS-dependent and personal.

[fooness]

Based on the current plan, we may do different groups that can be toggled. Since exclusions are really OS-dependent and personal.

Yes, the definitely are; been looking for the macOS-specific list.

We don't ship a default list yet. Still work in progress.

That’s the part I obviously have missed reading through this issue; will keep an eye on future updates.

PS: I’ll be doing some more research on this matter, and in case of any new insights, I’ll be editing/updating my list above.

[diivi]

• Okay, I'll change the current Patterns tab to Custom and make that accept only custom exclusion rules.
• I'll add a presets tab that has all the grouped rules, and will be similar to the Patterns tab, but will have preset names instead of rules and a tooltip to show the details. Or, I'll make it collapsible to view all excluded files (I can do this on Figma, but will need help in making a collapsible list on qt).
• Raw remains the same
• The add dialog will be a single input text field to add a single custom exclusion rule?

Finally, Preview would merge it all together to help debugging.

Do you mean there should be another tab? What will this tab show? I think the check input below is sufficient for debugging.
Also, the raw view will already show all the rules inside a preset group.

[m3nu]

Preview tab could show all rules combined how they are passed to Borg later.

Raw would be just for user input. If you combine both, how do you know what the user edited and what's from our own presets?

[diivi]

Okay, and how do you recommend the separation of user added rules and vorta preset rules be done in the preview tab?

[diivi]

Also, I'm concerned that having four tabs and too many features may be counterproductive, as it could make the interface more difficult for novice users to navigate.
I agree with your first point about the presets getting updated and a simple text field might not be the best solution. But I'm not so sure about adding so many things :/. Maybe we can just change the way presets are handled and added?

[real-yfprojects]

I like @m3nu suggestion. What about having two tabs: Patterns and Raw. The pattern tab shows the patterns and collapsible pattern groups including added presets. The raw tab shows the custom patterns at the beginning and the preset patterns appended to the end. However the section of the textedit with preset patterns is greyed out and can't be edited. There won't be any merging of preset and non preset patterns, borg can handle duplicates.

[diivi]

Cool, if that's what we decide, then I'll edit my design to accomodate groups and single rules both in the patterns tab.
What about the add tab? Do you think I should change something? Maybe disable the textfield when a preset is selected? Or, with the current approach:

• if it is edited then change the preset to none and treat them as seperate rules
• if an unedited preset is added, we can keep track of it and update it easily.

[vv111y]

This is great that you are doing this. While researching I found this repo for list of excludes for linux /home with rsync, it should be good for this feature
https://github.com/rubo77/rsync-homedir-excludes

[diivi]

Thankyou! This is really helpful :)

[diivi]

Thanks to everyone who participated in this thread!

The current iteration after community review and feedback from the maintainers can be found here - https://www.figma.com/file/Y70aJFOpJ1VT59lWzUZEoQ/Exclude-GUI?type=design&node-id=0-1&t=vXY1HiJ152wU6FhF-0.

Attaching screenshots here too:

This is one of my GSoC projects, and I'll start coding in a few weeks. Please let me know if anyone has other suggestions/improvements that I can look into before that!

[m3nu]

Looks good and what I imagined really. Minor thoughts:

• For the Preview tab it should be clear that the text isn't editable. Else I imagine it's the same as the Raw tab.
• The text box at the bottom with "Placeholder" is for testing? Writing a full path may not be feasible and we need a file selector there? Or leave out the feature for now.
• Custom tab: We want to add a comment/notes for custom patterns?
• Search feature: not sure that's useful. Maybe just in the Custom tab for filtering out lines? Not sure how easy it is to select text in Qt in a textbox.

[sammcj]

It might be good to include a field that you can provide exclusion lists to, it's not currently clear if the UI supports .lst files or not.

e.g, the vorta equivalent of running borg create --patterns-from=borg-exclude-macos.lst

Where the .lst contains items such as:

# Siri
! **/Library/Assistant/SiriAnalytics.db
! **/Library/Assistant/SiriSyncItems.db
! **/Library/Assistant/SiriVocabulary/sv.db

# Logs and Monitoring
- **/Logs/

[real-yfprojects]

The current mockup doesn't allow patterns afaik but only excludes. There is a TextEdit for pasting in your own excludes e.g. from a different file.

[diivi]

Hey, I have started work on this feature and will soon get started with the Presets tab. These presets will be shipped by Vorta and you'll just have to select some presets and all patterns included in that preset will be excluded from the created archive!

I need some feedback/suggestions for these presets though, the format I need is -

Preset Name
Description
/pattern1
/pattern2
...

Preset 2 Name
Description
/pattern
...

So if anyone has a grouped list (or even one set of patterns), like this - #907 (comment) it'd be helpful to get things started on this feature. Thanks!

P.S. you can add your preset by opening an issue after this feature is shipped too. These presets will be a json file in the Vorta codebase itself.

[m3nu]

Parsing this will be a minefield and we may want more metadata. Would use a dict/json format.

name, desc, os, rules format, list of actual rules, maybe?

[diivi]

Oh yeah, the implementation itself will be a json, I just meant to list what's needed 😅

[sammcj]

If it helps these are the excludes I use on macOS:

**/.minikube/
**/.npm/
**/.platformio
**/.rbenv
**/.rvm
**/.timemachine/
**/.Trash
**/.wine
**/*.DS_Store
**/*.git
**/*.temp
**/*.tmp
**/*diffusionbee/
**/*docker-desktop/
**/*lima/
**/*nstall*.dmg
**/*universal*.dmg
**/*x64*.dmg
**/docker-volumes/
**/iTunes/iTunes Media/podcasts/
**/iTunes/iTunes Music/
**/miniconda3/
**/node_modules/
**/nomic.ai/
**/temp/
**/tmp/
**/Virtual Machines/
**/Xcode.app/
/Applications
/Users/**/_NoSync_NoBackup/
/Users/*/.asdf
/Users/*/.cache
/Users/*/.cargo
/Users/*/.colima
/Users/*/.docker
/Users/*/.platformio
/Users/*/Applications
/Users/*/container_mount/
/Users/*/Documents/Adobe/
/Users/*/git/
/Users/*/go/
/Users/*/Library/Application Support/Autodesk/
/Users/*/Library/Application Support/Caches/
/Users/*/Library/Application Support/Code/Cache*/
/Users/*/Library/Application Support/coreMLCache/
/Users/*/Library/Application Support/Firefox/Profiles/*/storage/temporary/
/Users/*/Library/Application Support/Firefox/Profiles/*/storage/to-be-removed/
/Users/*/Library/Application Support/Firefox/Profiles/storage/default/
/Users/*/Library/Application Support/fnm/
/Users/*/Library/Application Support/Google/Chrome/*Cache*/
/Users/*/Library/Application Support/Google/Chrome/File System/
/Users/*/Library/Application Support/Google/Chrome/IndexDB/
/Users/*/Library/Application Support/Google/Chrome/Safe Browsing/
/Users/*/Library/Application Support/Google/Chrome/Service Worker/
/Users/*/Library/Arduino15/
/Users/*/Library/Caches/
/Users/*/Library/Containers/com.apple.AMPArtworkAgent/
/Users/*/Library/Containers/com.bitdefender.*/
/Users/*/Library/Containers/com.microsoft.*/
/Users/*/Library/Containers/F807763E-5B70-49A2-B0B0-E13291698A23/
/Users/*/Library/Developer/
/Users/*/Library/Metadata/
/Users/*/Library/Saved Application State/
/Users/*/Music/
/Volumes/*timemachine*/
/Volumes/*TimeMachine*/

[jakesmolka]

Sounds awesome @diivi 👍 Looking forward using this feature.
Please also check https://github.com/SterlingHooten/borg-backup-exclusions-macos not just for the lists, but also for the knowledge – if that helps.

[m3nu]

Love all the input, @jakesmolka and @sammcj . Will be great to bundle this knowledge with Vorta. We just need to group it all a bit.

@diivi , you see your feature is already very popular. I suggest to add the most obvious input from here to the first version, but also prioritize getting something usable out of the door and then add more exclusion rules over time.

[goebbe]

Thank you for the flexible interface for "exclude"!

There is one point that is quite confusing in borg - and since Vorta is referring to the borg documentation, this applies to Vorta too.

In borg there are two distinct sets of options which can be used for excludes:
--exclude and --exclude-from
AND
--pattern or --patterns-from

Only the second set of options uses the term "pattern" in its name.
To cite the borg documentation:

A more general and easier to use way to define filename matching patterns exists with the --pattern and --patterns-from options. Using these, one may specify the backup roots, default pattern styles and patterns for inclusion and exclusion.

If I understand correctly, currently, Vorta only supports --exclude. This should be stressed in the user interface (which links to the borg documentation) to reduce the confusion.

[m3nu]

Closing this, since the feature is done. Move the last comment by @goebbe to a new issue.

[notDavid]

So if anyone has a grouped list (or even one set of patterns) [...]

Hi there,

i'm only seeing this now while the issue was just closed, but perhaps this exclusion list for macOS is still useful...: https://github.com/SterlingHooten/borg-backup-exclusions-macos

[m3nu]

Could be. Made a new issue for this. Thanks for the link!