enum command: useful for debugging filter patterns

[SWB54]

This implements a new command, enum, which provides functionality useful for debugging filter patterns.

This command essentially executes just CreateSnapshotFromDirectory() (albeit a new version called CreateSnapshotFromDirectoryFilterDebug()) on the repository directory, but provides additional options for logging the snapshot creation process. For example:

• You can list entries included or excluded from the snapshot by the active filter patterns.
• You can list entries explicitly affected by a filter pattern or those implicitly processed by default.
• You can list files, directories, or both.

The output prints whether the entry is included or excluded in the snapshot, the rule (if any) that caused the entry to be included or excluded, and the full path of the entry. Output is tab-delimited, for ease of importing into a spreadsheet for further analysis.

NAME:
   duplicacy enum - enumerate the the repository

USAGE:
   duplicacy enum [command options]

OPTIONS:
   -all                 DEFAULT: include any file or directory (equivalent to -explicit -implicit)
   -explicit            include files or directories affected by a pattern in the filters file (explicit processing)
   -implicit            include files or directories not affected by any pattern in the filters file (implicit processing)
   -entries, -e         DEFAULT: list files and directories (equivalent to -ientries -xentries)
   -ientries, -ie       list included files and directories (equivalent to -ifiles -idirs)
   -xentries, -xe       list excluded files and directories (equivalent to -xfiles -xdirs)
   -files, -f           list files (equivalent to -ifiles -xfiles)
   -ifiles, -if         list included files
   -xfiles, -xf         list excluded files
   -dirs, -d            list directories (equivalent to -idirs -xdirs)
   -idirs, -id          list included directories
   -xdirs, -xd          list excluded directories

[CLAassistant]

All committers have signed the CLA.

[TheBestPessimist]

This is quite a nice idea! (didn't look at the implementation, just tried duplicacy enum > a.txt and checked the output).

I have some complains though:

1. May i complain that the name enum is not very clear about what the command actually does?
   • could test-filters or check-filters be a better name?
2. by just running duplicacy enum it seems that the folders are not listed in the same order as when running the backup command
   • from what i saw, duplicacy goes like this:
     1 . files in current folder
     2 . next unvisited subfolder in current folder
     3 . repeat from 1 with new subfolder as current
     4 . if there are no more subfolders: go up one level, repeat from 2
   • i think this helps a bit with understanding the order of the files on disk.
3. why the output is tab delimited and not comma delimited (as in CSV)?

[TheBestPessimist]

Also, just to be sure: by default enum is run as duplicacy enum -all -entries right? those 2 flags are used by default when i run duplicacy enum.

[SWB54]

the name enum is not very clear about what the command actually does … could test-filters or check-filters be a better name?

Well, it enumerates the repository, so it sort-of does what it says on the tin. Having said that, I'm not at all married to the name. The functionality is all I was after. I really wanted to call it list, but that was already taken. enum was the best alternative I was able to think up. test-filters or check-filters are both perfectly reasonable.

by just running duplicacy enum it seems that the folders are not listed in the same order as when running the backup command

All I do is insert logging output into the normal ListEntries() function, which is called by CreateSnapshotFromDirectory(), which in turn is used by Backup(). The output is therefore listed in exactly the same order that Duplicacy processes it, which might be helpful in diagnosing filter precedence issues.

That initially tripped me up as well: I couldn't find a directory mentioned in the output, for example, that I thought I had made a filter for. It turns out it was listed much earlier than where I was looking, which revealed that Duplicacy was first applying filters to everything in a directory (including subdirectory names) before descending into subdirectories.

just to be sure: by default enum is run as duplicacy enum -all -entries right?

Correct.

[TheBestPessimist]

Also: you already helped me fix a bug with my ignore file :D

[SWB54]

you already helped me fix a bug with my ignore file :D

Great! I am using your file as a base for my own. :-)

[TheBestPessimist]

@gilbertchen what's your opinion on the naming enum vs test-filters vs check-filters vs < other >?
I this this should be changed before the pull request acceptance.

[gilbertchen]

Instead of adding an enum command, I added an -enum-only option to the backup command in commit aadd2aa. I think this is simpler as you can see from the code changes. Moreover, the enum command would create two separate code paths for listing the repository which makes it less dependable for debugging purposes.

Those options for the enum command are very useful, but you can emulate them by running duplicacy -d -log backup -enum-only and then filtering the output with grep.

[gilbertchen]

This pull request has been mentioned on Duplicacy Forum. There might be relevant details there:

https://forum.duplicacy.com/t/backup-command-details/1077/1