Possible problem with the -spd switch and the option to exclude Mac resources?

[jajajaneeneenee]

Configuration

• Keka version: 1.4.5 -r5509
• macOS version: 10.14.6

Describe the bug

I've just started experimenting a bit with Keka's terminal support (it works well!), and I also switched on the verbose console output for compression as a test – when using the App (not via terminal). There I then saw which switches are set for the keka7zz binary when compressing something via the app (I tried with ZIP and had activated the "Exclude Mac resources" option in Keka).

I noticed the following:
The console output showed that these switches (among others) were set with wildcards to exclude Mac resources:

-xr!._*
-xr!.FBC*
-xr!.com.apple.timemachine.*

But in addition (according to the console output) the switch -spd is also set (which means: "disable wildcard matching for file names"). This also seems to apply to the -x switches ...

Therefore, the above-mentioned -x switches will probably not work, and I also checked it with a test file. The result: a file such as "._test" is NOT excluded (while files like ".DS_Store" – without wildcards in the switch, here -xr!.DS_Store – were correctly excluded).

To Reproduce

• Create a test file with the name "._test" (e.g. in a folder with other files).
• Try to compress this folder with Keka and the option "Exclude Mac resources" activated.
• Check if the file "._test" was excluded or not.

Expected behavior

• The exclusions with wildcards should also work.

[aonez]

Thanks a lot for pointing to this @jajajaneeneenee. -spd is being used since v1.3.0, crazy this wasn't detected before.

I've fixed this in this build: https://github.com/aonez/Keka/releases/download/dev-test-builds/Keka-v1.4.6.r5535.7z

I'll be still using that option since it fixes other issues (#488 #1236) but I've added another flag (-keka) so exclusions still use wildcards. I'll probably let know Igor about this one providing a proper flag (-spde?) and patch. Below is the patch I've applied.

Patch

From 87dc425b78b15a04815f45bd87cf4e8a1e001844 Mon Sep 17 00:00:00 2001
From: aone <aone@keka.io>
Date: Wed, 20 Nov 2024 11:07:02 +0100
Subject: [PATCH] Always enable wildcard on exclusions

https://github.com/aonez/Keka/issues/1548
---
 CPP/7zip/UI/Common/ArchiveCommandLine.cpp | 10 ++++++++++
 CPP/7zip/UI/Common/ArchiveCommandLine.h   |  2 ++
 2 files changed, 12 insertions(+)

diff --git a/CPP/7zip/UI/Common/ArchiveCommandLine.cpp b/CPP/7zip/UI/Common/ArchiveCommandLine.cpp
index 45baca5..9d073aa 100755
--- a/CPP/7zip/UI/Common/ArchiveCommandLine.cpp
+++ b/CPP/7zip/UI/Common/ArchiveCommandLine.cpp
@@ -181,6 +181,8 @@ enum Enum
   #ifndef Z7_NO_CRYPTO
   , kPassword
   #endif
+
+  , kKeka
 };
 
 }
@@ -332,6 +334,8 @@ static const CSwitchForm kSwitchForms[] =
   #ifndef Z7_NO_CRYPTO
   , { "p", SWFRM_STRING }
   #endif
+
+  , { "keka", SWFRM_SIMPLE }
 };
 
 static const char * const kUniversalWildcard = "*";
@@ -1314,6 +1318,8 @@ void CArcCmdLineParser::Parse2(CArcCmdLineOptions &options)
   }
   */
 
+  options.KekaMode = parser[NKey::kKeka].ThereIs;
+
   if (parser[NKey::kHashDir].ThereIs)
     options.ExtractOptions.HashDir = parser[NKey::kHashDir].PostStrings[0];
   
@@ -1387,8 +1393,12 @@ void CArcCmdLineParser::Parse2(CArcCmdLineOptions &options)
   if (parser[NKey::kExclude].ThereIs)
   {
     nop.Include = false;
+    bool bWildcardMatching = nop.WildcardMatching;
+    if (options.KekaMode)
+      nop.WildcardMatching = true;
     AddSwitchWildcardsToCensor(options.Censor,
         parser[NKey::kExclude].PostStrings, nop, codePage);
+    nop.WildcardMatching = bWildcardMatching; // Restore original setting
   }
  
   unsigned curCommandIndex = kCommandIndex + 1;
diff --git a/CPP/7zip/UI/Common/ArchiveCommandLine.h b/CPP/7zip/UI/Common/ArchiveCommandLine.h
index 439ee70..a5b7714 100755
--- a/CPP/7zip/UI/Common/ArchiveCommandLine.h
+++ b/CPP/7zip/UI/Common/ArchiveCommandLine.h
@@ -91,6 +91,8 @@ struct CArcCmdLineOptions
   UString Password;
   #endif
 
+  bool KekaMode;
+
   UStringVector HashMethods;
   // UString HashFilePath;
 
-- 
2.45.2

[jajajaneeneenee]

Many thanks for the quick response! I still have to try the dev test version 1.4.6...

Thanks also for the information about other problem with the asterisk *! Is this still a relic of 7z's origins as a Windows tool?

As far as I know, problematic or forbidden characters in Windows for file/foldernames are < > ? " : | \ / * (and for some applications also &, although it is actually allowed). And a point . at the end of filenames is also not allowed in Windows. And I know that under macOS there are other characters, especially : ...

Is there a risk that characters other than * (from the list above?) in paths can cause problems for Keka, e.g. the question mark ?? Have the critical characters ever been systematically tested?

I know that you should avoid special characters in file/folder names if possible, but if you are only used to macOS (in my case since OS6 ...), you don't necessarily see some of these characters as “special characters” that you should avoid in file/folder names. And I just checked if I have folders/files with * in the name in my personal local files – found over 300. And over 1400 with ? ... Surprise!

I forgot to mention: when using Keka via the terminal, other switches that allow wildcards (e.g. -i, -ai, -ax) should also be possible with wildcards without causing problems with * (or other problem characters) in path names. That would be nice anyway... I don't know if this is easily possible or if more extensive changes would be necessary in the customized version of 7z... I just wanted to mention it. It's probably not quite trivial (?).

And if it is not possible (or not soon), perhaps it should be mentioned on the Terminal support wiki page.

[aonez]

There's really not a problem with the asterisk per se, it's by default used as a wildcard. -spd ignores it as a wildcard and is used as a regular character but this also applies to exclusions and inclusions, so I've modified it so exclusions also treat it as a wildcard.

There should be no issues with any character as they are all converted to UTF8. That said this depends on the OS, app and filesystem you're using to code and decode, so it's always better to be safe and don't use special characters.

As a norm, I always avoid most special characters in filenames, usually even spaces (replaced with - or _) since it makes terminal work harder on occasions and even for websites. Even an extra dot . can sometimes give unexpected results when getting an extension or removing one. While you can even use / in Finder, note it is internally replaced with :. Probably that's why you can't use : directly.

I forgot to mention: when using Keka via the terminal, other switches that allow wildcards (e.g. -i, -ai, -ax) should also be possible with wildcards without causing problems with *

Again note this is not really a problem, but that * is by default a wildcard and -spd disables it globally.

[jajajaneeneenee]

Hello! OK, I understand ... I only asked for the question mark, because it can also be used as a wildcard (for a single character) – as far as I know – and if the asterisk in a file/foldername can cause problems as mentioned in #488/#1236, I wondered especially if something similar could happen with a question mark (I haven't tested it).

One question: Should -keka in the dev test version of v1.4.6 also be able to be used as a switch for 7z when calling Keka via the terminal? II tried it, but got the error message "Command Line Error: Unknown switch: -keka".

[And a little off-topic: I had problems with the context menu in the Finder after replacing v1.4.5 with the dev test version of v1.4.6 – the context menu commands were only processed after Keka was opened manually – is this “normal”, e.g. due to the dev test version? When I replaced it with v1.4.5, it worked again at once - although I had to deactivate and reactivate the context menu actions in the Keka settings once for them to even be displayed in the Finder. Or is that a case for a separate issue?]

[jajajaneeneenee]

Regarding the context menu: I have to correct myself. It doesn't work properly with v1.4.5 either, even after uninstalling, reinstalling Keka and restarting the Mac (before I installed 1.4.6 dev test, it worked). I don't know how to fix it.

Update: Could fix it with a reinstall of v1.4.5. v1.4.6 was still there, uncompressed in a subfolder. Maybe that was the reason...

[aonez]

@jajajaneeneenee in the dev build only 7zz is modified, not 7z (older). The final build has this flag in both versions but you should always use 7zz over 7z (p7zip) if possible, if using macOS 10.13 or newer.

As per the extension probably macOS sandbox magic is doing its thing. Just keep one version of Keka and restart your Mac if needed.

[jajajaneeneenee]

Hello @aonez! Thanks for the hints ... They are very helpful!

I apparently haven't figured out all the details of the binaries yet. I wasn't even aware that there is a 7z and a 7zz binary - my starting point for terminal exploration was https://github.com/aonez/Keka/wiki/Terminal-support, where these binaries are listed:

Binaries:
  7z
  brotli
  lbzip2
  lrzip
  lzip
  pbzip2
  pigz
  plzip
  rar
  tar
  unace
  unar
  unrar
  xz
  zstd

That's why I used 7z and assumed that this is the right one. But now I saw that the --help option of Keka has this output (and I never noticed the difference ...):

Binaries:
  7z
  7zz
  brotli
  kwet
  lbzip2
  lrzip
  lz4
  lzip
  pbzip2
  pigz
  plzip
  snzip
  tar
  unar
  unrar
  xz
  zstd

Maybe it would be useful to update the wiki info ... for stupid beginners ... There are also some outdated commands listed (I assume), such as (and here also with 7z an not 7zz):

To see each binary options just run --client [binary] --help.
Example: Keka --client 7z --help

(7z instead of 7zz and --client instead of --cli)

And with the context menu extension, it's really not that easy to get it working again if it doesn't work properly - I've had to restart several times, delete Keka and the extension in ~/Library, new install etc. until it finally worked again... I hope that doesn't happen too often.

[aonez]

Sadly the finder extension is totally controlled by macOS, Keka can’t do nothing about it other that have it inside.

I’ll update the wiki, thanks for pointing that 🙌🏻

[aonez]

@jajajaneeneenee wiki updated and v1.4.6 released with -keka flag. Thanks again for pointing this out 🦾

[jajajaneeneenee]

Thank you for the quick fix!

One last note (also on the subject of "special characters"): if I used Keka via the terminal, there were switches of 7zz such as -x[r[-|0]][m[-|2]][w[-]]{@listfile|!wildcard} (e.g. -xr!._*), where the exclamation mark ! was a problem because it is a reserved character in bash. So I either had to escape it with a backslash (e.g. -xr\!._*) or put the switches in single quotes (e.g. '-xr!._*'). Otherwise I got the error message -bash: !._*: event not found in the terminal.

You don't necessarily expect special characters to appear in switches for terminal commands. This is also a nasty stumbling block for Keka terminal beginners. Perhaps it would be worth mentioning on the terminal support wiki page?