filter-branch gets an error when file path contains a CJK char

[danny0838]

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.15.1.windows.2
built from commit: 5d5baf91824ec7750b103c8b7c4827ffac202feb
sizeof-long: 4
machine: x86_64

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [版本 6.1.7601]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: Notepad++
Path Option: BashOnly
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFCommitAsIs
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Bash

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git filter-branch -f --prune-empty --index-filter '
  git rm -r --cached --ignore-unmatch -- pages/中文
  ' -- public/dw

• What did you expect to occur after running these commands?

The command should run normally without an error.

• What actually happened instead?

An error occurred during rewriting:

$ git filter-branch -f --prune-empty --index-filter '
>   git rm -r --cached --ignore-unmatch -- pages/中文
>   ' -- mybranch
Rewrite febf19b92a6d800363b65c0194c2765202c58dfb (1/442) (0 seconds passed, remaining 0 predicted)
  C:\Program Files\Git\mingw64/libexec/git-core\git-filter-branch: line 414:
  git rm -r --cached --ignore-unmatch -- pages/中文
  : No such file or directory
index filter failed: "
  git rm -r --cached --ignore-unmatch -- pages/中文
  "

If "中文" is replaced by "English". The command works fine:

$ git filter-branch -f --prune-empty --index-filter '
>   git rm -r --cached --ignore-unmatch -- pages/English
>   ' -- mybranch
Rewrite febf19b92a6d800363b65c0194c2765202c58dfb (1/442) (0 seconds passed, remaining 0 predicted)  
Rewrite 5e3a55cb0ee855719ed57debc4e6ba00345f1464 (2/442) (0 seconds passed, remaining 0 predicted)  
......

[danny0838]

This issue still persists in 2.22.0:

$ git --version --build-options
git version 2.22.0.windows.1
cpu: x86_64
built from commit: d003d728ffa6c0006da875ec6318d3f6b28a4ddb
sizeof-long: 4
sizeof-size_t: 8

Error when the path contains a CJK char:

$ git filter-branch --index-filter '
>   git rm --cached --ignore-unmatch 中文.txt
>   ' -- master

Rewrite 114ff85a9e6516d9bda6bdcfa9b649fec66e9027 (1/4) (0 seconds passed, remaining 0 predicted)    C:/Program Files/Git/mingw64/libexec/git-core\git-filter-branch: line 427: $'\n  git rm --cached --ignore-unmatch 中文.txt\n  ': command not found
index filter failed: "
  git rm --cached --ignore-unmatch 中文.txt
  "

[dscho]

I cannot reproduce this.

Unfortunately, this report contains only incomplete information, so I had to execute this on a test repository over here. This is my output:

$ git filter-branch -f --prune-empty --index-filter '
>   git rm -r --cached --ignore-unmatch -- pages/中文
>   ' -- master
Rewrite cc5d7558673710c1da61856fa53718f4fe4029c5 (1/4) (0 seconds passed, remainRewrite 597c3b473dd87cb98d015e7469a1ff5d4e0352c7 (2/4) (1 seconds passed, remainRewrite 87894869fd521ba54060ee2a4b42a61725cd6130 (2/4) (1 seconds passed, remainRewrite 90db1139c6bc9cb0eb69e8f8a1b30a2ff3e22558 (2/4) (1 seconds passed, remaining 1 predicted)
WARNING: Ref 'refs/heads/master' is unchanged

This is with v2.22.0:

$ git version --build-options
git version 2.22.0.windows.1
cpu: x86_64
built from commit: d003d728ffa6c0006da875ec6318d3f6b28a4ddb
sizeof-long: 4
sizeof-size_t: 8

Maybe it has something to do with your Windows version? I tried this on the latest Windows 10.

[danny0838]

I did a further investigation and found that the command works on another computer. After some investigation it seems to be related with locale:

For the computer that fails to run the command:

$ locale
LANG=C.UTF-8
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_ALL=

And for the computer the succeeds to run the command:

$ locale
LANG=zh_TW.UTF-8
LC_CTYPE="zh_TW.UTF-8"
LC_NUMERIC="zh_TW.UTF-8"
LC_TIME="zh_TW.UTF-8"
LC_COLLATE="zh_TW.UTF-8"
LC_MONETARY="zh_TW.UTF-8"
LC_MESSAGES="zh_TW.UTF-8"
LC_ALL=

Is there somewhere for the locale to be defined? There's nothing special in the ~/.bash_profile and ~/.bashrc in both computers...

[dscho]

Is there somewhere for the locale to be defined?

You can define it locally in .bash_profile and/or ~/.bashrc, but we also have code to set LC_ALL unless it is set: https://github.com/git-for-windows/git/blob/v2.23.0.windows.1/compat/mingw.c#L3019-L3020

Having said that, it seems to be notoriously hard to get this right. IIRC we had it defined to C.UTF-8 at some stage and it broke something, but I don't recall the details...

[dscho]

Closing this as stale.