[Bug] unicode normalization problem

[GomPam]

mac use to NFD and windows is NFC mechanism. so, we get this normaliztion problem.
user text (like commit message, author..) display different between mac and windows if not used alphabet.
can you fix it?

Link: https://en.wikipedia.org/wiki/Unicode_equivalence

[alkee]

Voting up.
I need really this as Korean user in file names and commit messages.

[kueecc]

+1

[sckimos]

+1

[kueecc]

Any updates?

[DanPristupov]

How is that supposed to be fixed? Do other MacOS applications work properly?

[GomPam]

@DanPristupov
As far as I know, SourceTree is known to be fine.

I find another git repo issue like this problem
nodejs

nodejs/node#2165
https://nodejs.org/en/docs/guides/working-with-different-filesystems/

[alkee]

simply in swift, just make sure always use NFC encoding from user input like commit message and file path

   let nfd = "맥과 윈도우즈는 한글 저장방식이 다릅니다.ext" // string by user input
   let nfc = nfd.precomposedStringWithCanonicalMapping // converting to nfc
   // now, you can use `nfc` instead of `nfd`

little more detailed in

• https://jongampark.wordpress.com/2008/07/21/unicode-normalization-and-nsstring/

[GomPam]

@alkee good infomation 👍
@DanPristupov need more infomation?
we need a feedback :)

[DanPristupov]

I'm still not really sure why the problem happens and what should be a correct solution for that (from the Apple point of view). Are developers supposed to convert every input string? What will happen with other languages after that?

However I made a small test.

Made a "맥과 윈도우즈는 한글 저장방식이 다릅니다.ext" commit in Mac, and in Windows it became unreadable as you said.
Made a "맥과 윈도우즈는 한글 저장방식이 다릅니다.ext" commit in Windows. Everything seems to be correct on both platforms.

However passing strings in nfc form (using precomposedStringWithCanonicalMapping) to git process doesn't change anything (I used NSProcess().arguments). The string is still corrupted.

Docs (https://developer.apple.com/documentation/foundation/nstask/1414375-launchedtaskwithlaunchpath).

The NSTask object converts both path and the strings in arguments to appropriate C-style strings (using fileSystemRepresentation) before passing them to the task via argv[] .

[alkee]

Sorry for the not worked way. It's OS specific behavior for unicode, so there is no correct solution. In the point of git (not an Apple), they support configuration for the issue.

Can you test one more thing with the git config core.precomposeunicode ?

git config --global core.precomposeunicode true

It worked well with terminal git and happy to other platforms.

[GomPam]

@DanPristupov
um.. i make that sample repository
https://github.com/GomPam/UnicodeNormalizeTest

• MacOSX
  

• Windows
  

i use to same sentence.

유니코드 정규화 확인을 위한 문장. And Check With Alphabet.

[GomPam]

@alkee "core.precomposeunicode" config is affects filenames.

https://git-scm.com/docs/git-config#Documentation/git-config.txt-coreprecomposeUnicode
... When core.precomposeUnicode=true, Git reverts the unicode decomposition of filenames done by Mac OS.

so... i think its not working.. :(

[GomPam]

@DanPristupov
i make little program.
just print "arg + hex string"
and then i understand what you said.

However passing strings in nfc form (using precomposedStringWithCanonicalMapping) to git process doesn't change anything (I used NSProcess().arguments). The string is still corrupted.
(#444 (comment))

so
i think a trick.
make temp shellscript then execute it.
maybe i think this work.

===
updated.
test done. its works.
https://gist.github.com/GomPam/3986e317774085e3fb25cb5c00be0dd0

[GomPam]

@DanPristupov
can we use this way.. or not?
update plz :)