This repository has been archived by the owner on Dec 15, 2022. It is now read-only.
Normalize Unicode characters #340
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
winstliu
force-pushed
the
wl-unicode-normalization
branch
from
6f24f26
to
e925dbc
Compare
|
Test plan:
macOS:
Windows:
Linux:
|
|
Testing on macOS the first test case passes, the second one fails 😞 |
|
Hmmmmm ... now I can't get either one to pass 😡 |
|
Had a chance to test on Linux (Ubuntu 16.04) - off of this branch, both tests passed for me (off of fuzzy-finder master, the first test passed and the second test failed). |
|
Tested this with the help of @50Wliu. Confirmed on Mac OS X 10.13.3 using the APFS file system, both tests fail. According to our research, APFS does not do Unicode file name normalization at all. |
|
@gjtorikian Can you confirm which OS version and filesystem you tested this on? |
|
Another helpful article on APFS file system normalization. |
|
I'm also on 10.13.3 and APFS. |
ghost
reviewed
May 7, 2020
ghost
reviewed
May 7, 2020
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Requirements
Description of the Change
Background: different Unicode sequences can be regarded as equivalent. On Windows and Linux, filenames are converted to NFC, or where characters like
ñare coded as one character. On macOS, filenames are converted to NFD, whereñis coded asn+◌̃. Both forms are considered to refer to the same character.Fuzzy Finder was not taking Unicode normalization into account when searching for files. With this PR, NFD normalization for the filter query is now performed on macOS, while NFC normalization is performed for other platforms. This should allow for proper searching of filenames containing unicode characters.
Alternate Designs
NFKD and NFKC could be used instead, which would mean in addition the above,
ffwould also be treated as identical toff.Benefits
Fixes edge cases when searching for filenames with unicode characters.
Possible Drawbacks
None.
Applicable Issues
Fixes #69.
/cc @gjtorikian I think this should fix the issue you're seeing. If you'd like to test this PR but don't know how, I can provide guidance.