Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(unicode) detect supplementary private use area A #2102

Merged
merged 5 commits into from
Mar 19, 2020
Merged

fix(unicode) detect supplementary private use area A #2102

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
647463f
Detect unicode in the Supplementary Private Use Area-A
ahuth Mar 19, 2020
20c5df8
Use surrogate pair representation for unicode sequence
ahuth Mar 19, 2020
30aef40
Move detection of certain private use unicode characters to `getSuppl…
ahuth Mar 19, 2020
0e3777a
Replace unicode escape sequences in regex with the corresponding surr…
ahuth Mar 19, 2020
6a90379
Document which parts of the `getSupplementaryPrivateUseRegExp` regex …
ahuth Mar 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions lib/commons/text/unicode.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,10 @@ text.hasUnicode = function hasUnicode(str, options) {
return axe.imports.emojiRegexText().test(str);
}
if (nonBmp) {
return getUnicodeNonBmpRegExp().test(str);
return (
getUnicodeNonBmpRegExp().test(str) ||
getSupplementaryPrivateUseRegExp().test(str)
);
}
if (punctuations) {
return getPunctuationRegExp().test(str);
Expand Down Expand Up @@ -127,9 +130,11 @@ function getPunctuationRegExp() {
* @returns {RegExp}
*/
function getSupplementaryPrivateUseRegExp() {
/**
* Reference: https://www.unicode.org/charts/PDF/UD800.pdf
* https://www.unicode.org/charts/PDF/UDC00.pdf
*/
return /[\uDB80-\uDBBF][\uDC00-\uDFFD]/g;
// 1. High surrogate area (https://www.unicode.org/charts/PDF/UD800.pdf)
// 2. Low surrogate area (https://www.unicode.org/charts/PDF/UDC00.pdf)
// 3. Supplementary private use area A (https://www.unicode.org/charts/PDF/UF0000.pdf)
//
// 1 2 3
// ┏━━━━━━┻━━━━━━┓┏━━━━━━┻━━━━━━┓ ┏━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
Copy link
Contributor Author

@ahuth ahuth Mar 19, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got a little clever documenting which parts of the regex refer to which unicode sequence. Say the word and I'll remove this.

return /[\uDB80-\uDBBF][\uDC00-\uDFFD]|(?:[\uDB80-\uDBBE][\uDC00-\uDFFF]|\uDBBF[\uDC00-\uDFFD])/g;
}
14 changes: 14 additions & 0 deletions test/commons/text/unicode.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,13 @@ describe('text.hasUnicode', function() {
});
assert.isTrue(actual);
});

it('returns true for a string with characters in supplementary private use area A', function() {
var actual = axe.commons.text.hasUnicode('\uDB80\uDC19', {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The string here is the same as the unicode escape sequence '\u{F0019}'. However, ES5 doesn't support those, and the tests are parsed by eslint as ES5. Instead this is the corresponding surrogate pair.

nonBmp: true
});
assert.isTrue(actual);
});
});

describe('text.hasUnicode, characters of type Emoji', function() {
Expand Down Expand Up @@ -201,6 +208,13 @@ describe('text.removeUnicode', function() {
assert.equal(actual, '');
});

it('returns the string with supplementary private use area A characters removed', function() {
var actual = axe.commons.text.removeUnicode('\uDB80\uDC19', {
nonBmp: true
});
assert.equal(actual, '');
});

it('returns string removing combination of unicode characters', function() {
var actual = axe.commons.text.removeUnicode(
'The ☀️ is orange, the ◓ is white.',
Expand Down