ES6 regular expressions unicode mode: case insensitivity case folding inconsistency

[vsemozhetbyt]

• Version: 5.9.1
• Platform: Windows 64-bit

Here is a good article on the topic with this code example:

var es5regex = /\W/i;
var es6regex = /\W/iu;
console.log(
  es5regex.test('☃'),     es6regex.test('☃'),      // true true
  es5regex.test('S'),      es6regex.test('S'),      // false true
  es5regex.test('\u017F'), es6regex.test('\u017F'), // true true
  es5regex.test('K'),      es6regex.test('K'),      // false true
  es5regex.test('\u212A'), es6regex.test('\u212A')  // true true
);

(https://mathiasbynens.be/notes/es6-unicode-regex#impact-i)

The last Chrome Canary with V8 5.1.216 gives the same output.

However the 5.9.1 Node with --harmony-unicode-regexps and V8 4.6.85.31 gives this one:

true true
false false
true true
false false
true true

Is this temporary inconsistency while the 'u' flag is feature in progress?

[cjihrig]

Anything behind a harmony flag can't be trusted to be 100% implemented and correct.

[hashseed]

This is just how it is specified:

In [0], the character class escape 'W' is defined as the inverse of 'w', and 'w' is defined as exactly 63 characters, i.e [a-zA-Z0-9_].

With the ignore-case flag, we just add the case-equivalency closure over the characters of the character class. The way case-equivalency (canonicalize) is defined differs between unicode and non-unicode. See [1].

Now let's look at one of your examples:
es5regex.test('S') // false
Well obviously 'S' is not in \W. Nothing in \W converted to uppercase is 'S' either.

The unicode version though:
es6regex.test('S') // true
Now this is interesting. In unicode, we use the common case folding mapping provided by the unicode standard. See [2]. If you look for 'LATIN CAPITAL LETTER S', you will see that 0x0053 (uppercase 'S') maps to 0x0073 (lowercase 's'). If you look for other entries that also map to 0x0073, you will find

017F; C; 0073; # LATIN SMALL LETTER LONG S

C stands for 'common'. And since 0x017F is not in \w, it's part of \W. Using ignore-case means that both the regexp and the input is canonicalized (case-folded) upfront. So 0x17F as part of \W is folded to 's', just as well as 'S' in the input. Consequently, they match.

Internally, V8 creates a closure over case-equivalent characters for the regexp so that we don't have to canonicalize the input, but the result is the same.

The rest of your examples can be explained similarly.

This is not a bug in V8. It's just how unicode regexp is specified. You are welcome to bring it up to TC39 to get it fixed though, and I'll happily implement any change to the spec.

[0] https://tc39.github.io/ecma262/#sec-characterclassescape
[1] https://tc39.github.io/ecma262/#sec-runtime-semantics-canonicalize-ch
[2] http://www.unicode.org/Public/UNIDATA/CaseFolding.txt

[cjihrig]

Thanks for the explanation @hashseed. I'm going to close this as not a bug in Node or V8.

[vsemozhetbyt]

Then why do V8 5.1.216 and V8 4.6.85.31 give different results? Which one is right and which one is buggy?

[hashseed]

V8 4.6 does not have unicode regexp enabled by default. V8 5.1 does. V8 5.1 is also newer. Given my explanation above, why would you think V8 4.6 is correct?

[vsemozhetbyt]

I don't think V8 4.6 is correct, I thought V8 5.1 was. As the last Firefox with its engine.

I've just tried to find out if the current Node.js behavior would change according with current browswers behavior when unicode regexp get out from the flag. Sorry for my messy English and description (