New \p{Letter} Unicode property escape

[bhamiltoncx]

@mike-lischke asked for this in his review of #1633 . I think it's a great way to show the power of the new full Unicode functionality in ANTLR4.

This PR adds two new lexer escapes suitable for use in a charset, so:

[a-z]

could become:

[\p{Ll}]

or, equivalently:

[\p{Lowercase_Letter}]

to match both a-z as well as exciting Unicode code points like 𝐚 (U+1D41A) through 𝐳 (U+1D433).

I included both matching and non-matching variants:

1. \p{Letter}: Include all Unicode code points with the general category "Letter"
2. \P{Letter}: Include all Unicode code points which do not have the general category "Letter"

This also works for:

1. Scripts (\p{Latin}, \p{Hiragana}, \p{Cyrillic})
2. Binary properties (\p{Emoji}, \p{Changes_When_Uppercased}, \p{Quotation_Mark})
3. Blocks prefixed with "In" (since their names collide with scripts): (\p{InHiragana}, \p{InArabic_Ext_A}, \p{InGreek})

The names of properties are case-insensitive. In addition, - and _ are treated the same.

[mike-lischke]

I guess you followed http://www.regular-expressions.info/unicode.html with your names. Do you also support short forms (like \p{Ll})? And what about the Unicode block syntax (\p{InXXX}?

[bhamiltoncx]

Yes, short and long forms are supported. I haven't done Unicode blocks yet, but that'd be easy to add.

…

On Thu, Feb 23, 2017 at 3:10 AM Mike Lischke ***@***.***> wrote: I guess you followed http://www.regular-expressions.info/unicode.html with your names. Do you also support short forms (like \p{Ll})? And what about the Unicode block syntax (\p{InXXX}? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1688 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AApAU5NqIdLxKPp1Em-nkXWj8M1zbzPtks5rfWkxgaJpZM4MJWlJ> .

[bhamiltoncx]

OK, added support for Unicode blocks with names prefixed with In.

Since block and script names overlap, \p{InGreek} means the (basic) Greek block, but \p{Greek} means the entire Greek script, which includes multiple blocks like \p{InGreek_Ext}.

[bhamiltoncx]

Split off two PRs which this is stacked on:

#1692
#1693

[bhamiltoncx]

OK, I refactored everything and this should be good to go once the two dependent PRs (#1692 and #1693) land.

I'll rebase it once those two land and update the PR.

[bhamiltoncx]

OK, rebased and ready for review! (Hopefully all the tests pass.)

[parrt]

boom, jack!

[bhamiltoncx]

Awesomesauce.