antlr's lack of unicode is disturbing

[emberian]

It's impossible to correctly form XID_start and XID_continue in antlr, so for now the reference lexer just ignores unicode entirely.

[emberian]

In particular, it only knows about UCS2. You need to encode characters outside of the BMP using surrogates. However, the range syntax ('a' .. 'b') only accepts one character in each string, and encoding the surrogates requires two.

[emberian]

Note that this is also going to affect spans, since spans are in bytes but antlr gives them to us in characters. We can possibly correct this by translating the BytePos to CharPos.

[fhahn]

I am currently looking into this. I've added two additional files which contain the definitions of XID_Start and XID_Continue, like @jbclements used in https://github.com/jbclements/rust-antlr.

To generate the rules, I used noidejs, because https://github.com/mathiasbynens/unicode-4.0.0 provides all codepoints for XID_Start and XID_Continue as arrays (and I did not find something similar for Python)

But when I try to convert a BytePos of the spans to a CharPos, I get the following panic

'assertion failed: bpos.to_uint() >= mbc.pos.to_uint() + mbc.bytes', /home/flo/projects/rust/rust/src/libsyntax/codemap.rs:487

Here is the Rust code I use to convert them: fhahn@e85d830#diff-cc371bf2fbfcbeb87f125d3c45fd8fc3R237

I would really appreciate any hints what could be wrong.

Note that at the moment I did only include symbols up to \uFFFF, but according to antlr/antlr4#276 surrogates could be specified like

'\uD812\uDC34'..'\uD813\uDC56'

becomes

( '\uD812' '\uDC34'..'\uDFFF'
| '\uD813' '\uDC00'..'\uDC56'
)

[pczarn]

@fhahn, I'm going to send you a patch with other symbols. But I have an issue where antlr sees 2 surrogates, but Rust's span counts them as one.