Fix unicode escapes in jsx identifiers and extended unicode characters in jsdoc

[weswigham]

Which are the two other places we used isIdentifierStart in the scanner.

Fixes #32701

[ajafff]

is it intentional not to allow unicode escape sequences in JSDoc identifiers?

[weswigham]

I believe so, but I dunno. Given that it's a comment, it just means that we'll interpret the \u0061 verbatim. @sandersn would be the authority here.

[sandersn]

I'm confused. Doesn't this change cause us to allow escape sequences in JSDoc identifiers?

jsdoc.app doesn't mention unicode as far as google can tell, so we are free to do whatever we want. I vote that JSDoc identifiers work the same as normal ones, which should mean allowing unicode escape sequences.

[sandersn]

I am confused after reading @ajafff's question. How does this change affect jsdoc? The tests make it seems like unicode identifiers (and escapes?) now work, but his question implies that it doesn't.

[sandersn]

stupid question: why isn't charSize(c) === 2 when c >= 0x10000 instead of c > 0x10000?

[sandersn]

I'm confused. Doesn't this change cause us to allow escape sequences in JSDoc identifiers?

jsdoc.app doesn't mention unicode as far as google can tell, so we are free to do whatever we want. I vote that JSDoc identifiers work the same as normal ones, which should mean allowing unicode escape sequences.

[weswigham]

he tests make it seems like unicode identifiers (and escapes?) now work, but his question implies that it doesn't.

Unicode characters now scan as identifiers in jsdoc, escapes are still not parsed.

[weswigham]

Doesn't this change cause us to allow escape sequences in JSDoc identifiers?

No.

[sandersn]

That makes sense. It would be nice to have JSDoc support escapes as well, but I think the right way to do that is to make JSDoc use the normal scanner. That's a complete rewrite of the low-level parsing code too, so I'm not sure it's worth it to us.

[weswigham]

@sandersn Oh, too late. I already pushed a commit that adds unicode escape support into jsdoc.

[orta]

I'm pretty sure this deprecates #32631 - can someone double check me?

[weswigham]

I don't think so - is jsdoc, at least, we're still going to stop parsing before the hyphen.

[HiEv]

I don't think so - is jsdoc, at least, we're still going to stop parsing before the hyphen.

I'm a bit confused by this reply. In JSDoc you shouldn't stop parsing before the hyphen if there's no space. The JSDoc documentation says that it should be able to handle dashes in its member names. For example:

/**
 * @typedef {object} node
 * @property {string} aria-activedescendant - Description.
 */
/** @type {node} */
var node;

That does work in JSDoc from the commandline, but not in VSCode. (See the comments here.)

In VSCode, if you hover the mouse over the "node" on the "@typedef" line, it should show:

type node = {
	aria-activedescendant: string;
}

However, instead you get:

type node = {
	aria: string;
}

with the "-activedescendant" part missing.

If you put quotes around "aria-activedescendant", then you get this instead:

type node = {
	(Missing): string;
}

#32631 was supposed to fix that problem (#14395).