Need to process win32 line break correctly

[3cp]

I noticed we skipped lots of test files in test262-parser-tests, I tried to unblock them to see what happens.

For example the first skipped file:

meriyah/test/test262-parser-tests/parser-tests.ts

Line 9 in 756d990

'110fa1efdd0868b8.js',

The 110fa1efdd0868b8.js file contents is

"Hello\
world"

But the file is using win32 line break \r\n which causes issue in meriyah.

[3cp]

It seems our code didn't handle LineContinuation inside string literal at all, the successful pass on \ + \n was an accident.

[KFlash]

strange. Can you do something like this

It's just skipped here, but according to the specs only CR and LF are invalid char in string literals now. For PS and LS I'm not 100% sure if the column value should be increased?

Similar to what you see here

[3cp]

I did not see you handle LineContinuation in that escaya code either.

[KFlash]

A LineContinuation is \r, so what's done is to pass all chars starting with \ to escaped chars. Then increase the index. If no "switch match" then return String.fromCharCode(ch); and you get your empty code point

[3cp]

Thx, so it's not an accident. Just a missing case on \r\n.

BTW, it seems test262-parser-tests are being maintained poorly, many unanswered issues.

[3cp]

The LineContinuation supports \n, \r, \r\n, and unicode LS and PS.

http://www.ecma-international.org/ecma-262/11.0/index.html#prod-LineTerminatorSequence

[KFlash]

test262-parser-tests is no good. Either try the real test suite or look into https://github.com/v8/v8/blob/master/test/cctest/test-parsing.cc for some edge cases etc

LS and PS are now allowed in string literals, but \n, \r, and \r\n are dissallowed.

If you read what I wrote in my last comment on how to grab the empty code point, it can be done the same for the sequence \r\n if you slice and append the string

\r - res += res + slice...
\n - res += res + slice...

The tricky part is for this cases \\r and \\n

[KFlash]

@3cp I made it just now. https://gist.github.com/KFlash/a8452b70d3596e0f79a2b8d9b3288047

[KFlash]

I'm trying to give some "super powers" to the gist I linked too so this can be solved once for all. Give me 30 min.

[KFlash]

Done! https://gist.github.com/KFlash/a8452b70d3596e0f79a2b8d9b3288047

Taken from the top of my head, so didn't run any tests against this, but it should work

[KFlash]

@3cp In my gist the cases should be ordered 0, 1, 2 , and you can index++ - skip the char - before the loop if you update the backslash consumer. I had no time to look into that.

[KFlash]

Updated the gist and added escape char scanning. I tweaked it, but you still need to "adjust". I think this should solve this issue

[3cp]

Any reason you write these?

export function scanString(parser: ParserState, cp: number): Token {
  parser.pos++;
  let quote = cp;

Instead of keeping existing param name quote?

export function scanString(parser: ParserState, quote: number): Token {
  parser.pos++;
  let cp = quote;

[KFlash]

No reason. I just tried to avoid assigning of the variable inside the loop. Costs memory. It was all done fast and without too much thinking.

[3cp]

It turns out a very small bug in existing code, it was sitting there because of missing test coverage.
Let's just patch it for now, worry about the bigger refactoring later.