Lex until division

[pvdz]

Magic brain fart;

If we're only interested in a token stream and our main problem for it is the division/regex choice, couldn't we postpone this at least until we actually encounter a forward slash? Would it, at this point, at least be possible to detect whether it should parse a regex or division?

This may require some backtracking. But it might be worth that overhead over much simplified parsing. You'd go a bit backwards, but hey.

Essentially you would do simple lexer parsing until you encounter a division. At that moment you skip back on the tokens, ignoring the actual values unless you have to know (puncs). Hm no I guess you would still need to know more about the environment to resolve reserved identifier situations properly. Hmmm. But what if you don't bother with that? What if you're fine with an over-accepting parser, or if you can postpone those checks to a second phase? Could that be worth it? Are there other reasons to properly parse?

Worst case, backtracking means going back on a very long expression. Very worst case, that's from the end of the Program to the start of it. Though that'd be more of an edge case and unlikely to cover a big Program anyways.

So when do you know for sure whether or not the next slash ought to be a division?

And what about ASI? Don't we also need to have more semantics of our surroundings? Especially to properly detect divisions because they can heavily depend on ASI.

Okay, let's freeze this brain fart for a bit.