Trimming TEXT token values while lexing

[PolyPik]

In TwigPHP, the use of whitespace control modifiers on statements can affect the value of the preceding or following TEXT_TYPE token depending on where the modifier was used.

Should this lexer implementation do the same and trim the preceding or following TEXT Token value when it encounters a statement with whitespace control modifiers?

[ericmorand]

I don't think it should. It would make the lexer lossy.

I'll have something for you (this issue and #10) in the next few days. Certainly in the form of a token stream (see Kocal/prettier-plugin-twig#17 (comment)) that will be easily filterable and come with a toAst function that will return a token list that only contains the token needed for the final output - so without WHITESPACE or COMMENT or trimming tokens and with the TEXT tokens trimmed accordingly.

It is already done and mainly tested (thanks to @Kocal), I just have to move it there.

[PolyPik]

Nice to hear that. However is this token stream's toAst function going to turn TEST_OPERATOR tokens into OPERATOR tokens as well? TEST_OPERATOR has no equivalent in TwigPHP.

[PolyPik]

@ericmorand I'm waiting for the changes on your part.

[ericmorand]

The token stream will be deployed by the end of the day.

Note that the TEST_OPERATOR token will remain. I really think that the is operator is not a traditional operator. It is actually a language structure that instanciate a test. It can't be customized or replaced without breaking the language, and it's impossible to create a custom operator that would mimic its behavior.

This is backed up by the fact that in both TwigPHP and Twing, there is a test at one point in the parser that goes "if the token is an operator > if its value is is then let's do something special; else let's instanciate an operator". Clearly, would is be an operator, it would fall into the same case as all other operators and would not need a special treatment.

It should be trivial to replace TEST_OPERATOR tokens by OPERATOR tokens in the stream, would you need it.

[PolyPik]

Got it.