WIP chore: drop twing for twig-lexer

[Kocal]

@ericmorand have created the perfect package to lex Twig code: twig-lexer

When using Twing, we had some issues about whitespace control modifier ({%-, {{-, ...) because they were simply removed from tokens and nodes (see ericmorand/twing/issues/321 and ericmorand/twing/pull/326).

But with twig-lexer everything is preserved as tokens, example:

And so it means that we will be able to print those modifiers back :)

Actually I will have to rewrite everything since we will use tokens instead of nodes to prettify Twig code. In fact not everything, but we have to make sure we are using an AST and not only tokens (even with a token stream), otherwise it will be a pain in the ass.
Since we won't directly use tokens for an AST, we need to find a way to reserve whitespace control modifiers. That can be weird, but I think we can store (on certain types of node) the opening/closing tags with whitespace modifier controls. 🤔

And also use FastPath instead of our own print function because of (prettier/plugin-php#12):

• comment handling around every single node
• access to the parents all the way to the root
• It also allows the multiparser feature to exist

This PR will close #15, close #4 and close #13

[ericmorand]

Glad to see that this will help you. It will be the lexer used by the next major version of Twing.

Just a remark about the "WHITESPACE" tokens. They may seem redundant - after all they can easily be resolved by simple comparison of value and line/column properties of the tokens - but I felt they would make life easier for tools like yours - less code from your side is always better.

The drawback is that they have to be ignored by eventual parsers (like Twing parser) but this is trivial.

Another remark : I see that you implemented a token stream which is very good. I have been torned between having one included in twig-lexer or not. Actually, Twing will also need one so maybe it could be beneficial to have it included in twig-lexer and tested there instead of having it spread in multiple projects. It would come as a separate module that export a constructor that takes an array of tokens as parameter. With tree shaking, it would add zero byte to consumer packages if not used.

[Kocal]

Don't worry for the WHITESPACE tokens, in fact I think I will skip everyone of them since it's Prettier that will decide how to render whitespaces (see https://github.com/Kocal/prettier-plugin-twig/blob/master/src/printer.js#L59 and https://github.com/prettier/prettier/blob/master/commands.md#line).

For the token stream, it would be really nice since the one I wrote is like 80% copy/pasted from Twing. 😅

[ericmorand]

It's a deal. I'll add it to twig-lexer. And loot your test suite if you don't mind.

[Kocal]

Yeah go for it! :)

[ericmorand]

@Kocal there is some breaking changes in twig-lexer@0.5. Namely, the getters are now using the native syntax - fro exemplae Token::getValue has been replaced by Token::value. I want to take the opportunity to have the best code possible while we are still below 1.0 - but 1.0 should be there in no time, I feel like the API is quite complete now.

Also the token stream is now included in the library. Thanks for your work on it.

[Kocal]

Thanks for the info @ericmorand, I will try it when I will have some time :)