[RFC] Add a tokenizer as intermediate step

[torkleyy]

Description

Add a tokenizer which emits Tokens given a byte sequence.

Those tokens will then be feed into the deserializer.

Motivation

This allows to simplify the logic in the deserializer, makes testing easier and is more likely to avoid bugs.

Additionally, we could refactor the Serializer to emit tokens, too (at a second step). If we do that, we might be able to address #175 by formatting the tokens; the pretty config would be mostly irrelevant for the serialization.

Aaand in a third step, we could even utilize this architecture to implement a formatter, simply by tokenizing an input file and pretty-formatting the tokens.

[cynecx]

Instead of implementing a new lexer, isn't it possible to actually use rust's lexer? (https://crates.io/crates/).

[CAD97]

Specifically, rustc's lexer is now available as rustc_lexer at https://crates.io/crates/rustc_lexer and while it explicitly has no API guarantee other than adhering to semver, it is highly desired to have the crate compile on stable (with no intention of breaking that status). Since RON explicitly wants to "look like Rust", using the standard Rust lexer makes sense.

The other option is to use proc_macro2 (and maybe syn?) to parse in all cases, which gives the ron! macro for "free" (as a proc-macro).

[CAD97]

I have way too much experience writing syn-based parsers.

Here's a parser for the RON syntax using syn/proc_macro2: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=18dc90ed4b5244507cb6660fd2f39de7

Feel free to use it. Also, double check me, I could've missed something.

(EDIT: Added ToTokens implementations)

[github-actions]

Issue has had no activity in the last 180 days and is going to be closed in 7 days if no further activity occurs

[torkleyy]

I think we can leave this closed, it seems outside the scope of this repository and I've started something like this in ron-rs/ron-reboot