Switch rustdoc from `lexer::StringReader` to rustc_lexer

[matklad]

In #75528, we would like to tweak the structure of the tokens rustc uses. One of the components which heavily depends on the current structure is rustdoc (html and check_code_block_syntax module use lexer::StringReader).

I think, for this issue and in general, it makes sense to switch rustdoc over to rustc_lexer, which produces lower-level guaranteed lossless tokens, which look like a better fit for the task of syntax highlighting.

I plan to do this refactor myself, but I want to check with @rust-lang/rustdoc if this is a good idea.

EDIT: cc #63284

[Manishearth]

sgtm

[GuillaumeGomez]

I'm fine with it as well. :)

[matklad]

BTW, is there some chat room for rustdoc team on zulip on discord? I'd like to ask some questions :)

[GuillaumeGomez]

There is one on discord (simply called "rustdoc").

[jyn514]

It's linked at https://www.rust-lang.org/governance/teams/dev-tools#rustdoc (which is I agree pretty hard to find).

Also the text there is wrong and mentions docs.rs, we should probably fix that.