Skip to content

Latest commit

 

History

History
44 lines (31 loc) · 1.48 KB

lex.md

File metadata and controls

44 lines (31 loc) · 1.48 KB

Lex

Table of contents

Overview

Lexing converts input source code into tokenized output. Literals, such as string literals, have their value parsed and form a single token at this stage.

Bracket matching

The lexer handles matching for (), [], and {}. When a bracket lacks a match, it will insert a "recovery" token to produce a match. As a consequence, the lexer's output should always have matched brackets, even with invalid code.

While bracket matching could use hints such as contextual clues from indentation, that is not yet implemented.

Alternatives considered

Bracket matching in parser

Bracket matching could have also been implemented in the parser, with some awareness of parse state. However, that would shift some of the complexity of recovery in other error situations, such as where the parser searches for the next comma in a list. That needs to skip over bracketed ranges. We don't think the trade-offs would yield a net benefit, so any change in this direction would need to show concrete improvement, for example better diagnostics for common issues.