Lexical Analyzer

The lexical analyzer translates a string that represents a mypy source file into a list of tokens.

You can use the mypy/lex.py module as a script to experiment with it (I assume that mypy is an alias to the mypy.py script):

$ cd mypy              # mypy repo
$ cat samples/hello.py
print('Hello, world')
$ mypy lex.py samples/hello.py
Name(print)
Punct(()
StrLit('Hello, world')
Punct())
Break(\n)
Eof()
$

The names Name, Punct etc. refer to different token classes defined in mypy/lex.py.

A list of tokens is not usually very useful as such, and it is typically immediately passed to the Python Parser.