SystemVerilog Preprocessor

[fangism]

Today, Verible's parser reads unpreprocessed source files because it was originally designed for single-file operations. The limitations of this approach are understood and accepted.
If there is interest in working toward a proper compiler toolchain, the very first thing we'd need is a preprocessor, to handle `includes and expand `defined macros, etc.

It is known that Verilog preprocessing is sophisticated: macro definitions may contain preprocessing directives that should be interpreted as they are evaluated and expanded. A classic paper on this subject can be found here: https://www.veripool.org/papers/Preproc_Good_Evil_SNUGBos10_paper.pdf

Test suite:
The most important first step is amassing a test suite solely around preprocessing correctness. A comprehensive, implementation-independent, (and open-source) test suite would set the goal post (for all projects). For example, a SystemVerilog parsing test suite is being collected here
https://github.com/SymbiFlow/sv-tests .

Design considerations:
Can start with a standalone tool, but would really like something that can be integrated, for performance benefits and serializability.

Pseudo-preprocessing, or limited preprocessing with partial information.
This would help with single-source file operations, for example by evaluating locally defined macros, and helping the parser see inside macros where possible.

Ping this issue if you are interested in discussing or helping.