Source file lookup, macros (and UVM)

[suzizecat]

Hi there !

I am struggling at performing sourcefile lookup when macro are involved.
More precisely, working in the context of a language server, I want to improve my symbol detection and definition/references lookup.

The final objective is to be able to lookup a symbol and all its references from a file path and a range.
This was fairly straightforward while only synthetizable RTL was involved : I perform an elaboration, visit what I believe is the CST (the Symbol/Scopes tree) in order to detect all symbols as declarations then go through the AST to detect all kinds of syntax nodes that would be a reference and lookup the symbol based upon the context.

This was done using some tricks but works fairly well.
Problems arise when trying to include UVM in the equation.

The main issue is that UVM uses and abuses of macros, and macros breaks the equivalence between a buffer and the original file. Hence, instead of having a syntax node like MacroUsageSyntax (or whatever) the Macro is expanded and it results in a mess.

The solution I am envisaging at the moment is something like:

1. Reading the AST of all files (maybe individually) before elaborating (in hope that Macros are not yet expanded).
2. Trying to determine all SyntaxNode that might later be a reference to a symbol (while taking into account the fact that I need to determine the scope in which this node will be for future lookup)
3. Elaborating the design
4. Visiting the CST and, depending on the scope, looking up the reference to attach to this symbol, and then using the symbol to lookup the actual definition.
5. Match everything and create a "Position -> Reference" table that allows me to look-up anything "by position".

This approach seems rather tedious, in particular "detecting" the scope in step 2, but I am not aware of another solution with slang.

Could you tell me if slang have some mechanisms that would ease this or should I dive in the hard way ?

[MikePopoloski]

It's not exactly clear what you're trying to achieve here; some guesses based on what you've said:

1. You want to be able to "go to definition" on a macro usage itself? So you place your cursor on a macro usage and it takes you to the original macro definition. This should not be particularly hard with slang as it is currently.
2. You want to be able to "go to definition" where the definition is inside a macro somewhere? This should also be fairly straightforward with slang right now.
3. You want to be able to find all possible references given any possible combination of macros being defined or not defined? This is somewhere between difficult and impossible, since the answer actually depends on which macros the user defines or not.

Some background to help discussion:

• The "CST" or "Concrete Syntax Tree" is the tree formed of SyntaxNodes. It's "concrete" in the sense that it represents exactly what the user typed.
• The "AST" or "Abstract Syntax Tree" is the tree formed by Symbols, Scopes, Expressions, Statements, etc. It's abstract since it abstracts away details across different ways of typing the code.
• The CST / SyntaxTree stuff is all after preprocessing and parsing. You can't parse code without first doing preprocessing, which is why there is no "MacroUsageSyntax". Code can have wildly different meanings and structure depending on preprocessing.

[suzizecat]

Thanks for your clarifications !

I hadn't though of the point you expose as 2. However that would be a nice addition.
So, given this kind of code extract:

`define MY_MACRO(x) (x+1)
...
int foo;
`MY_MACRO(foo);

I would want to be able to find:

1. MY_MACRO declaration (the define line) when placing my cursor on MY_MACRO from MY_MACRO(foo)
2. All usage of MY_MACRO when placing my cursor on MY_MACRO from the define line.
3. Find the usage of foo as argument of MY_MACRO (on the MY_MACRO usage) from the foo declaration.
4. Find the declaration of foo when having my cursor on foo from MY_MACRO(foo).

Optionally, having the x references/declaration from the macro definition would be nice too.

Few important points:

1. It is very important to get all references, while not having too much reference, as this is used as a starting point for a "Rename Symbol" function.
2. It is equally important to retrieve the exact source range in the original source file, for the same feature. (The Rename symbol function takes a list of source ranges and the new value and replace everything).

With that said, one of the points for which I struggle the most is the fact that, in the CST, the Macros has already been expanded, and includes has been processed. Therefore the buffer does not match the original source file.
UVM shows this very well because the full content of uvm_pkg is included in a single 10-lines file (uvm_pkg.sv).
Looking up SyntaxNodes in this packages will return stuff from uvm_pkg:956 while it is actually in an included file.

I found getFullyOriginalLoc that helps quite a bit, but the arguments of a macro seems to return a char number of 0 (while the line is OK)

[suzizecat]

After looking at a (very) simple testcase, I might be starting to understand why I had such trouble.
For context, I made a tool that prints the CST in a structured(-ish) fashion and the result for this testcase:

// Test comment
`define FOO(x) x+1

class bar;
    function new()
        int i = 0;
        `FOO(i);
    endfunction;
endclass

Is the following stuff:

Hidden CST, as fairly long

Element	Meaning
>Foo	A SyntaxNode of kind Foo which has children
-Foo	A SyntaxNode of kind Foo which has no child
~Foo	A Trivia of kind Foo

>CompilationUnit
| >SyntaxList
| | >ClassDeclaration
| | | >SyntaxList
| | | -ClassKeyword...................................................: class
| | | | ~Directive
| | | | >DefineDirective
| | | | | -Directive..................................................: `define
| | | | | | ~LineComment
| | | | | | ~EndOfLine
| | | | | -Identifier.................................................: FOO
| | | | | | ~Whitespace
| | | | | >MacroFormalArgumentList
| | | | | | -OpenParenthesis..........................................: (
| | | | | | >SeparatedList
| | | | | | | >MacroFormalArgument
| | | | | | | | -Identifier...........................................: x
| | | | | | -CloseParenthesis.........................................: )
| | | | | >TokenList
| | | | | | -Identifier...............................................: x
| | | | | | | ~Whitespace
| | | | | | -Plus.....................................................: +
| | | | | | -IntegerLiteral...........................................: 1
| | | | ~EndOfLine
| | | | ~EndOfLine
| | | -Identifier.....................................................: bar
| | | | ~Whitespace
| | | -Semicolon......................................................: ;
| | | >SyntaxList
| | | | >ClassMethodDeclaration
| | | | | >SyntaxList
| | | | | >TokenList
| | | | | >FunctionDeclaration
| | | | | | >SyntaxList
| | | | | | >FunctionPrototype
| | | | | | | -FunctionKeyword........................................: function
| | | | | | | | ~EndOfLine
| | | | | | | | ~Whitespace
| | | | | | | >SyntaxList
| | | | | | | >ImplicitType
| | | | | | | | >SyntaxList
| | | | | | | | -Placeholder
| | | | | | | >ConstructorName
| | | | | | | | -NewKeyword...........................................: new
| | | | | | | | | ~Whitespace
| | | | | | | >FunctionPortList
| | | | | | | | -OpenParenthesis......................................: (
| | | | | | | | >SeparatedList
| | | | | | | | -CloseParenthesis.....................................: )
| | | | | | >SyntaxList
| | | | | | | >DataDeclaration
| | | | | | | | >SyntaxList
| | | | | | | | >TokenList
| | | | | | | | >IntType
| | | | | | | | | -IntKeyword.........................................: int
| | | | | | | | | | ~EndOfLine
| | | | | | | | | | ~Whitespace
| | | | | | | | | >SyntaxList
| | | | | | | | >SeparatedList
| | | | | | | | | >Declarator
| | | | | | | | | | -Identifier.......................................: i
| | | | | | | | | | | ~Whitespace
| | | | | | | | | | >SyntaxList
| | | | | | | | | | >EqualsValueClause
| | | | | | | | | | | -Equals.........................................: =
| | | | | | | | | | | | ~Whitespace
| | | | | | | | | | | >IntegerLiteralExpression
| | | | | | | | | | | | -IntegerLiteral...............................: 0
| | | | | | | | | | | | | ~Whitespace
| | | | | | | | -Semicolon............................................: ;
| | | | | | | >ExpressionStatement
| | | | | | | | >SyntaxList
| | | | | | | | >AddExpression
| | | | | | | | | >IdentifierName
| | | | | | | | | | -Identifier.......................................: i
| | | | | | | | | | | ~Directive
| | | | | | | | | | | >MacroUsage
| | | | | | | | | | | | -Directive....................................: `FOO
| | | | | | | | | | | | | ~EndOfLine
| | | | | | | | | | | | | ~Whitespace
| | | | | | | | | | | | >MacroActualArgumentList
| | | | | | | | | | | | | -OpenParenthesis............................: (
| | | | | | | | | | | | | >SeparatedList
| | | | | | | | | | | | | | >MacroActualArgument
| | | | | | | | | | | | | | | >TokenList
| | | | | | | | | | | | | | | | -Identifier...........................: i
| | | | | | | | | | | | | -CloseParenthesis...........................: )
| | | | | | | | | -Plus...............................................: +
| | | | | | | | | >SyntaxList
| | | | | | | | | >IntegerLiteralExpression
| | | | | | | | | | -IntegerLiteral...................................: 1
| | | | | | | | -Semicolon............................................: ;
| | | | | | -EndFunctionKeyword.......................................: endfunction
| | | | | | | ~EndOfLine
| | | | | | | ~Whitespace
| | | | >EmptyMember
| | | | | >SyntaxList
| | | | | >TokenList
| | | | | -Semicolon..................................................: ;
| | | -EndClassKeyword................................................: endclass
| | | | ~EndOfLine
| -EndOfFile

I can see that the DefineDirective SyntaxNode it attached to the class Keyword that should not have any child. Is that linked to the "Skipped tokens" thing that appear in the Trivia documentation ?

If so, would you have a method to reliably detect and position that macro in the context of scopes and such ?
My hunch is that macro declaration is "global to the file" regardless of where it is actually defined.
Therefore, the following code would be strictly equivalent to the first one.

// Test comment


class bar;
    function new()
// Define placed at a very weird place...
`define FOO(x) x+1
        int i = 0;
        `FOO(i);
    endfunction;
endclass

In this case, just using the SyntaxVisitor and handle the DefineDirectiveSyntax would probably do the trick.

Moreover, for the macro usage, the CST would mean that, post processing, there is something like

`FOO(i) // Actually is not really used, is a MacroUsageSyntax
i+1 // Is inserted and actually doing the job.

Which would mean that I need to lookup the MacroUsageSyntax and, after processing, ignore the content of MacroUsageSyntax::parent.

Finally, I found out that declaring `define FOO(x) 1+x instead of `define FOO(x) x+1
Produce a fairly different result upon using the macro. In particular, there is no more mention of FOO.

// When using x+1:
| | | | | | | >ExpressionStatement
| | | | | | | | >SyntaxList
| | | | | | | | >AddExpression
| | | | | | | | | >IdentifierName
| | | | | | | | | | -Identifier.......................................: i
| | | | | | | | | | | ~Directive
| | | | | | | | | | | >MacroUsage
| | | | | | | | | | | | -Directive....................................: `FOO
| | | | | | | | | | | | | ~EndOfLine
| | | | | | | | | | | | | ~Whitespace
| | | | | | | | | | | | >MacroActualArgumentList
| | | | | | | | | | | | | -OpenParenthesis............................: (
| | | | | | | | | | | | | >SeparatedList
| | | | | | | | | | | | | | >MacroActualArgument
| | | | | | | | | | | | | | | >TokenList
| | | | | | | | | | | | | | | | -Identifier...........................: i
| | | | | | | | | | | | | -CloseParenthesis...........................: )
| | | | | | | | | -Plus...............................................: +
| | | | | | | | | >SyntaxList
| | | | | | | | | >IntegerLiteralExpression
| | | | | | | | | | -IntegerLiteral...................................: 1
| | | | | | | | -Semicolon............................................: ;

// When using 1+x

| | | | | | | >ExpressionStatement
| | | | | | | | >SyntaxList
| | | | | | | | >IdentifierName
| | | | | | | | | -Identifier.........................................: i
| | | | | | | | | | ~SkippedTokens
| | | | | | | | | | -IntegerLiteral...................................: 1
| | | | | | | | | | -Plus.............................................: +
| | | | | | | | -Semicolon............................................: ;