Hit a serious problem with Antlr

[Korporal]

I've hit a roadblock experimenting with a simplified PL/I grammar. The parser is interpreting an assignment statement as if it were a keyword statement.

The PL/I language will treat a statement as an assignment if it satisfies the grammar of an assignment, only if it does not will it then attempt a keyword based parse.

So, is there a way in Antlr to force that? given this parser rule:

statement
    :   assign_stmt 
    |   keyword_stmt 
    ;

If the text satisfies the first rule, can we not simply leave it at that? can we forcibly prioritize the two rules?

Here's my draft grammar:

grammar nores;

/*
options
{
backtrack=true;
}
*/

prog: statement*  
    ;

statement
    :   assign_stmt 
    |   keyword_stmt 
    ;

assign_stmt
    :   reference '=' expression NEWLINE
    ;

reference
    :   reference ARROW basic_reference arguments_list?
    |   basic_reference arguments_list?
    ;

arguments
    :   LPAR subscript_commalist? RPAR
    ;

arguments_list
    :   arguments+
    ;

basic_reference
    :   structure_qualification_list? IDENTIFIER 
    ;

structure_qualification
    :   IDENTIFIER arguments? DOT
    ;

structure_qualification_list
    :   structure_qualification+
    ;

subscript
    :   expression 
    ;

subscript_commalist
    :   subscript (COMMA subscript)*
    ;

    /*
unsubscripted_reference
    :   (unsubscripted_reference DOT)? IDENTIFIER
    ;
    */

keyword_stmt
    :   GOTO IDENTIFIER NEWLINE
    |   CALL IDENTIFIER NEWLINE
    ;

expression
    :   expression ('*'|'/') expression
	|   expression ('+'|'-') expression
	|   INT
	|   '(' expression ')'
	;

WS:         (' ')+ -> skip ;
NEWLINE:    [\r\n]+ ;
INT:        [0-9]+ ; 
GOTO:       'goto' ;
CALL:       'call' ;
IDENTIFIER: [a-zA-Z]+ ;
ARROW:      '->' ;
DOT:        '.' ;
COMMA:      ',' ;
LPAR:       '(' ;
RPAR:       ')' ;

and here is the input used to test in grun

ball = 1234
call = 1234

They are both assignments according to the grammar, the second line satisfies the assignment grammar and does not satisfy the call keyword grammar so why does the parser complain? the text obeys the grammar!

[Korporal]

OK I have more insights if I gen C# and debug the parser.

The issue is that when it sees the call text it is not seeing it as an identifier, it is a token of type 'CALL' or something, as specified in the lexer portion of the g4 file.

If I remove that token and just define the statement as 'call' IDENTIFIER NEWLINE that also fails, again, because the token it sees for 'call' is not an IDENTIFIER...

[Korporal]

This is the problem. I need a way to tell the parser that if it sees IDENTIFIER and that identifier is the text "call" then parse a call_stmt.

Right now I just have this:

keyword_stmt
    :   'call' IDENTIFIER NEWLINE
    |   'goto' IDENTIFIER NEWLINE
    ;

And it does not see 'call' as being an identifier so always parses a statement starting with 'call' as being a keyword statement...

[ericvergnaud]

you can achieve this using semantic predicates

…

Le 15 déc. 2022 à 15:26, Hugh Gleaves ***@***.***> a écrit : This is the problem. I need a way to tell the parser that if it sees IDENTIFIER and that identifier is the text "call" then parse a call_stmt. — Reply to this email directly, view it on GitHub <#4018 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZNQJC6ISEUDS3AX5ECPSLWNMS7XANCNFSM6AAAAAAS65GUAY>. You are receiving this because you are subscribed to this thread.

[Korporal]

@ericvergnaud - Thanks Eric, I've looked for that recently but struggling a little. I did get someway toward a resolution, this grammar recognizes things now:

grammar nores;


prog: statement*  
    ;

statement
    :   assign_stmt 
    |   keyword_stmt 
    ;

assign_stmt
    :   reference '=' expression NEWLINE
    ;

reference
    :   reference ARROW basic_reference arguments_list?
    |   basic_reference arguments_list?
    ;

arguments
    :   LPAR subscript_commalist? RPAR
    ;

arguments_list
    :   arguments+
    ;

basic_reference
    :   structure_qualification_list? (CALL | GOTO | IDENTIFIER) 
    ;

structure_qualification
    :   (CALL | GOTO | IDENTIFIER) arguments? DOT
    ;

structure_qualification_list
    :   structure_qualification+
    ;

subscript
    :   expression 
    ;

subscript_commalist
    :   subscript (COMMA subscript)*
    ;

keyword_stmt
    :   call_stmt 
    |   goto_stmt
    ;

call_stmt
    :   CALL IDENTIFIER '(' ')' NEWLINE
    ;

goto_stmt
    :   GOTO IDENTIFIER NEWLINE
    ;

expression
    :   expression ('*'|'/') expression
    |   expression ('+'|'-') expression
    |   INT
    |   '(' expression ')'
    ;

WS:         (' ')+ -> skip ;
NEWLINE:    [\r\n]+ ;
INT:        [0-9]+ ; 
CALL:       'call' ; 
GOTO:       'goto' ;
IDENTIFIER: [a-zA-Z]+ ;
ARROW:      '->' ;
DOT:        '.' ;
COMMA:      ',' ;
LPAR:       '(' ;
RPAR:       ')' ;

This input parses correctly in grun:

call = 1234
goto done
ball = 1234
call ball()
goto = 999

But you'll see stuff like this:

basic_reference
    :   structure_qualification_list? (CALL | GOTO | IDENTIFIER) 
    ;

Where I must call out the keywords individually.

[Korporal]

OK I solved this now, this seems to be the way to go:

grammar nores;


prog: statement*  
    ;

statement
    :   assign_stmt 
    |   keyword_stmt 
    ;

assign_stmt
    :   reference '=' expression NEWLINE
    ;

reference
    :   reference ARROW basic_reference arguments_list?
    |   basic_reference arguments_list?
    ;

arguments
    :   LPAR subscript_commalist? RPAR
    ;

arguments_list
    :   arguments+
    ;

basic_reference
    :   structure_qualification_list? (keyword | IDENTIFIER) 
    ;

structure_qualification
    :   (keyword | IDENTIFIER) arguments? DOT
    ;

structure_qualification_list
    :   structure_qualification+
    ;

subscript
    :   expression 
    ;

subscript_commalist
    :   subscript (COMMA subscript)*
    ;

keyword_stmt
    :   call_stmt 
    |   goto_stmt
    |   procedure_stmt
    ;

call_stmt
    :   CALL IDENTIFIER '(' ')' NEWLINE
    ;

goto_stmt
    :   GOTO IDENTIFIER NEWLINE
    ;

procedure_stmt
    :   PROCEDURE IDENTIFIER ('(' ')')? '{' '}'
    ;

expression
    :   expression ('*'|'/') expression
    |   expression ('+'|'-') expression
    |   INT
    |   '(' expression ')'
    ;

keyword
    : CALL
    | GOTO
    | PROCEDURE
    | PROC
    ;

WS:         (' ')+ -> skip ;
NEWLINE:    [\r\n]+ ;
INT:        [0-9]+ ; 
CALL:       'call' ; 
GOTO:       'goto' ;
PROCEDURE:  'procedure' ;
PROC:       'proc' ;
IDENTIFIER: [a-zA-Z]+ ;
ARROW:      '->' ;
DOT:        '.' ;
COMMA:      ',' ;
LPAR:       '(' ;
RPAR:       ')' ;

Earlier I define keyword lexically rather than grammatically, so defining it like:

keyword
    : CALL
    | GOTO
    | PROCEDURE
    | PROC
    ;

seems to give me what I need !

Thx

[kaby76]

Yes, this will work. This is the usual trick employed in many of the grammars in grammars-v4. An example of this "trick" is one of the Java grammars, where it's stated in the Spec what is possible for an identifier, and you can see in the Antlr grammar how it is implemented. Note, there is no --operator in Antlr. Your expression rule may need to allow identifiers, too.