How can I add a grammar-file for Java20 based on https://docs.oracle.com/javase/specs/jls/se20/jls20.pdf ?

[AtKnight]

I have made a grammar for Java20. How can I add the grammar-file to this repository?

[kaby76]

I suggest you clone the repo, create a branch for changes, add a directory java/java20/, and copy the pom.xml, desc.xml, and any base classes you need into the directory from one of the other java grammars. Test then commit and create a PR. Once in a PR we can review it.

Or, if you want, given the grammar, I can set up a directory, tar/gzip it, and hand it over to you as an initial version.

[AtKnight]

I made a clone of grammars-v4, added a directory java/java20, copied Java20.g4 to this directory.
I use make and not maven or git, so other files are not added.
Commit is given.
A PR (Pull Request) does not enable me to push/upload Java20.g4.

If you like, I can send a tar/gzip file with only one file.

[AtKnight]

I forgot: Of course a branche was created.

All activities are locally executed on my PC, working under Debian Linux.

[kaby76]

@AtKnight Sure, please attach the grammar in this thread by copying Java20.g4 to Java20.txt, and in the comment box where it says "Attach files by dragging & dropping, selecting or pasting them", click on that message and attach the .txt file. I will create a PR.

[AtKnight]

Java20.txt

Thank you!

Suggestions for improvements are welcome.

[KvanTTT]

Hi! Is it possible to merge Java 20 grammar into existing Java grammar? I don't see big sense to have multiple Java grammars for different versions especially considering the fact that newer grammars are back compatible.

What grammar does Java 20 grammar base on?

[AtKnight]

The grammar of Java 20 is based on the official specification: https://docs.oracle.com/javase/specs/jls/se20/jls20.pdf

It is not or hardly possible to merge the grammar with other grammars and still make a difference between the grammars.

[kaby76]

I think we should add the grammar separately to java20/, then try to merge the changes into the existing java/java/ grammar. I haven't analyzed the performance and correctness of this new grammar. It should be tested on the OpenJDK sources. To merge the changes into java/java/ will require a lot more time, more than I have at this moment.

But the main problem I have the concept of "the latest version of grammar X" in grammar-v4. We don't know what version java/java/ supports because people make single-feature changes. I tend to think of java/java/ should be more of a "scratchpad" grammar, where people test out refactorings. Because people are pretty much free to make a single change at a time, we end up with a grammar that doesn't look like the Spec, with unclear correctness and performance.

If we have java20/, then we know exactly what version the grammar supports. I expect the next JLS to come out in the next few months.

I should note that I have a program that scrapes the JLS .pdf and extracts a grammar in Antlr4 syntax. https://github.com/kaby76/ScrapeJavaSpec. My plan is to use that on JLS 20 and see what refactorings were done with Java20.g4.

Also, I am planning to split Java20.g4 and clean up the hardwired Java actions.

[AtKnight]

The time necessary to parse 846 files with 39433 lines of Java code is 9 min 50 seconds = 590 seconds, so 66.8 lines/sec.
CPU: i5-6400 CPU @ 2.70GHz × 4
Java version: 11.0

[kaby76]

Could you attach a .zip or .tar.gz file of the test suite, please? Thank you.

[AtKnightOU]

The 846 files are subjected to a privacy statement. Therefore I can not send the files.
I will search next evening for another set of files.

[kaby76]

That's okay. I have the 13000 java files in the OpenJDK. I am running the parser over these now.

[kaby76]

• I've created a branch with the java20 grammar. https://github.com/kaby76/grammars-v4/tree/g4-3656

• The source contains a readme with @AtKnight attributed.

• The original grammar contained Java code. I changed the grammar to be in target agnostic format so that it can be ported to other targets and tested. All my tools are CSharp based, which has much better development tools than Java.

  To bring the grammar into target agnostic format, the grammar was split. Several string literal constants in the original grammar needed to be explicitly declared in the split grammar, which is the list of rules I added to the beginning of the lexer grammar.

• The grammar was missing an EOF-terminated start rule. Unlike most parser generators, Antlr does not have a syntax to declare a start rule. In addition, with some grammars, a rule that is not terminated with EOF will cause the parse of an invalid input to succeed. trgen, which is used to test all grammars in grammars-v4, allows one to specify the start rule via an EOF-terminated rule, or declaration in the desc.xml. For comparison, parsing was tested for compilationUnit and start rules.

• For testing, I ran the grammar on 2869 files of the OpenJDK (find ~/jdk/src/ -name '*.java' | head -2869) on java20 and java/java/. I was planning to run the parser on all 13-plus thousand files, but the parser was very slow. I ran the parser two ways: "single" input file per parse program; "grouped" input files in one parse program. Both ways are important as "grouped" parsing, also known as "warm-up parsing", should be faster than single parsing because Antlr4 has cached decision tables. The runtimes below are the combined time for parsing all 2869 files. All parsers are for the CSharp target. My machine is a standard AMD Ryzen 7 2700, 16G DDR4 1311 MHz, SSD, B450 motherboard. There is no statistical analysis because the parse times for java20 were too long.

  Grammar	Single	Grouped	# passed	# failed
  java20 (start)	7777 s	(not tested)	2624	245
  java20 (compilationUnit)	7812 s	23120 s	2624	245
  java	333 s	62 s	2868	1

  This test does not check the validity of the parse tree of any parse.

  Note, the java20 grammar contains many ambiguous rules.

  The java20 grammar performed worse than java/java/ in both acceptance of valid input and performance. As I noted in StackOverflow just in the last few days), a grammar which is ambiguous results in a parser which is slow, and may perform worse for grouped parsing than single parsing.

• [java20] New grammar, Java version 20 #3659

• grouped.txt single.txt grouped-java.txt single-java.txt

[kaby76]

What is your OS (Windows, Linux, Mac) and target (CSharp, Java, Cpp, Dart--I don't have the other ports written)? I will generate a driver, build files, etc., for you to see how to build a program with the grammar.

Try this.
java-linux.zip

[kaby76]

Here's how you should resolve the "can't find Java20LexerBase":

git clone https://github.com/antlr/grammars-v4.git foobar
cd foobar
cd java/java20
cp Java/* .
# For targets that have a "transformGrammar.py", do this:
# python transformGrammar.py
# Generate.
export version=4.13.0
antlr4 -Dlanguage=Java Java20Lexer.g4
antlr4 -Dlanguage=Java Java20Parser.g4
# Compile.
# Note. "antlr4" downloads the JRE for generating the tool. You should use the same JRE downloaded.
# These lines set up the classpath for java. Make sure you have "." in the classpath.
JAR=`python -c "import os; from pathlib import Path; print(os.path.join(Path.home() , '.m2',  'repository', 'org', 'antlr', 'antlr4', '$version', 'antlr4-$version-complete.jar'))"`
CLASSPATH="$JAR:."
javac -cp "$CLASSPATH" *.java

[AtKnight]

Thank you for your explanation, but sadly it did not solve the problem.
I have solved the problem!

(1) Add options { tokenVocab = Java20Lexer; } to Java20Parser.g4

(2) Remove from Java20Lexer.g4
options
{
superClass = JavaLexerBase;
}
Notice class JavaLexerBase does not exist

(3) Java20Lexer.g4 is partly copied from Java9.
However, it contains errors.
It contains:
{ this.Check1() }?
{ this.Check2() }?
{ this.Check3() }?
{ this.Check4() }?
These methods are not implemented.
This will result in compilation errors.
Aside, it is advised not to add Java code to the lexer grammar.

[kaby76]

(1) Add options { tokenVocab = Java20Lexer; } to Java20Parser.g4

That makes absolutely no sense. The code already has the tokenVocab declaration, line 3 of java/java20/Java20Parser.g4, of the "master" branch of the grammars-v4 repo. Right here:

grammars-v4/java/java20/Java20Parser.g4

Line 3 in d5c0376

options { tokenVocab = Java20Lexer; }

(2) Remove from Java20Lexer.g4
options
{
superClass = JavaLexerBase;
}
Notice class JavaLexerBase does not exist

Again, that makes absolutely no sense. JavaLexerBase does exist. It exists right here:

grammars-v4/java/java20/Java/JavaLexerBase.java

Line 36 in d5c0376

public abstract class JavaLexerBase extends Lexer

It's all right there, right now, in the repo on https://github.com/antlr/grammars-v4/blob/master/java/java20/Java/JavaLexerBase.java

I suggest you delete the entire set of files and reclone grammars-v4 from scratch, from https://github.com/antlr/grammars-v4.git

The builds all work, too. Here was the Github Action build for the merge of the PR to gramamrs-v4:master. https://github.com/antlr/grammars-v4/actions/runs/5842274625

The result of the test is here, https://github.com/antlr/grammars-v4/actions/runs/5842274625/job/15843232718#step:22:77

And, to prove that the code builds and runs, here is a end-to-end clone, generate, compile, and run on Linux.

Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.10.102.1-microsoft-standard-WSL2 x86_64)
 ...
ken@DESKTOP-DL44R7B:~$ java --version
openjdk 11.0.19 2023-04-18
OpenJDK Runtime Environment (build 11.0.19+7-post-Ubuntu-0ubuntu120.04.1)
OpenJDK 64-Bit Server VM (build 11.0.19+7-post-Ubuntu-0ubuntu120.04.1, mixed mode, sharing)
ken@DESKTOP-DL44R7B:~$ rm -rf grammars-v4/
ken@DESKTOP-DL44R7B:~$ git clone https://github.com/antlr/grammars-v4.git
Cloning into 'grammars-v4'...
remote: Enumerating objects: 45878, done.
remote: Counting objects: 100% (4978/4978), done.
remote: Compressing objects: 100% (455/455), done.
remote: Total 45878 (delta 4665), reused 4654 (delta 4505), pack-reused 40900
Receiving objects: 100% (45878/45878), 43.89 MiB | 26.02 MiB/s, done.
Resolving deltas: 100% (24949/24949), done.
ken@DESKTOP-DL44R7B:~$ cd grammars-v4/java/java20
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20$ ls
CSharp  Cpp  Dart  Java  Java20Lexer.g4  Java20Parser.g4  README.md  desc.xml  examples  pom.xml
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20$ trgen -t Java
/home/ken/grammars-v4/java/java20
CSharp  Java20Parser.g4 success 0.0832017
CSharp  Java20Lexer.g4 success 0.0117898
Rendering template file from Java/st.build.ps1 to Generated-Java/st.build.ps1
Rendering template file from Java/st.build.sh to Generated-Java/st.build.sh
Rendering template file from Java/st.clean.ps1 to Generated-Java/st.clean.ps1
Rendering template file from Java/st.clean.sh to Generated-Java/st.clean.sh
Rendering template file from Java/st.ErrorListener.java to Generated-Java/st.ErrorListener.java
Rendering template file from Java/st.makefile to Generated-Java/st.makefile
Rendering template file from Java/st.perf.sh to Generated-Java/st.perf.sh
Rendering template file from Java/st.run.ps1 to Generated-Java/st.run.ps1
Rendering template file from Java/st.run.sh to Generated-Java/st.run.sh
Rendering template file from Java/st.Test.java to Generated-Java/st.Test.java
Rendering template file from Java/st.test.ps1 to Generated-Java/st.test.ps1
Rendering template file from Java/st.test.sh to Generated-Java/st.test.sh
Copying source file from /home/ken/grammars-v4/java/java20/pom.xml to Generated-Java/pom.xml
Copying source file from /home/ken/grammars-v4/java/java20/Java20Parser.g4 to Generated-Java/Java20Parser.g4
Copying source file from /home/ken/grammars-v4/java/java20/Java20Lexer.g4 to Generated-Java/Java20Lexer.g4
Copying source file from /home/ken/grammars-v4/java/java20/README.md to Generated-Java/README.md
Copying source file from /home/ken/grammars-v4/java/java20/desc.xml to Generated-Java/desc.xml
Copying source file from /home/ken/grammars-v4/java/java20/Java/JavaLexerBase.java to Generated-Java/JavaLexerBase.java
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20$ cd Generated-Java/
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ ls
ErrorListener.java  Java20Parser.g4     README.md  build.ps1  clean.ps1  desc.xml  makefile  pom.xml  run.sh    test.sh
Java20Lexer.g4      JavaLexerBase.java  Test.java  build.sh   clean.sh   examples  perf.sh   run.ps1  test.ps1
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ grep -i vocab *.g4
Java20Parser.g4:options { tokenVocab = Java20Lexer; }
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ grep superClass *.g4
Java20Lexer.g4:   superClass = JavaLexerBase;
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ version=4.13.0
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ antlr4 -v $version -encoding utf-8 -Dlanguage=Java   Java20Lexer.g4
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ ls
ErrorListener.java  Java20Lexer.java    JavaLexerBase.java  build.ps1  clean.sh  makefile  run.ps1   test.sh
Java20Lexer.g4      Java20Lexer.tokens  README.md           build.sh   desc.xml  perf.sh   run.sh
Java20Lexer.interp  Java20Parser.g4     Test.java           clean.ps1  examples  pom.xml   test.ps1
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ antlr4 -v $version -encoding utf-8 -Dlanguage=Java   Java20Parser.g4
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ ls
ErrorListener.java  Java20Lexer.tokens   Java20Parser.tokens            README.md  clean.ps1  makefile  run.sh
Java20Lexer.g4      Java20Parser.g4      Java20ParserBaseListener.java  Test.java  clean.sh   perf.sh   test.ps1
Java20Lexer.interp  Java20Parser.interp  Java20ParserListener.java      build.ps1  desc.xml   pom.xml   test.sh
Java20Lexer.java    Java20Parser.java    JavaLexerBase.java             build.sh   examples   run.ps1
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ JAR=`python -c "import os; from pathlib import Path; print(os.path.join(Path.home() , '.m2',  'repository', 'org', 'antlr', 'antlr4',  '$version', 'antlr4-$version-complete.jar'))"`
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ CLASSPATH="$JAR:."
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ echo $JAR
/home/ken/.m2/repository/org/antlr/antlr4/4.13.0/antlr4-4.13.0-complete.jar
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ echo $CLASSPATH
/home/ken/.m2/repository/org/antlr/antlr4/4.13.0/antlr4-4.13.0-complete.jar:.
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ javac -cp "$CLASSPATH" *.java
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ ls
 ErrorListener.class
 ErrorListener.java
 Java20Lexer.class
 ...
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$ java -cp "$CLASSPATH" Test examples/*.java
Java 0 examples/AllInOne8.java success 1.002
Java 1 examples/Instanceof.java success 0.03
Java 2 examples/ManyStringsConcat.java success 0.024
Java 3 examples/TryWithResourceDemo.java success 0.028
Java 4 examples/Unicode.java success 0.003
Java 5 examples/helloworld.java success 0.003
Java 6 examples/module-info.java success 0.034
Total Time: 1.292
ken@DESKTOP-DL44R7B:~/grammars-v4/java/java20/Generated-Java$

[AtKnight]

The grammar files are improved and tested.
An attachment with an extension pdf is not accepted.
The extension has been modified to txt.
improvements_pdf.txt
Java20Lexer.txt
Java20Parser.txt

[kaby76]

Thanks. I'll integrate the changes this week.

[daleanson]

Hi, I just ran across this conversation. Is the purpose of this java 20 grammar to make a reference grammar that follows the specification closely, regardless of performance? As far as I know, there are no language changes to java since java 17, so the java 17 grammar should parse java 20 files with no issues. Do I have that right?

[kaby76]

java/java20 is not a highly optimized grammar. But, it is closer aligned, apparently, to the JLS version 20. The java/java/ grammar is an optimized grammar. But, it's not clear exactly what version it recognizes as people have thrown in bits and pieces of newer syntax.

[AtKnight]

The grammar of java/java, compared to my grammar, results in a parser that is twice as fast!
How can my grammar be optimized, except by removing the ambiguities?

The grammar of java/java handles the Contextual Keywords, such as transitive and open correct. Well done!

The grammar of java/java contains two tiny mistakes. The attached Java file can not be parsed, because of the comma-separated lists.

module-info.txt

[AtKnight]

Sorry, the file is
moduleInfo.txt

[kaby76]

There's a slew of refactorings that were made to the JLS EBNF to get the java/java/ grammar. Unfortunately, there's no record of these refactorings. People just made the changes directly to the grammar. (We should not be developing software this way.) At one point I was cataloging the refactorings so I could scrape the EBNF from the JLS and then derive an optimized grammar using the CRUD tools in Trash. There were about 50 I got through, but there are many more. I'm not sure where I put all that info, but I will try to find and put it in Github.

There's more to the issues than just removing ambiguity. The parser "falls back" to a full parse context when it cannot predict the decision of what alt in a rule to take. Prediction context caching helps, but not completely, e.g., the sql/plsql grammar. A parse slows down with full context prediction.

[AtKnight]

The purpose of the pdf-attachment is only to inform you.
I ask no questions.
causesSpeedDiff_pdf.txt