Running on Java 16 fails

[sormuras]

Running GJF google-java-format-1.9-all-deps.jar on Java 16 (16-ea+21) fails with the follow error

java.lang.NoSuchFieldError: reader
	at com.google.googlejavaformat.java.JavacTokens$CommentSavingTokenizer.processComment(JavacTokens.java:131)
	at jdk.compiler/com.sun.tools.javac.parser.JavaTokenizer.readToken(JavaTokenizer.java:949)
	at jdk.compiler/com.sun.tools.javac.parser.Scanner.nextToken(Scanner.java:115)
	at com.google.googlejavaformat.java.JavacTokens.getTokens(JavacTokens.java:88)
	at com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:364)
	at com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:334)
	at com.google.googlejavaformat.java.JavaInput.<init>(JavaInput.java:276)
	at com.google.googlejavaformat.java.Formatter.getFormatReplacements(Formatter.java:280)
	at com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:267)
	at com.google.googlejavaformat.java.FormatFileCallable.call(FormatFileCallable.java:45)
	at com.google.googlejavaformat.java.FormatFileCallable.call(FormatFileCallable.java:26)

[sormuras]

Due to

openjdk/jdk@4f9a1ff#diff-9dd71a3938fb726eb0eb9b7b30d8727167dad2402854b38464446c7a63ca1638L90-L94

the JavaTokenizer no longer has a protected UnicodeReader reader field but extends class UnicodeReader in the same package. The driving issue was https://bugs.openjdk.java.net/browse/JDK-8254073 Tokenizer improvements

[sormuras]

With #544 being merged, it'd be a good idea to add Java ea job to the test matrix (allowing it to fail) that test-drives the current early-access build of OpenJDK.

[sormuras]

Preparing a PR here: https://github.com/sormuras/google-java-format/runs/1455621338 -- the JDK Early-Access job fails as expected with the error message shown in the issue description.

The overall workflow is marked as successful.

Although the JDK Early-Access job failed.

[sormuras]

Did the commit labeled Work around change to tokenization in JDK-8254073 fix this issue?

It seems like google-java-format-1.9-all-deps.jar now runs on Java 16 ... except for some illegal access errors:

java -jar google-java-format-1.9-all-deps.jar [...]
[...]
java.lang.IllegalAccessError:
  class com.google.googlejavaformat.java.JavaInput (in unnamed module @0x442e0405)
  cannot access class com.sun.tools.javac.parser.Tokens$TokenKind (in module jdk.compiler) 
  because module jdk.compiler does not export com.sun.tools.javac.parser to unnamed module @0x442e0405
        at com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:349)
        at com.google.googlejavaformat.java.JavaInput.buildToks(JavaInput.java:334)
        at com.google.googlejavaformat.java.JavaInput.<init>(JavaInput.java:276)
        at com.google.googlejavaformat.java.Formatter.getFormatReplacements(Formatter.java:280)
        at com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:267)
        at com.google.googlejavaformat.java.FormatFileCallable.call(FormatFileCallable.java:45)
        at com.google.googlejavaformat.java.FormatFileCallable.call(FormatFileCallable.java:26)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:831)

As --illegal-access=<value> is deprecated for removal, adding these exports directives to the java launcher helps:

java
--add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
-jar google-java-format-1.9-all-deps.jar [...]

[cushon]

I think it works on the latest JDK 16 ea as of 0a2f864. (The CI isn't passing with 16 ea yet because I need to update Error Prone, I'll take a look at that.)

And yes, we'll also want to document the set of --add-exports needed for 16 ea, similar to google/error-prone#1157 (comment).

[sormuras]

So far so good. Looking forward to the 1.10 release of gjf.

Any clue, why does the google-java-format-1.9-all-deps.jar I downloaded in October 2020, with the set of --add-exports applied, run happily on Java 16+36-2231? I expected to see the java.lang.NoSuchFieldError: reader pop up... 🤔 ...as a) the commit you mentioned dates to December 2020 and b) my local JAR file did not change.

[cushon]

google-java-format-1.9-all-deps.jar works for me with 16-ea+34-2216 as long as I avoid the code path that was try to reference reader, i.e. by only formatting files that don't contain comments. Any chance you were testing on a trivial example with no comments?

class T {}

$ java  --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED -jar google-java-format-1.9-all-deps.jar Y.java
class T {}

/** test */
class T {}

$ java  --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED --add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED -jar google-java-format-1.9-all-deps.jar Y.java
...
T.java: error: reader
java.lang.NoSuchFieldError: reader
	at com.google.googlejavaformat.java.JavacTokens$CommentSavingTokenizer.processComment(JavacTokens.java:131)
	at jdk.compiler/com.sun.tools.javac.parser.JavaTokenizer.readToken(JavaTokenizer.java:949)

[sormuras]

Any chance you were testing on a trivial example with no comments?

Yes. Two .java files, no comments. Adding a single // ... provokes the error. Thanks for double-checking.

[agavrilov76]

JDK 16 is out today. Looking forward to the next release of the google formatter.

[agavrilov76]

@cushon Any plans to release a new version soon? We would like to migrate our projects google-java-formatter to Java 16. Thanks!

[cushon]

I published a release, and documented the --add-exports= flags that are needed for JDK 16: https://github.com/google/google-java-format/releases/tag/v1.10.0

[agavrilov76]

@cushon Great! Looking forward to start using it.
Perhaps it makes sense to increment the version number in the read me file?

[sormuras]

Can you please push the 1.10 binaries to Maven central?

They're here: https://repo.maven.apache.org/maven2/com/google/googlejavaformat/google-java-format/1.10.0/

[cushon]

Thanks for the reminder about the README, I'm updating it.

[agavrilov76]

Thanks!
Looks like the version in the README should be changed to 1.10.0

Can you also please release a new version of IntelliJ plugin? :)

[cushon]

I'm adding the missing .0, and there should be a release of the IntelliJ plugin soon-ish