Plugin does not work with IntelliJ 2020.2 #55

fhoeben · 2020-08-10T13:22:49Z

(copied from review at https://plugins.jetbrains.com/plugin/7908-fitnesse by Mathias Rühl)

FitNesse Plugin 1.10.1 leads to exceptions in IntelliJ 2020.2 with FitNesse Version "fitnesse-standalone-20190716.jar".
I had to disable/uninstall that Plugin to have a working IDE, as IntelliJ does not behave correctly when the Exception occur.

Error while indexing /user/home/delevelop/FitNesseRoot/FitNesse/UserGuide/content.txt
To reindex this file IDEA has to be restarted

com.intellij.util.indexing.impl.MapReduceIndex$MapInputException: Failed to map data for input 15878
	at com.intellij.util.indexing.impl.MapReduceIndex.mapByIndexer(MapReduceIndex.java:300)
	at com.intellij.util.indexing.impl.MapReduceIndex.mapInput(MapReduceIndex.java:285)
	at com.intellij.util.indexing.impl.storage.VfsAwareMapReduceIndex.mapInput(VfsAwareMapReduceIndex.java:157)
	at com.intellij.util.indexing.impl.storage.VfsAwareMapReduceIndex.mapInput(VfsAwareMapReduceIndex.java:41)
	at com.intellij.util.indexing.impl.MapReduceIndex.mapInputAndPrepareUpdate(MapReduceIndex.java:223)
	at com.intellij.util.indexing.FileBasedIndexImpl.updateSingleIndex(FileBasedIndexImpl.java:1387)
	at com.intellij.util.indexing.FileBasedIndexImpl.lambda$doIndexFileContent$25(FileBasedIndexImpl.java:1292)
	at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl.freezeFileTypeTemporarilyIn(FileTypeManagerImpl.java:557)
	at com.intellij.util.indexing.FileBasedIndexImpl.doIndexFileContent(FileBasedIndexImpl.java:1262)
	at com.intellij.util.indexing.FileBasedIndexImpl.indexFileContent(FileBasedIndexImpl.java:1214)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.lambda$indexOneFileOfJob$1(IndexUpdateRunner.java:195)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.insideReadAction(NonBlockingReadActionImpl.java:498)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.lambda$attemptComputation$2(NonBlockingReadActionImpl.java:446)
	at com.intellij.openapi.application.impl.ApplicationImpl.tryRunReadAction(ApplicationImpl.java:1110)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.lambda$runInReadActionWithWriteActionPriority$0(ProgressIndicatorUtils.java:77)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runActionAndCancelBeforeWrite(ProgressIndicatorUtils.java:153)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.lambda$runWithWriteActionPriority$1(ProgressIndicatorUtils.java:114)
	at com.intellij.openapi.progress.ProgressManager.lambda$runProcess$0(ProgressManager.java:56)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:170)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:629)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:581)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:60)
	at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:157)
	at com.intellij.openapi.progress.ProgressManager.runProcess(ProgressManager.java:56)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runWithWriteActionPriority(ProgressIndicatorUtils.java:111)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runInReadActionWithWriteActionPriority(ProgressIndicatorUtils.java:77)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.attemptComputation(NonBlockingReadActionImpl.java:463)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.executeSynchronously(NonBlockingReadActionImpl.java:401)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl.executeSynchronously(NonBlockingReadActionImpl.java:181)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.indexOneFileOfJob(IndexUpdateRunner.java:198)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.indexJobsFairly(IndexUpdateRunner.java:152)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.lambda$indexFiles$0(IndexUpdateRunner.java:98)
	at com.intellij.util.concurrency.BoundedTaskExecutor.doRun(BoundedTaskExecutor.java:215)
	at com.intellij.util.concurrency.BoundedTaskExecutor.access$200(BoundedTaskExecutor.java:26)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.execute(BoundedTaskExecutor.java:194)
	at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:207)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.run(BoundedTaskExecutor.java:183)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.intellij.diagnostic.PluginException: Discontinuous sequence of tokens is generated by lexer: fitnesse.idea.lexer.FitnesseLexer [Plugin: FitNesse]
	at com.intellij.ide.plugins.PluginManagerCore.createPluginException(PluginManagerCore.java:290)
	at com.intellij.diagnostic.PluginProblemReporterImpl.createPluginExceptionByClass(PluginProblemReporterImpl.java:12)
	at com.intellij.diagnostic.PluginException.createByClass(PluginException.java:58)
	at com.intellij.openapi.editor.ex.util.ValidatingLexerWrapper.throwException(ValidatingLexerWrapper.java:126)
	at com.intellij.openapi.editor.ex.util.ValidatingLexerWrapper.advance(ValidatingLexerWrapper.java:94)
	at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.doSetText(LexerEditorHighlighter.java:441)
	at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.setText(LexerEditorHighlighter.java:409)
	at com.intellij.psi.impl.cache.impl.id.PlatformIdTableBuilding$TokenSetTodoIndexer.map(PlatformIdTableBuilding.java:104)
	at com.intellij.psi.impl.cache.impl.id.PlatformIdTableBuilding$TokenSetTodoIndexer.map(PlatformIdTableBuilding.java:83)
	at com.intellij.psi.impl.cache.impl.todo.TodoIndex$4.map(TodoIndex.java:100)
	at com.intellij.psi.impl.cache.impl.todo.TodoIndex$4.map(TodoIndex.java:77)
	at com.intellij.util.indexing.CompositeDataIndexer.map(CompositeDataIndexer.java:54)
	at com.intellij.util.indexing.CompositeDataIndexer.map(CompositeDataIndexer.java:18)
	at com.intellij.util.indexing.impl.MapReduceIndex.mapByIndexer(MapReduceIndex.java:294)
	... 43 more

The text was updated successfully, but these errors were encountered:

fhoeben · 2020-08-10T13:25:22Z

It looks like the new IntelliJ version checks whether all token generated by the plugin's lexer have continuous offsets.
The FitNesseLexer seems to reset the offset to 0 when a collapsible section is ended, resulting in a lower start offset after the section than the end offset of the section. This causes the exception I believe.

fhoeben · 2020-08-10T14:27:23Z

On closer inspection the problem seems to be that the lexer generates 1 token for the entire section, and then tokens for the section's content. That is not what IntelliJ expects...

fhoeben · 2020-08-11T12:20:58Z

Via Slack I queried why these new checks were added. I got the following reply:

Yann Cébron:jb:
these checks were added inspired by problems like this SeeSharpSoft/intellij-csv-validator#204, see com.intellij.openapi.editor.ex.util.ValidatingLexerWrapper javadoc

  12 /**
  13  * This class verifies that delegate lexer generates a continuous sequence of tokens (without gaps), and that it
  14  * does not stall during iteration (generating empty tokens for the same offset continuously).
  15  */
  16 public class ValidatingLexerWrapper extends Lexer {

fhoeben · 2020-08-11T12:35:33Z

I believe the following test in src/test/scala/fitnesse/idea/lexer/MiscLexerSuite.scala shows the issue

  test("Collapsed section") {
    assertResult(
      List(
        (FitnesseTokenType.COLLAPSIBLE_START, 0, 7, "!****< "),
        (FitnesseTokenType.WORD, 7, 10, "abc"),
        (FitnesseTokenType.LINE_TERMINATOR, 10, 11, "\n"),
        (FitnesseTokenType.WORD, 11, 14, "def"),
        (FitnesseTokenType.LINE_TERMINATOR, 14, 15, "\n"),
        (FitnesseTokenType.COLLAPSIBLE_END, 15, 17, "*!"),
        (FitnesseTokenType.WHITE_SPACE, 17, 18, " "),
        (FitnesseTokenType.WORD, 18, 22, "word")
      )) {
      lexWithOffset("!****< abc\ndef\n*! word")
    }
  }

Some other tests in this suite will probably also need to be adjusted when this issues is solved.

JThoennes · 2021-05-03T11:11:10Z

Hello,
since version 1.12.0 it seems this issue has been reappeared:

com.intellij.diagnostic.PluginException: Discontinuous sequence of tokens is generated by lexer: fitnesse.idea.lexer.FitnesseLexer [Plugin: FitNesse]
	at com.intellij.ide.plugins.PluginManagerCore.createPluginException(PluginManagerCore.java:300)
	at com.intellij.diagnostic.PluginProblemReporterImpl.createPluginExceptionByClass(PluginProblemReporterImpl.java:12)
	at com.intellij.diagnostic.PluginException.createByClass(PluginException.java:81)
	at com.intellij.openapi.editor.ex.util.ValidatingLexerWrapper.throwException(ValidatingLexerWrapper.java:126)
	at com.intellij.openapi.editor.ex.util.ValidatingLexerWrapper.advance(ValidatingLexerWrapper.java:94)
	at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.doSetText(LexerEditorHighlighter.java:453)
	at com.intellij.openapi.editor.ex.util.LexerEditorHighlighter.setText(LexerEditorHighlighter.java:420)
	at com.intellij.openapi.fileEditor.impl.text.EditorHighlighterUpdater.createHighlighter(EditorHighlighterUpdater.java:142)
	at com.intellij.openapi.fileEditor.impl.text.EditorHighlighterUpdater.lambda$updateHighlightersAsync$0(EditorHighlighterUpdater.java:128)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.insideReadAction(NonBlockingReadActionImpl.java:521)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.lambda$attemptComputation$3(NonBlockingReadActionImpl.java:468)
	at com.intellij.openapi.application.impl.ApplicationImpl.tryRunReadAction(ApplicationImpl.java:1096)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.lambda$runInReadActionWithWriteActionPriority$0(ProgressIndicatorUtils.java:79)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runActionAndCancelBeforeWrite(ProgressIndicatorUtils.java:157)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.lambda$runWithWriteActionPriority$1(ProgressIndicatorUtils.java:119)
	at com.intellij.openapi.progress.ProgressManager.lambda$runProcess$0(ProgressManager.java:57)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:178)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:688)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:634)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:64)
	at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:165)
	at com.intellij.openapi.progress.ProgressManager.runProcess(ProgressManager.java:57)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runWithWriteActionPriority(ProgressIndicatorUtils.java:116)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runInReadActionWithWriteActionPriority(ProgressIndicatorUtils.java:79)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.attemptComputation(NonBlockingReadActionImpl.java:486)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.lambda$transferToBgThread$1(NonBlockingReadActionImpl.java:408)
	at com.intellij.util.concurrency.BoundedTaskExecutor.doRun(BoundedTaskExecutor.java:216)
	at com.intellij.util.concurrency.BoundedTaskExecutor.access$200(BoundedTaskExecutor.java:27)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.execute(BoundedTaskExecutor.java:195)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.run(BoundedTaskExecutor.java:187)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
	at java.base/java.lang.Thread.run(Thread.java:834)

It also happens all the time with IDEA 2021.1.1.

fhoeben · 2021-05-03T16:25:21Z

@JThoennes Do you have a sample wiki page that shows the issue? The tests I added at the time still seem to be green, and locally I don't see the issue immediately either...

JThoennes · 2021-05-03T16:35:17Z

@fhoeben will try to find a sample among the many pages we have. Thanks for taking care of that.

JThoennes · 2021-05-04T11:47:26Z

@fhoeben , the page boils down to this simple example:

!***> heading
*!

The error happens if there is no line between the !* and the final *!.

In addition, this sequence does not look correct in IntelliJ:

!***> heading 1
...
...
*!
!***> heading 2
...
*!

The trailing *! seems to be merged with the leading !*.
Looking forward to your changes :-)

JThoennes · 2021-05-04T12:06:33Z

@fhoeben For example:

  test("Collapsed section without content") {
    assertResult(
      List(
        (FitnesseTokenType.COLLAPSIBLE_START, 0, 6, "!***> "),
        (FitnesseTokenType.WORD, 6, 9, "abc"),
        (FitnesseTokenType.LINE_TERMINATOR, 9, 10, "\n"),
        (FitnesseTokenType.COLLAPSIBLE_END, 10, 12, "*!"),
        (FitnesseTokenType.LINE_TERMINATOR, 12, 13, "\n")
      )) {
      lexWithOffset("!***> abc\n*!\n")
    }
  }

results in:

- Collapsed section without content *** FAILED *** (6 milliseconds)
  Expected List((FitnesseTokenType.COLLAPSIBLE_START,0,6,!***> ), (FitnesseTokenType.WORD,6,9,abc), (FitnesseTokenType.LINE_TERMINATOR,9,10,
  ), (FitnesseTokenType.COLLAPSIBLE_END,10,12,*!), (FitnesseTokenType.LINE_TERMINATOR,12,13,
  )), but got List((FitnesseTokenType.COLLAPSIBLE_START,0,6,!***> ), (FitnesseTokenType.WORD,6,9,abc), (FitnesseTokenType.LINE_TERMINATOR,9,10,
  ), (FitnesseTokenType.COLLAPSIBLE_END,12,13,
  )) (MiscLexerSuite.scala:199)

JThoennes · 2021-05-05T13:38:53Z

@fhoeben I am struggling with:

scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.

so I cannot debug this with IntelliJ. Any help appreciated.

fhoeben · 2021-05-05T14:15:42Z

I'm no scala programmer, so I don't know. But some Googling let me to:
https://stackoverflow.com/questions/39791996/scala-reflect-internal-missingrequirementerror-object-java-lang-object-in-compi#answer-39795029

Which Java version are you debugging with? I expect it should be Java 8

JThoennes · 2021-05-05T14:52:30Z

@fhoeben So at least the test case fails on the command line as expected.
Googling myself, I previously also found this SO article. And I am using Java 8.
But I did not get it working so far... Guess you have working setup :-)

timovd mentioned this issue Aug 11, 2020

Lexer exception #50

Open

fhoeben added a commit to fhoeben/idea-fitnesse that referenced this issue Aug 17, 2020

Ensure tokens generated by Lexer are continous, fixes gshakhn#55

3e07a4b

fhoeben mentioned this issue Aug 17, 2020

Ensure tokens generated by Lexer are continous fitnesse/idea-fitnesse#7

Merged

fhoeben added a commit to fhoeben/idea-fitnesse that referenced this issue Aug 17, 2020

Ensure tokens generated by Lexer are continous, fixes gshakhn#55

272aadf

fhoeben closed this as completed in fitnesse/idea-fitnesse#7 Aug 17, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plugin does not work with IntelliJ 2020.2 #55

Plugin does not work with IntelliJ 2020.2 #55

fhoeben commented Aug 10, 2020

fhoeben commented Aug 10, 2020

fhoeben commented Aug 10, 2020

fhoeben commented Aug 11, 2020

fhoeben commented Aug 11, 2020

JThoennes commented May 3, 2021

fhoeben commented May 3, 2021

JThoennes commented May 3, 2021

JThoennes commented May 4, 2021

JThoennes commented May 4, 2021 •

edited

Loading

JThoennes commented May 5, 2021

fhoeben commented May 5, 2021

JThoennes commented May 5, 2021

Plugin does not work with IntelliJ 2020.2 #55

Plugin does not work with IntelliJ 2020.2 #55

Comments

fhoeben commented Aug 10, 2020

fhoeben commented Aug 10, 2020

fhoeben commented Aug 10, 2020

fhoeben commented Aug 11, 2020

fhoeben commented Aug 11, 2020

JThoennes commented May 3, 2021

fhoeben commented May 3, 2021

JThoennes commented May 3, 2021

JThoennes commented May 4, 2021

JThoennes commented May 4, 2021 • edited Loading

JThoennes commented May 5, 2021

fhoeben commented May 5, 2021

JThoennes commented May 5, 2021

JThoennes commented May 4, 2021 •

edited

Loading