Some multi-line edits can cause Exception

[mickaelistria]

I've run the "Format" action on a .json file to expand it from

{
	
	"a": "b", "c" : {"d" : "e"}
}

to

{
	"a": "b",
	"c": {
		"d": "e"
	}
}

and then syntax highlighting stopped working and I got this exception

Exception in thread "org.eclipse.tm4e.ui.internal.model.TMDocumentModel" java.lang.IndexOutOfBoundsException: Index 6 out of bounds for length 6
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:372)
	at java.base/java.util.ArrayList.get(ArrayList.java:458)
	at java.base/java.util.Collections$SynchronizedList.get(Collections.java:2424)
	at org.eclipse.tm4e.core.model.AbstractLineList.get(AbstractLineList.java:67)
	at org.eclipse.tm4e.core.model.TMModel$TokenizerThread._updateTokensInRange(TMModel.java:240)
	at org.eclipse.tm4e.core.model.TMModel$TokenizerThread.lambda$1(TMModel.java:162)
	at org.eclipse.tm4e.core.model.TMModel._withModelTokensChangedEventBuilder(TMModel.java:324)
	at org.eclipse.tm4e.core.model.TMModel.access$3(TMModel.java:321)
	at org.eclipse.tm4e.core.model.TMModel$TokenizerThread._revalidateTokensNow(TMModel.java:114)
	at org.eclipse.tm4e.core.model.TMModel$TokenizerThread.run(TMModel.java:100)

Closing the editor and opening it again fixes the issue.

[norru]

This is very annoying in Corrosion.

It's now ~100% on autoformat. Sometimes it's a bit better but it probably depends on the source file's length (ie, the larger the file, the more likely this is to happen). Formatting in-editor is pretty much useless. I need to close and open editor windows again every single time I hit the format shortcut. If the file is large I lose track.

cargo fmt is the only viable workaround.

Besides, it's not just the syntax colouring which is affected. It also screws the error parser (via RLS?) and I end up with a lot of spurious errors in the Problems window.

[norru]

Still not fixed as 2019-08-21 (snapshots). This is a very annoying as it basically renders the editor's formatter completely unusable.

[norru]

Still not fixed as of 2019-09-26.

[mickaelistria]

I tried with gen.rs and conrod_gfx.rs from https://github.com/itadinanta/rust-oids and I still cannot reproduce deterministically. I could reproduce the issue once, in various attemps, when undoing a Format, but still fail at reproducing it often enough to debug it properly.

[norru]

I still cannot reproduce deterministically

Yep, I don't think it's deterministic, there must be a race condition somewhere, but should be frequent enough.Can you get at least 1 in 3? I don't think it's realistic to expect much more your side.

It would be nice to find out why I do get it so often. It clearly isn't a machine thing.

[mickaelistria]

@norru: If you're still interested, you can install the snapshot from http://download.eclipse.org/tm4e/snapshots , then use https://help.eclipse.org/2019-09/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Freference%2Fmisc%2Fruntime-options.html to set an option file that sets org.eclipse.tm4e.ui/trace=true and then you should get in the eclipse log file more info from TM4E parsing/tokenizing the file.
Hopefully, in case of failure, a pattern will emerge.

[norru]

Still open as of 2020-01-29.

[norru]

Just upgrade to Eclipse 2019.12, gonna try org.eclipse.tm4e.ui/trace=true. Will it work as a -D argument?

[mickaelistria]

Thanks.
I think you need to put that in a file and pass this file as -debug argument to the application, see https://help.eclipse.org/2019-12/index.jsp?topic=%2Forg.eclipse.pde.doc.user%2Fguide%2Ftools%2Flaunchers%2Feclipse_application_launcher.htm for reference.

[norru]

Ah, ok! I need more time for setting this up then, I hoped I could test it on-the-fly.

[mickaelistria]

I added a document to give some hints about how to troubleshot issues: https://github.com/eclipse/tm4e/blob/master/documentation/TROUBLESHOOTING.md

[norru]

Still open as of 2020-12

[mickaelistria]

Cannot reproduce with latest snapshots. Please reopen with details to reproduce if you can reproduce it.