editor.maxTokenizationLineLength is not used leading to bogus coloring

[akosyakov]

Reproducible with https://github.com/opencaesar/vocabularies/blob/9b8c1ff9d07803420cebe03f44687a4da253373b/src/imce.jpl.nasa.gov/foundation/base.oml and https://github.com/opencaesar/oml-vscode

[melaasar]

Do I need to change something in the oml-vscode extension or in the opencaesar/vocabularies repo to get this fix?

[akosyakov]

@melaasar Hi 👋 We limit tokenisation to max 400 characters by default since it can hang the browser for big files. But a user can change this default with editor.maxTokenizationLineLength preference. VS Code is using 20000 characters but they are running natively on desktop, so performance it is better. You don't need to change anything in the extension.

[melaasar]

Thanks. I went ahead and added a workspace setting file where I added a value of 800 to editor.maxTokenizationLineLength:
https://github.com/opencaesar/vocabularies/blob/master/.theia/settings.json

However, when I start a new workspace, and browse to https://github.com/opencaesar/vocabularies/blob/master/src/imce.jpl.nasa.gov/foundation/base.oml
I still see the issue at line 5 (not colored properly) because the token at line 4 is long despite it now being shorter than 800 limit. If I try to split the "..." token on line 4 across 2 lines, even by just putting the ending " on the next line, then the color problem goes away.

[akosyakov]

@melaasar It was not deployed to Gitpod yet. It was only merged for Theia. Check out it tomorrow or on Friday maybe it will be there.