Backslashes in DataTable messes with highlighting

[maave0]

👓 What did you see?

I made a .feature file that included a DataTable and Examples table. Some lines had the last several characters uncolored - The lines that had backslashes in them.

✅ What did you expect to see?

All the text inside the pipe formatting is colored.

📦 Which tool/library version are you using?

Cucumber extension v1.7.0 for VSCode installed on VS Code v1.74.0

🔬 How could we reproduce it?

Steps to reproduce the behavior:

1. Install Cucumber extension v1.7 onto VS Code v.1.74.0
2. Create a file called 'whatever.feature'
3. Write a scenario that includes either DataTable or Examples table
4. Put numerous backslashes inside the table's text

Example:

@BugReport
Feature: BugReport

Scenario: DEID117 Success
    Given I submit a bug report

    And I put backslashes in a DataTable or Examples table
    | this line works fine |
    | C:\\Users\\this line won't highlight the last 2 characters |
    | I see it with escaped newlines and escaped backslashes. ( \n \n \n \n  ) will produce 4 uncolored characters|

    Then Cucumber team fixes the highlighting :)

[kieran-ryan]

Thanks for catching this issue! Capturing some notes in case someone is in a position to take a look.

What's happening is the gherkin is being read and parsed as strings; and as they are escaped when evaluating their length, they are being incorrectly calculated as shorter than their full unescaped length. Thus, the syntax highlighting is not fully applied.

language-service/src/service/getGherkinSemanticTokens.ts

Lines 49 to 63 in aa2843d

	function makeToken(
	lineNumber: number,
	character: number,
	token: string,
	type: SemanticTokenTypes,
	data: readonly number[]
	) {
	const charDelta = lineNumber === lastLineNumber ? character - lastCharacter : character
	lastCharacter = character
	const lineOffset = lineNumber - lastLineNumber
	lastLineNumber = lineNumber
	const length = token.length
	const typeIndex = indexByType[type]
	return data.concat([lineOffset, charDelta, length, typeIndex, 0])
	}

There is also an issue where when parsing we may lose information e.g. \ and \\ may both evaluate to \\.

Text	Token	Length	Expected Length
valu\e	valu\\e	6	6
valu\\e	valu\\e	6	7