Expand usage of standard token type inside of the auto indentation to ignore comments and strings for indentation

[aiday-mar]

We currently have some issues with indentation when strings and comments are involved. We should ignore the indentation rules when the cursor is inside of a comment or string. This would help resolve certain issues including but not limited to:

• [Bug]: Incorrect indentation when object contains string keys #209517
• Auto indent on paste is incorrect in strings #209418
• Incorrect indentation inside of comments #208232

[aiday-mar]

After some discussion with the team, we have decided on the following.

• When a comment is encountered, it can be completely stripped off from the line and the regex pattern can be applied.
• When a string is encountered, either:
  • If the string is a multi line string, then ignore the lines which are completely contained within the string
  • If the string is a single line string or is on a line which contains tokens other than strings, then change the brackets (from the configuration file) inside of the string to some dead character (or remove them) and apply the indentation rule on this cleaned string.
• When a regex is encountered, apply the same logic as for strings

[RedCMD]

looking at this
I'm trying to set some decent indentation rules for YAML

I have this for a block-scalar with a explicit 9 indent indicator
pressing enter will indent 9 spaces

{
	// indent 9 spaces on block-scalar with indentation-indicator 
	"action": {
		"appendText": "         ",
		"indent": "none",
	},
	"afterText": "^(?![ \t]*#)",
	"beforeText": "(^[ \t]*|[-?:][ \t]+)(>|\\|)[+-]?9"
}

however, I don't want to indent if the text is already within a string (or comment)
atm the current VSCode feature set allows me to get decently restrictive, but not perfect

I have <> set as brackets
but I'm unable to do the same to |
["|", ""] doesn't work (and causes issues)

>

 >9

 |9

pressing enter after >9 correctly yields one space indent
but pressing enter after |9 incorrectly yields 9 spaces of indent

"previousLineText" can help slightly, but will still fail in that example

and to also throw a spanner in the works
block-scalar is a multiline string
however the indentation on each line is not of type string.
which I assume VSCode will detect it as a single line string instead of multi-line

it would be nice if I could define custom character sets to remove from strings rather than just relying on brackets
which would allow me to remove : (yes colon and space combined, not separate) combinations from strings to help with block-maps: