Adjusts csharp semantic token scopes to match 1.26 outputs

[fuzzlebuck]

This PR adjusts the semantic token scopes to match those in earlier versions, 1.26 etc.

Directly relates to Issue #5731

I've compared the foreground outputs using the Inspect Editor Tokens in version 1.26.0 and in this version an confirmed the primary foreground now matches for all of the updated scopes in this PR.

For example the two listed in the original issue:

Namespace in 1.26.0:

Namespace in latest:

Parameter in 1.26.0:

Parameter in latest:

[fuzzlebuck]

I believe the tests are failing for an unrelated reason and I'm unable to restart the pipeline due to lack of access if someone could retry them for me?

[dibarbet]

Thanks for the contribution - one suggestion to try out.

[dibarbet]

So looking at the 1.x version, it doesn't specify any fallback textmate scopes for semantic tokens. Which I believe means that it uses the default fallbacks as defined here - https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#predefined-textmate-scope-mappings .

I think removing the fallbacks for the core LSP semantic token names (aka anything in that table) would have the same affect (as suggested by Joey here - #5731 (comment)). Would you be able to try that? If it works I think that is the direction we should take.

[fuzzlebuck]

Makes sense, thank you, I've made the change and it looks good!

It's also resolved a 1.x issue I had noticed where local variables had the .cs namespace so looked incorrect in some themes.

[dibarbet]

So trying it out locally - it definitely doesn't look right (mainly keywords I think?) on Visual Studio 2019 Dark

What the colors should look like:

It looks like its getting the keyword.control scope which is purple in the theme. But these aren't control keywords. I do see that there isn't a fallback scope listed for keyword in the table - we might need to keep that one?

cc @JoeRobich in case he knows of another idea

[dibarbet]

`typeParameter doesn't list a fallback in the table, we probably need to keep this one

[fuzzlebuck]

This one is interesting, in 1.x it maps as entity.name.type.parameter

Putting it back in the scopes list for 2.x it comes out as expected entity.name.type.type-parameter.cs, but should it be the more generic one or the .cs specific one?

[fuzzlebuck]

I can see in the VS theme's it's using entity.name.type.type-parameter

[fuzzlebuck]

I think we should set it to "entity.name.type.type-parameter" in the scopes

[dibarbet]

This one is interesting, in 1.x it maps as entity.name.type.parameter

Interesting - I don't know where that is coming from.

I think we should set it to "entity.name.type.type-parameter" in the scopes

Yup, I'm good with that. If though entity.name.type.parameter looks better on other themes, and it works with the VS theme, I'm also OK with setting to that. But entity.name.type.type-parameter seems to be what O# used.

[fuzzlebuck]

I checked this one on a couple of themes and it looks ok to me.

[dibarbet]

same with keyword here.

[dibarbet]

If keyword.cs doesn't work on the other themes - try just 'keyword'. It looks like it should have the correct color in the theme json
https://github.com/dotnet/vscode-csharp/blob/main/themes/vs2019_dark.json#L311

[fuzzlebuck]

Okay, thank you, I will update and will double check them one by one and compare the outputs.

[fuzzlebuck]

Added this back in, looks ok now, please see my latest comment with comparisons.

[fuzzlebuck]

Updated with changes and compared each token that was removed against v 1.26.0:

1.26.0:

Latest update:

A couple of small discrepancies to mention:

• The preprocessor has a different color now due to it being mapped to entity.name.function.preprocessor whereas in 1.26.0 it was mapped to keyword.preprocessor.cs, let me know if you think I should change it to the .cs one.

• The type parameter for generics is now mapping to entity.name.type.type-parameter whereas in 1.26.0 it was mapped to entity.name.type.parameter although in the themes I looked at it made no difference aesthetically.

• I couldn't work out how to match a plain entity.name.type in code.

[dibarbet]

The preprocessor has a different color now due to it being mapped to entity.name.function.preprocessor whereas in 1.26.0 it was mapped to keyword.preprocessor.cs, let me know if you think I should change it to the .cs one.

I'm OK with this change, it looks to be the same in the new version with or without this change. If we get a ton more feedback we can always switch it later.

The type parameter for generics is now mapping to entity.name.type.type-parameter whereas in 1.26.0 it was mapped to entity.name.type.parameter although in the themes I looked at it made no difference aesthetically.

Yup works for me.

Thanks for the contribution!

[fuzzlebuck]

Happy to help, and thank you!

[JoeRobich]

It looks like its getting the keyword.control scope which is purple in the theme

Blame VS Code for that. No idea why they only included one core keyword token type and choose the keyword.control scope to represent it. For O#, we originally didn't utilize any of the built in token types and had a plainKeyword token type. We have since taken changes to use the LSP core token types when possible to work better with other LSP enabled editors.