Standard Way of Syntax Highlighting in Xtext Framework

Syntax highlighting is a concept where different parts of code are distinguished by various font types, font widths, font, shapes, font colors, background colors, etc. The following text describes how to create and configure a syntax highlighter for the newly created language in the Xtext framework.

Text Styles

Before any font type, font width, font shape, font color or background color is associated with a part of code, it a text style has to be created that comprises all attributes needed for differentiation of the part of the code. By default, the Xtext framework offers several predefined text styles named such as comments, numbers, keywords, etc. which can be modified by a coder of a particular language through editor preferences in GUI.

Now the question arises how to create new text styles. One possible solution is to implement the IHighlightingConfiguration interface and register it with a usage of the Google Guice. Implement the mentioned interface involves overriding the method named configure. The method has only one parameter whose type is the IHighlightingConfigurationAcceptor. The implementation of the configure method should contain method calls related to the acceptDefaultHighlighting method of the parameter. The method serves to register a text style under some identifier and further the method has three parameters. The first one is identifier that should be unique. The second one is the name of the style that will be used in GUI for style's representation. And the last one is the text style by itself that contains information about a font type, a font style, a font color, a background color, etc. The listing below demonstrates what an implementation of the IHighlightingConfiguration interface may look like.

Listing 1

A highlighting configuration containing four font styles dedicated for numbers, keywords, methods and other text kinds.

public class ExampleHighlightingConfiguration implements IHighlightingConfiguration
{
    public static final String DEFAULT_ID = "default";
    public static final String KEYWORD_ID = "keyword";
    public static final String METHOD_ID = "method";
    public static final String NUMBER_ID = "number";

    @Override
    public void configure(IHighlightingConfigurationAcceptor acceptor) {
        acceptor.acceptDefaultHighlighting(DEFAULT_ID, "Default", defaultTextStyle());
        acceptor.acceptDefaultHighlighting(KEYWORD_ID, "Keyword", keywordTextStyle());
        acceptor.acceptDefaultHighlighting(METHOD_ID, "Method", methodTextStyle());
        acceptor.acceptDefaultHighlighting(NUMBER_ID, "Number", numberTextStyle());
    }
	
    protected TextStyle defaultTextStyle() {
        TextStyle textStyle = new TextStyle();
        textStyle.setColor(new RGB(0, 0, 0));
        return textStyle;
    }

    protected TextStyle keywordTextStyle() {
        TextStyle textStyle = defaultTextStyle().copy();
        textStyle.setColor(new RGB(127, 0, 85));
        textStyle.setStyle(SWT.BOLD);
        return textStyle;
    }

    protected TextStyle methodTextStyle() {
        TextStyle textStyle = defaultTextStyle().copy();
        textStyle.setColor(new RGB(85, 0, 127));
        textStyle.setStyle(SWT.ITALIC);
        return textStyle;
    }
	
    protected TextStyle numberTextStyle() {
        TextStyle textStyle = defaultTextStyle().copy();
        textStyle.setColor(new RGB(127, 127, 127));
        return textStyle;
    } 
}

Although, the implementation of the IHighlightingConfiguration interface is a good possibility how to add new text styles, all default text styles are lost with this option. The possible solution how to preserve the default styles is to extend the DefaultHighlightingConfiguration class already implementing the interface and override the configure method to add new text styles. In order not to lose all default text styles, method's implementation has to contain call of the ancestor's implementation.

Lexical Highlighting

Now the situation concerning the text styles is defined. The methods of how to use text styles and therefore associate them with parts of code are two. One of them is the lexical method. Essence of the method is to associate text styles with tokens that are the result of the lexical analysis. A mere association is being performed by extending the AbstractAntlrTokenToAttributeIdMapper class and overriding the calculateId method. The method is called from the outside for a concrete token, whose name is passed by the parameter. Thus it is up to the language developer to design the procedure making decisions what identifier of a text style will be returned by the method. The name of a token can take the following values. In such a case that the token represents a rule call referencing a terminal rule, then the name has the format ”RULE_” plus the name of the terminal rule. In other cases the name of the token is its value. Therefore the language developer can use regular expressions, whose strength is on the lexical level. The following listing outlines what the described situation might look like in practice.

Listing 2

An extension of the AbstractAntlrTokenToAttributeIdMapper class which associates numbers, keywords and other tokens with relevant text styles.

public class ExampleAntlrTokenToAttributeIdMapper extends AbstractAntlrTokenToAttributeIdMapper
{
    private static final Pattern QUOTED = Pattern.compile("(?:^'(\\w[^']*)'$)|(?:^\"(\\w[^\"]*)\")$", Pattern.MULTILINE);
	
    @Override
    protected String calculateId(String tokenName, int tokenType) {
        if(tokenName.equals("RULE_INT")) {
            return ExampleHighlightingConfiguration.NUMBER_ID;
        }
        else if(QUOTED.matcher(tokenName).matches()) {
            return ExampleHighlightingConfiguration.KEYWORD_ID;
        }
        return ExampleHighlightingConfiguration.DEFAULT_ID;
    }
}

Semantic Highlighting

It was mentioned that two methods exist allowing for associating text styles with parts of a code. The semantic method is the second one. The method has an access to the grammar and therefore it can make decisions based on the grammar's semantics unlike the lexical method arbitrating by the type or the format of the token. The semantic method can be realized by implementing the ISemanticHighlightingCalculator interface and overriding the provideHighlightingFor method. The method has two parameters. The first one is a resource representing a code file and the second one is an acceptor on which the addPosition method should be called. The addPosition method associates a text style with a code fragment and has three parameters. The first one is an offset of a code fragment. The second one is a length of a code fragment. And the last one is an identifier of a text style. However, theaddPosition method allows for specifying a code fragment by a certain offset and a certain length, it is not yet known how to find out these values. For this case there is a node model which is AST-based structure of the code consisting of elements called node. The node holds information about the offset and the length of a code fragment which are related to the node and further information about a corresponding grammar element that are necessary for creation of decisions whether to associate the code fragment with the text style or not. The node model can be obtained from the mentioned resource. The whole element is represented by one root node and it is possible to get to descendants by given getters. Although, the node model may at first glance look like a typical AST, but it is not due to some details, which are not important in this situation. The following listing represents possible implementation of the ISemanticHighlightingCalculator interface.

Listing 3

An implementation of the ISemanticHighlightingCalculator interface that associates a text style dedicated for keywords with names of all methods.

public class ExampleSemanticHighlightingCalculator implements ISemanticHighlightingCalculator
{
    @Override
    public void provideHighlightingFor(XtextResource resource, IHighlightedPositionAcceptor acceptor) {
        // It gets a node model.
        INode root = resource.getParseResult().getRootNode();
        for(INode node : root.getAsTreeIterable()) {   
            EObject grammarElement = node.getGrammarElement(); 
            if(grammarElement instanceof RuleCall) {
                RuleCall rc = (RuleCall)grammarElement;
                AbstractRule r = rc.getRule();
                EObject c = grammarElement.eContainer();
			   
                // It tests whether the node represents a name of an element (class, method, parameter).
                if(r.getName().equals("ID") && c instanceof Assignment && ((Assignment)c).getFeature().equals("name")) {
                    INode p = node.getParent();
                    if (p != null && p.getGrammarElement() instanceof RuleCall) {
                        rc = (RuleCall)p.getGrammarElement();
                        AbstractRule r = rc.getRule();
			   
                        // It tests whether the parent node represents a method.                        
                        if(r.getName().equals("Method")) {
                            acceptor.addPosition(node.getOffset(), node.getLength(), ExampleHighlightingConfiguration.METHOD_ID);
                        }
                    }
                }
            }
        }
    }
}