Nested comments grammar/lexer

[CertHaramas]

Hello,
I'd like to define a lexer which would recognize "nested comments" or "nested conditional sequences". Suppose having C code (in a .c or .cpp file):

#if 0
  This is a comment.
#endif

#if 1
  This is not a comment.
#else
  This is a comment.
#endif

#if 0
  This is a comment.
#else
  #if 1
    This is not a comment.
  #else
    This is a comment.
  #endif
#endif

How should I define the lexer (grammar) to be able to recognize text "This is a comment." as a comment and to apply a style (e.g. italics) for that? I made some attempts but no success...

TIA.

[orbitalquark]

You could try playing with the existing #if 0 rule in lexers/ansi_c.lua: https://github.com/orbitalquark/scintillua/blob/44d0dd1d7d77b5391c0f4044b8853bbd511dc339/lexers/ansi_c.lua#L40

Perhaps something like lexer.range('#if' * S(' \t')^0 * '0' * lexer.space, P('#endif') + '#else'). I haven't tested it though. Note that that would only catch the 'if' block. I don't know how to write a simple pattern that would match the 'else' block. You'd probably have to use a function pattern, kind of like how the bash lexer tries to match heredocs: https://github.com/orbitalquark/scintillua/blob/44d0dd1d7d77b5391c0f4044b8853bbd511dc339/lexers/bash.lua#L26-L34

(e.g.

a=<<EOF
foo
bar
baz
EOF

)

[CertHaramas]

Hello,
is there a way to pass a whole grammar to the lexer, and to mark various parts of the grammar as a token (to be colored by assigning a style to it later), something like (not working):

local lexer = require('lexer')
local token, word_match = lexer.token, lexer.word_match
local P, R, S = lpeg.P, lpeg.R, lpeg.S
local V, B = lpeg.V, lpeg.B

local lex = lexer.new('ansi_c')

local if_kw = P"#if"
local else_kw = P"#else"
local endif_kw = P"#endif"

local if_true = if_kw * lexer.space ^ 0 * P"1"
local if_false = if_kw * lexer.space ^ 0 * P"0"

local G -- Grammar
        = P{
  if_blk = if_kw * (V"if_blk" + (1 - endif_kw)) ^ 0 * endif_kw,
}

Here, maybe I would be able to build the whole grammar which would recongnize nested "comments".
But how do I tell the lexer that a part of grammar - G.if_blk - is "something" which could have it's visual style? And how do I bind the grammar with the lexer?

[orbitalquark]

No, sorry, I don't think that's possible. If you want to use another grammar within a lexer, you'd have to use a child language and lexer.embed() (https://orbitalquark.github.io/scintillua/api.html#lexer.embed) with start and end rules. Embedding a language recursively is not supported. You'd have to lexer.load() (https://orbitalquark.github.io/scintillua/api.html#lexer.load) with an alternate name. It could get really messy.

[CertHaramas]

OK.
And one more question: does the lexer perform:
a) always whole-buffer parsing, or
b) a somehow reduced-buffer parsing, limited e.g. to the visible area?
Were a) be true, there might be a way to keep track of all "#if 0", "#if 1", "#else" and "#endif" of the buffer, and knowing this, I might be able to get back information about yes-/no-comment space to the lexer, so it could decide what style to assign to the currently parsed pattern. Or am I wrong?

[orbitalquark]

It's up to the editor to invoke the lexer with input (https://orbitalquark.github.io/scintillua/api.html#lexer.lex). Editors can do whole-buffer lexing, lexing by chunks, or lexing by lines. For example, Scintilla typically lexes from the beginning of the current line to the end of the document (though there are options to lex in chunks during idle time).

You can keep track of state per line using the lexer.line_state table (https://orbitalquark.github.io/scintillua/api.html#lexer.line_state). Normally editors would read from this state (e.g. the lexer marks a line as having some piece of useful information), but I suppose lexers could make use of this information to help perform lexing. I'm not sure how exactly, and I suspect there could be recursion or subtle runtime issues that would arise from this.

If all else fails, I suppose you could just override the lexer's lex() function to do your own parsing. (You ultimately just return a table of positions and styles to highlight as.) You give up Scintillua's convenience patterns and functions, though.