Include related tokens in expression object

[nickBes]

Hello :)
I've developed some utilities within an organization for providing context based autocomplete, quick actions, markers (etc), for a SQL code editor integrated with a data platform. The utilities were written in Typescript with antrl4 for trinosql dialect. However, with the growth of the project there are more services that require sql parsing and extracting syntactic metadata that is not dialect dependent. Which is the reason that this project has appealed to us.

In order to provide language based features, we need a way to find which tokens are included within an expression. This feature exists in antlr4 parser where each rule context (an equivalent for sqlglot expressions) has a start token and an end token. Unfortunately sqlglot doesn't have this ability yet.

An example for this use could be when we want to mark uses of select *, we'll do something like this:

markers = []

for star in expression.find_all(Star):
    star_token = star.token # can't do that yet in sqlglot
    markers.append({
        "start_column": star_token.col,
        "end_column": star_token.col + 1, 
        "start_row": star_token.row, 
        "end_row": star_token.row,
    })

I'd be happy to hear if someone knows how to implement this or if this issue is included in the future milestones of this project 😁

[tobymao]

it should be easy to put in an estimate token, but there’s no guarantee it would match the correct one. like for a select, it would probably match the last statement. there’s no plan to get it to be an exact match.