Preserve comments

[mdeland]

Is there anyway way to preserve comments in the output of the queryparser? It seems that comments aren't part of the pg parsetree, but it would be nice to be able to preserve their content/location somehow.

[lfittl]

@mdeland Thats a good question - I think it would unfortunately be quite invasive to patch up the parser.

It seems PostgreSQL treats comments as whitespace in the lexical scanner: https://github.com/postgres/postgres/blob/master/src/backend/parser/scan.l#L201

Since pg_query just takes that same code, it has to live within those constraints. Whats the use case for you?

I assume we might actually have an easier time using a regular expression to find the comments (and we could look into making that easier, if helpful).

[mdeland]

Yeah, I assumed the scanner was probably just ignoring the comments since no internal PG node representation represents a comment. I am trying to use the output here to format sql, but it would be nice to preserve the comments interspersed where they fall within the query.

[lfittl]

@mdeland Yeah, that makes sense.

I don't think there's an easy way to support this as it stands - I'd be willing to integrate a small parser patch, since I'm already carrying one to support parsing ? replacement characters (https://github.com/lfittl/libpg_query/blob/9.5-latest/patches/01_parse_replacement_char.patch).

Do you think you'd be interested in diving into the C code yourself and see if you can make this work? (I'm too short on time right now to do so)

[mdeland]

I'll take a look at it - I'm not familiar with the bison (?) parser framework but could be interesting.

[lfittl]

@mdeland Yeah, I'll warn you though - it can be a bit painful :-)

Note also that we want to avoid making any invasive changes (i.e. changing the way things work too much), since this needs to be synced with every new PostgreSQL release.

In any case, have fun!

[lfittl]

Just adding a note here that pg_hint_plan also needs to parse comments, so we may be able to borrow some code: https://github.com/ossc-db/pg_hint_plan/blob/c7888a113ac952c39af91c2d448f864d06e73837/pg_hint_plan.c#L1899

Also note that pg_hint_plan is licensed under BSD-3-clause, so if we do end up using some of their code we should attribute accordingly.

[lfittl]

Closing this, since this has been implemented in the scanner method released in pg_query 2.0:

https://github.com/pganalyze/libpg_query#usage-scanning-a-query-into-its-tokens-using-the-postgresql-scannerlexer

(note it doesn't really fit the logic to put this information in the parser, so you would have to use both the scanner and the parser method in some cases)