question: support/implement positions for dst nodes and decorations?

[lwsanty]

could you please help me to figure out one thing?
in the docs here

dst/dst.go

Line 24 in e3c2080

// All nodes contain position information marking the beginning of

it is said that

// All nodes contain position information marking the beginning of
// the corresponding source text segment; it is accessible via the
// Pos accessor method. Nodes may contain additional position info
// for language constructs where comments may be found between parts
// of the construct (typically any larger, parenthesized subpart).
// That position information is needed to properly position comments
// when printing the construct.

but at the same time the interface and it's implementations do not contain this method at all.
I have a doubt if dst nodes contain positions at all

If there're no positions provided, what approach would you suggest in implementing them?

[dave]

Hi @lwsanty, yes - this comment is incorrect. This file was forked from the ast.go in the standard library and it looks like I forgot to update this doc. Since in dst, nodes can be moved around the tree, the position had to be removed completely. To find the position for a dst node you could convert the file to ast and use restorer.Map to find the corresponding ast node and read the position.

[lwsanty]

@dave using restorer.Map seems to be a good idea.
However will it give an ability to get positions of decorations also?

I was considering running file restorer and mapping the current cursor position to dst node and decoration . So each time after manipulation on dst tree were performed, we can "restore" dst nodes and decorations positions.

Could you please tell me if it makes sense?

---

The background of my research is to check if it's possible to use dst in go-driver
The idea is to perform some transformations on go ast, but positions are dropped, because they are static and not valid anymore after the transformations.

If positions were preserved or could be easily restored using dst that would be a great enhancement for go-driver.

[dave]

When the restorer runs, decorations are converted to comments, which do have positions I think, however I don't think there's any mapping from decorations to comments, so locating the comment corresponding to a particular decoration might be problematic.

The other problem you might come across is that the dst decorate / restore functions are probably very slow. At no time did I ever think about optimising them for performance and I assume they're very slow compared to the ast parser / printer.

[dave]

I'm still a little hazy on what exactly you want to do... Perhaps you can explain some more and maybe there's an easy way that I'm not thinking of right now...

[lwsanty]

@dave eventually I wanted to implement a refactoring tool that does the next steps

1. accepts desired input and output code snippets
2. converts snippets to AST nodes
3. generates required transformation using DSL
   and I actually was considering using DST nodes for step 2 with an aim to preserve positions(or with ability to reconstruct them), Currently they're skipped in my prototype because the DSL is unable to detect a suitable transformation.

The DSL requires an exact match on nodes, thus we currently drop positions before transforms. This works, but breaks comments since they are "free floating" and rely on positions.

So I was considering using DST instead of a native Go AST. This ways positions will be implicitly encoded in the decorations and comments will be placed correctly. The Babelfish DSL can use DST's nodes with no issues, however to integrate the DST into Babelfish we'll need to expose position info for each node. It's required for other use cases not related to my prototype or transformations.

[dave]

The problem is that dst doesn't really care about the position (e.g. byte offset in the file) of the nodes. It does assign positions to the ast nodes in the restore process, but these are not actual correct byte offsets in the file. It just assigns positions to get the comments and nodes in the correct order. When the resultant ast file is rendered and formatted, all sorts of whitespace is added.

If you need the position of a dst node or decoration, you'd need to restore it to ast, render it with the go formatter, and parse the resultant source file again. Obviously not ideal at all. But if you think how complicated the go formatter is - adding whitespace to align lines etc... it's the only way I think.

[Julio-Guerra]

That would be indeed useful knowing the original position when the source file was parsed. I also would like to have it, in my case in order to use line directives.

[dave]

If you want the position in the input file of an AST node that corresponds to a DST node, then I believe restorer.Map will help... you can look up the AST node and use the Pos() method?

[budiadiono]

Thanks @dave for the clue! The restorer.Map.Ast.Nodes did the job!, here's the working example:

func getPos(restorer *decorator.Restorer, structType *dst.StructType) *token.Pos {
  astNode := restorer.Map.Ast.Nodes[structType]
  if astNode != nil {
    pos := astNode.Pos()
    return &pos
  }

  return nil
}