AST nodes with loc and/or range fields.

[knightedcodemonkey]

It would be great to have AST nodes with a loc and/or range fields.

Here's some context: #10139 (comment)

Possibly dependent upon #9705 being completed first.

[alexander-akait]

@Boshen sorry for ping, I am interesting in it too, trying to integrate oxc-parser into webpack, and loc is a blocker, thank you

[overlookmotel]

We can and should support this, but we should probably complete #9705 first. Making this change will hamper progress on TS-ESLint AST alignment otherwise.

I think parseSync and parseAsync should have 3 options to control location information:

{
    startEnd?: boolean, // Default `true`
    range?: boolean, // Default `false`
    loc?: boolean, // Default `false`
}

Defaults match current behavior, so not a breaking change.

range is easier to support than loc because it's the same as start+end, just in a different form.

[alexander-akait]

@overlookmotel yeah, I understand this, I wanted to do a fairly powerful testing through many webpack tests and places, but without these things, especially without loc, it is not possible, will be wait updates

[overlookmotel]

More testing would be very welcome! Thank you.

On TS-ESLint alignment, in 2 weeks, we've gone from 1% of tests passing to 75%. So hopefully it won't be too long until #9705 is complete, and then we can look at this. By all means please get involved in the effort on #9705 if you'd like to. More hands will get us to the finish line faster.

[overlookmotel]

#11728 will cover adding range.

Next steps - range

After it's merged, we'll need to:

• Adapt ESTree conformance testing to check range field is present and correct in all AST nodes (it'd be easy to forget it in some of the types which have custom ESTree implementations).
• Support range field in raw transfer too.
• Update TS type definitions to include range field.

loc

Unfortunately I think supporting loc is more tricky, due to UTF-8 / UTF-16 offsets.

Here are the steps I think we'd need to take to support it:

• Move start, end and range to be the last properties in each node object.
• Adapt acorn-test262 to do the same, so snapshots in conformance test have these properties in right order.
• Remove the extra AST pass converting offsets UTF-8 -> UTF-16. Convert during serialization instead.
• Do the same in raw transfer.
• Alter the offsets translation table to also represent line breaks.
• Use the new translation table to also convert UTF-8 offset -> line + column during serialization.

The last 4 steps are non-trivial.

I have only a vague idea of what a new translation table would look like. Ideally we probably want a single translation table that performs both UTF-8 to UTF-16 offset conversion, and UTF-8 offset to line / column conversion - rather than 2 separate tables.

Why do location properties have to be last?

Conversion is far more efficient if you convert offsets in strictly ascending order (i.e. source code order).

The current converter (generated code, codegen) works around various kinks in the AST to ensure this is the case.

But generally speaking, you need to convert the start offset of parent first, then all that nodes's children, and then finally the parent's end.

e.g. for x + y, order of conversion is:

• start of BinaryExpression.
• start of Identifier x.
• end of Identifier x.
• start of Identifier y.
• end of Identifier y.
• end of BinaryExpression.

Because JSON is constructed as a stream, end property must be the last property you output, because you don't know what it is until after all the children have been serialized.

{
  "type": "BinaryExpression",
  "left": {
    "type": "Identifier",
    "name": "x",
    "start": 0,
    "end": 1,
    "range": [0, 1]
  },
  "operator": "+",
  "right": {
    "type": "Identifier",
    "name": "y",
    "start": 4,
    "end": 5,
    "range": [4, 5]
  },
  "start": 0,
  "end": 5,
  "range": [0, 5]
}