Add debugging facilities to RuleNode/TokenNode exposed via N-API

[Xanewok]

Closes #466

JSON serialization

Adds the toJSON method to the RuleNode/TokenNode classes with the comment stating that it's unstable and for debugging purposes only. I explicitly didn't type it (it's a string) as I didn't want to publicly encode any assumptions about the shape of the data.

We can discuss further what we want to explore as part of having pure data external interface (JSON) for Slang, whether that's exporting or importing JSON, but I'd consider this a separate issue as it requires more thought and design; the original issue, from what I understand, was primarily about improving the debugging/inspection experience.

Debugger

Adds hidden getter properties that is only eagerly evaluated by debugger; since it's a getter function, it shouldn't add overhead to the runtime but I have to admit, I didn't double-check that with numbers. It adds both __children and __text, allowing to quickly explore the tree structure recursively and also the enclosing text scope of the node:

vscode-js-debug (built-in to VS Code)

Chrome DevTools for Node

[changeset-bot]

🦋 Changeset detected

Latest commit: eb63cda

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@nomicfoundation/slang	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Xanewok]

Custom formatting

I also went down the rabbit hole of trying to leverage custom inspection formatters like nodejs.util.inspect.common property for objects, here's the branch: https://github.com/Xanewok/slang/tree/cst-napi-debug-inspect.

In DevTools, we'd need to modify the global window.devtoolsFormatters, so without a dedicated extension this seems like a no-go.

In vscode-js-debug, they allow to customize the default header for an object but with caveats:

• only parent and direct children get their custom properties rendered (https://github.com/microsoft/vscode-js-debug/blob/91a3e5417cf79d4a05b91f1c700067caa8181cb5/src/adapter/variableStore.ts#L789).

  This means that it has limited use for recursive/deeply nested structure like our syntax tree. You'd have to call inspect(...) directly in the debug console on a more deeply nested object to preview its custom header or add it directly under the Watch section (e.g. ruleNode.children()[3].children()[5]...).

• a regular toString() class method can be used to preview the custom header but only if it's not exposed via N-API and defined as a regular JS code (!) (https://github.com/microsoft/vscode-js-debug/blob/91a3e5417cf79d4a05b91f1c700067caa8181cb5/src/adapter/templates/getStringyProps.ts#L115).

  They employ it as a heuristic not to stringify Node built-in objects like errors etc. but this makes it impossible for us to define a toString and use it ourselves.

• using a [Symbol.for("nodejs.util.inspect.common")] or [Symbol.for("debug.description")] (introduced by vscode-js-debug) (https://github.com/microsoft/vscode-js-debug/blob/91a3e5417cf79d4a05b91f1c700067caa8181cb5/src/adapter/templates/getStringyProps.ts#L7-L12) class method property would work, however it is currently not possible to implement a property that uses a Symbol as key in napi-rs; this is not yet implemented.

Future work

Having above in mind, we could, with increasing level of work required:

1. always inject the custom formatter ourselves at Rust -> JS conversion (what's implemented in my branch above)
   a. wasteful; it'd be better to define it once at the class prototype level when registering the Node native addon rather than for every node conversion
   b. hacky; that's quite a bit of explicit lower-level N-API machinery involved
2. Implement the support upstream for custom formatters or Symbol-keyed properties
   a. more optimal and declarative
   b. may be more helpful when printf-style debugging with util.inspect
   c. however, still not as useful in the actual debugger, as the custom preview is limited to depth = 1
3. Somehow convince the folks responsible for vscode-js-debug to allow for stringified props of arbitrary depth
   a. there's probably more process and back-and-forth involved
   b. they probably added it for a reason (performance?); we'd probably have to opt-in in yet another way to this behavior
   c. however, the end result would resemble our interactive CST snapshots, which would be great for exploration and debugging

Overall, I think that just having the hidden, eagerly evaluated children and unparsed text vastly improves on what we had before and having the custom formatting would be a great bonus but is not required for good enough debugging story.

I would probably not pursue the remaining options for now as we have enough to work on in separate issues but if the napi-rs/vscode-js-debug support story changes, it'd be good to revisit this in the future.

@AntonyBlakey @OmarTawfik what do you guys think?

[Xanewok]

I've dropped the "unstable" mention for the JSON serialization per our f2f and I've clarified that "eager evaluation" is specifically for the inspected parent rather than in general (make it more clear that it's not recursive in nature by default).

[OmarTawfik]

Left a few suggestions about missing catch_unwind and left over comment.

Otherwise, LGTM!