Lots of whitespace differences introduced by endo bundler

[erights]

On the right is the original lockdown-shim.js. On the left is the corresponding portion of lockdown.umd.js aligned for a visual line by line comparison. There are lots of little divergences. I do not remember seeing these divergences when we were using some previous bundler (rollup?).

These differences will impede the debugging experience. It means that at least the columns are different, and so we cannot simply sourceURL the original source file to appear in the debugger in lieu of the bundled source.

Related issues:

• Using XS native Compartments for Agoric Vat Workers #400

[kriskowal]

We discussed this and resolved that this probably can be addressed by patching the babel-generator code that catches up with the cursor of the next written AST, such that it advances the column in addition to the line.

[kriskowal]

We also discussed trying to preserve untouched subtrees of the AST by reusing the original source. This is possible but more invasive. It requires annotating which nodes were “touched” and modifying bundle-source to attend to that detail.

[pyramation]

I would encourage the use of source maps vs. expecting babel to allow customization of whitespace:

https://github.com/evanw/node-source-map-support
https://github.com/chocolateboy/babel-plugin-source-map-support

If you integrate source maps correctly, all debugging should theoretically use the source code files instead of the transpiled code in stack traces.

If possible, I'd encourage and suggest finding a path that is not dependent on comments or whitespace output of babel. As far as I understand the deparser (babel/generate) or the transformer, whitespace (and comments) are two areas that I've empirically found that babel doesn't address perfectly.

Given the dependency on babel (which is the industry standard), if sourcemaps don't solve this issue, I'd suggest making an issue in babel. Looks like it has been brought up before as early as 2015, but has been closed babel/babel#497

[pyramation]

also, looking more deeply into the original babel issue I found a linked issue that helped me find this library: https://github.com/js-cst-tokens/cst-tokens

they are not using ASTs, but rather, CSTs which appear to preserve whitespace. I would still think sourcemaps are the better path, but figured I'd share this link.

[erights]

Thanks! If the cst works (it looks like a great find!), I prefer it to the sourcemap path. With sourcemaps, the programmer is often looking at their original source text which can differ from the code actually being executed in devious ways. Even with the cst, I'd expect programmers to often be looking at their original source test, via sourceURL rather than text being executed. But, since these are supposed to differ only on import and export,

• programmers can avoid the sourceURL view and have essentially the same experience
• differences on lines other than import and export can be trivially caught by a diff.

CST FTW! Thanks much. I hope this works!

All that said, if we do sourcemaps, we can easily write a checker to ensure that the pre vs post source text differs only on import, export, and whitespace. We should probably do that anyway. Once that's verified, debugging through a sourcemap view would be safe.

[erights]

Actually, the current babel output seems to differ on more than whitespace, for example unnecessary parens, so that a correspondence verifier for babel output is no longer such a trivial exercise. This pushes harder for us to convert to CSTs.

[pyramation]

makes sense! From what I can tell, the intention of CSTs is more aligned with your goals.

From what I can see on the readme, the flow should be somewhat similar to babel. So if you guys have a babel transformer for import and export, it should be decently portable to take a crack at using CSTs via these newly discovered libraries.

Keep me in the loop!

[erights]

makes sense! From what I can tell, the intention of CSTs is more aligned with your goals.

From what I can see on the readme, the flow should be somewhat similar to babel. So if you guys have a babel transformer for import and export,

We do!

it should be decently portable to take a crack at using CSTs via these newly discovered libraries.

Keep me in the loop!

Will do. Much appreciated.

[erights]

See also #1044

[kriskowal]

I’ve threaded source maps into a per-user cache as of #1689. This is as much as we can do to improve debugging a module through the module-to-program transform in our shim until we hear back from @nicolo-ribaudo.

[erights]

Another one. In order to send a breakpoint in the handle method shown in my source on the left panel, I waited until the debugger took me into the corresponding bundled file shown on the right, line them up (which is why there's a very long first line scrolled off the top on the right). Fortunately, the line numbers do correspond accurately, so with these lined up I can interact with the debugger on the right pane while looking at the left pane.

Thus, the difficulty due to the loss of horizontal whitespace can be coped with awkwardly. What's harder is getting to the code in the right hand pane. The only way I've done it is to wait until I can step into it in the debugger, which requires following execution that reaches there somehow. This does not enable the normal debug experience of navigating to a position in a source file and placing a breakpoint ahead of time.