caching related bug with possibly babel/register

[dnalborczyk]

unfortunately I have nothing really actionable, just assumptions. I'm trying to re-create the bug for several months now, with no success. :/

I'm also struggling on how to describe the bug: once in a while, depending on the order of certain actions, I get a SyntaxError, followed by a ReferenceError. The only way to move on coding is to delete the cache: ./node_modules/.cache (which includes the esm as well the babel cache).

I only know it happens when I change an export, e.g. named to default, but forget to also change the consuming import, (e.g. from named to default)

when I run the code, I correctly get the SyntaxError:

file:///path_to_project/some-file.js:1
SyntaxError: ES Module 'file:///path_to_project/some-path/index.js' does not provide an export named 'foo'
    at Module._compile (file:///path_to_project/node_modules/pirates/lib/index.js:91:24)
    at Module._extensions..js (internal/modules/cjs/loader.js:665:10)

though when I correct the importing consumer of that module, and run again, I get the following ReferenceError:

ReferenceError: _dbb‍ is not defined
    at Object.<anonymous> (/path_to_project/some-path/index.js:3:1)
    at Module._compile (/path_to_project/node_modules/pirates/lib/index.js:91:24)

from here on, the only way to keep coding is to delete the formerly mentioned cache.

as a side note, if I un-correct the last import (without having deleted the cache yet), I get the same SyntaxError again. and correcting it, I get the same ReferenceError again. I can swap back and forth.

I also checked, the _dbb‍ reference exists in both caches, esm as well as babel, but I'm thinking that it is an older caching value, as I'm assuming it shouldn't exist at least for the second run. the _dbb‍ reference exists in both caches in both cases (SyntaxError and ReferenceError)

I also tried to duplicate the project in order to try some things out. unfortunately that seems to render the cache invalid, and creates a new one, and the error goes away. I suppose because the path (and therefore some caching hash) is changing.

not sure if any of this helps in any way. I'm just guessing that esm or babel is using a stale cache for the second run. I'm also not quite sure if babel's involvement is required.

[jdalton]

It's @babel/register's cache for sure. We have a check in esm to try to clear it. They cache all of their generated source in a big json file. You can confirm this by disabling the cache (by config) for @babel/register.

We try to detect dirty caches when erroring, so I might get more aggressive with that.

Update:

Patched 7d74d00

[dnalborczyk]

just tried your suggestion, when I disable the cache with BABEL_DISABLE_CACHE=1, I get the following:

import * as foo from './foo'
^^^^^^

SyntaxError: Unexpected token import
    at new Script (vm.js:51:7)
    at Module._compile (/path_to_project/node_modules/pirates/lib/index.js:91:24)

when I delete just the babel cache (and leave the esm cache alone), the same exception remains.

[jdalton]

Once you disable their cache you should clear everyones and try again. Also if you could produce a repro I could double check that our @babel/register cache clearing is kicking in.

[dnalborczyk]

yeah, it seems the issue might be babel's cache. I enabled the cache again (just for babel), and the reference to _dbb‍ went away (in babel's cache, but remained in esm's cache), and everything worked again.

I'm trying to come up with a repro for some time now, but had no success yet. :)

one other possibility would be to code with both caches disabled, but part of me is thinking that would defeat the purpose.

[dnalborczyk]

oh, wow, already fixed!!

thanks a bunch @jdalton !!

I'll report back if I ever see the bug again ...

[jdalton]

The patch I did was to just be a bit more aggressive in marking both cases as dirty. So no guarantees.

[dnalborczyk]

yeah, I understand, no problem. 👍

[jdalton]

@babel/register is agressive with its cache writes. It does one on startup and one on shutdown. So we handle cleanup in 2 phases. During runtime we detect a syntax error in ESM and then mark the cache as dirty. On shutdown we create a .dirty file and delete all of our cache files. Now on the next startup, since esm should be loaded before @babel/register, we detect the .dirty file and clear @babel/register's cache (before it starts up). So then we both should start with fresh caches.

Update:

Ah found the problem! When I renamed @std/esm to esm I forgot to update the @babel/register lookup path since it did ../../ to walk back from esm then @std.

Patched 01f3df3.

So the issue was our cache clearing code was firing but not clearing the @babel/register cache.

[dnalborczyk]

ah, super nice!!

I was just thinking what to write as response to the other comment, while you already found and fixed the issue ... again! 😆

[jdalton]

v3.0.18 is published 🎉