should one disable cache during development?

[dkebler]

Found myself updating a package I'm working on in another project and then wondering why the new code (although there in the node_modules) is ignored only to find that the folder has a node_modules/esm cache in it as well. When I delete that then happily the new code is loaded instead of I assume the code that was in the cache.

So two questions:

1. Is it recommended to turn off the cache in development using the env variable to avoid these frustrations? or is there some better way.
2. What will be the issues in production when updating dependencies. Won't the same thing happen?

I suppose if you could elaborate a bit on the cache and it use/purpose I'd understand better the implications of use. I saw some other issues about invalidating the cache(s) is that something that should happen automagically on dependency updates? Thx

[jdalton]

Hi @dkebler!

Our cache is generated by an mtime of the source + file path + loader config. If any one of them changes the old cache is discarded and a new entry is created. Do you know if an upgrade also updates the mtime of the files or if it preserves the original?

If it preserves the original that could be the source of the problem. If it's a common enough issue I can see about also adding the package version of the module to the cache hash.

[dkebler]

This maybe be happening to me because when I am fixing an issue with an underlying package I use npm link (or an rsync if remote) to get the working package code in place. Seems like that would not invalidate the cache. Still in this case what file path does it use? the main from package json? Most of my packages use an index.js file for the main and load exported/imports from the actual source files. Would that not cause invalidation? So yea maybe package version is a good idea to add in as well although sometimes in development I force push on the same version for small bug fixes :-

So per my question 1. If I set the env variable I won't have any of these issues. No caches, every run starts from scratch?

[jdalton]

This maybe be happening to me because when I am fixing an issue with an underlying package I use npm link (or an rsync if remote) to get the working package code in place.

If you could create a small repro repo of this scenario I can investigate and possibly add it to our test suite.

So per my question 1. If I set the env variable I won't have any of these issues. No caches, every run starts from scratch?

Disabling the cache disables the cache 🙃

[dkebler]

I'm trying to complete a project but this issue is important to me as well as esm sooo I will get around to making you a repo. Otherwise if it's not a big deal to invalidate the cache by package version (including a patch) I'd do it for no other reason than to cover all bases.

I am a bit confused about where the cache(s) are made. It is only supposed to be in the main node_modules/.cache/esm? Or does esm create one for each dependency? because that's what I notice. It's the later that causes the problem.

i.e. node-program-root/node_modules/mylibpackage/node_modules/.cache/esm

If I do an npm link mylibpackage then I have to delete that corresponding cache or I have the problem.

[jdalton]

@dkebler

I am a bit confused about where the cache(s) are made. It is only supposed to be in the main node_modules/.cache/esm?

Caches are made at package boundaries where esm is used unless otherwise customized by the cache option. So there can be multiple .cache folders. Caches are based on the resolved filenames.

[ghost]

Hi @jdalton,

cc/ @dkebler

I just found myself in a very similar cache invalidation problem. My development workflow is

1. develop the JS in a local project folder (using TypeScript but that's beside the point)
2. npm pack the folder (e.g. to myproject-0.1.0.tgz)
3. npm install -g myproject-0.1.0.tgz
4. test things out and then npm uninstall -g myproject
5. repeat from 1)

The package I'm developing is intended to be installed globally so that's why this particular workflow.

Anyway problem is after step 4 npm uninstall -g leaves .cache/esm directories intact in the global lib/node_modules/myproject/node_modules tree which aside from cache files is empty.

After making code changes, repacking and reinstalling globally, test runs do not always reflect changes in the code. It was only after 2 days of frustration that I discovered that the cache files where not being removed by npm uninstall -g so to really and truly uninstall I now do this to make sure that updated code gets to run:

npm uninstall -g myproject
rm -rf ~/lib/node_modules/myproject

(My global node lib is in my home folder)

Aside from this tearing-my-hair-out episode, I fully appreciate the enormous amount of work you have put into esm so thank you very much!

Oh, also when you say as above

Disabling the cache disables the cache

do .cache/esm files still get created anyway even if only for temporary purposes?

[jdalton]

Thanks @indiescripter!

It looks like npm pack does change the mtime to Oct 26 01:15:00 1985.
I'll patch to account for this.

do .cache/esm files still get created anyway even if only for temporary purposes?

Nope.

Update:

Patches 51d31ef, 0da1bf4;

Update:

esm v3.2.21 is released 🎉

Update:

Welp, touching files broke people (#771) so I had to revert that. Looks like for your scenario disabling the cache with the environment variable or through config is the way to go 😄

Update:

esm v3.2.22 is released 🎉

[ghost]

Thanks for trying that fix @jdalton . Sorry it didn't pan out.

[dkebler]

Disabling the cache disables the cache upside_down_face

"ESM_DISABLE_CACHE=true nodemon -r esm ./"

caches still being made. Docs do not indicate value for this env var.

[jdalton]

Any truthy value should do. See

esm/src/caching-compiler.js

Line 185 in 91a154b

dirty = !! parseJSON(getEnv("ESM_DISABLE_CACHE"))

[dkebler]

ok so the cache should be disabled per the env variable I set but it's not.

Even if I delete all the node_moudles/.cache folders applicable and rerun I'm still running some cached code.

So either

1. I'm not really turning off/deleting all the esm caches
2. There is some other non esm cache being used?

wish I could track down why as it's making it hard to work on a dependency.

[jdalton]

The esm loader stores it's cache in a node_modules/.cache/esm folder. Others like ava, babel, and nyc use different folders within node_modules/.cache/.

[michaelfig]

The fs.stat data on all my npm installed modules is something like:

{ ...
  "atimeMs": 1563058196864.3647,
  "mtimeMs": 499162500000,
  "ctimeMs": 1562991851221.0933,
  "birthtimeMs": 499162500000,
  "atime": "2019-07-13T22:49:56.864Z",
  "mtime": "1985-10-26T08:15:00.000Z",
  "ctime": "2019-07-13T04:24:11.221Z",
  "birthtime": "1985-10-26T08:15:00.000Z"
  ...
}

Could you maybe use the following heuristic?

const aLongTimeAgoMs = new Date('1999-01-01T00:00Z').getTime();
const stat = fs.statSync(moduleFile);
const cacheTime = stat.mtimeMs > aLongTimeAgoMs ? stat.mtimeMs : stat.ctimeMs;

I'm having a hard time explaining to everyone on my team that we need to do find . -name .cache -print0 | xargs -0 rm -rf && npm install ... everytime we need to install an updated module.

Otherwise, thanks so much for esm, it's generally making my life easier,
Michael.

[jdalton]

@michaelfig I'm up for trying it out.

[michaelfig]

@jdalton, would you like a PR, or to do this your own way?

[jdalton]

@michaelfig A PR would be super helpful. Thank you!

[celeduc]

I have the same (or a related) issue in @lorena-ssi/wallet-lib which I work around with this pr; the problem can be seen in v1.1.0 of that repo.