json-6 require not working?

[JakobJingleheimer]

Sorry, I'm not sure exactly what the problem is, but it appears json-6/lib/require doesn't work (and I'm seeing that require.extensions is deprecated).

When I try to import a .json6 file and run it with mochajs/mocha (which I think uses standard-things/esm), I get the following error:

TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".json6"
for …/dummy-data.json6
    at new NodeError (node:internal/errors:259:15)
    at Loader.defaultGetFormat [as _getFormat] (node:internal/modules/esm/get_format:65:15)
    at Loader.getFormat (node:internal/modules/esm/loader:101:42)
    at Loader.getModuleJob (node:internal/modules/esm/loader:230:31)
    at async ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:53:21)
    at async Promise.all (index 0)
    at async link (node:internal/modules/esm/module_job:58:9)

I've included json-6/lib/require in mocha's --require and confirmed the file is executed; however, the function set for '.json6' never executes.

I see that there is specific mention of JSON6 in esm's readme, so I tried adding the cited config to my package.json, but to no effect:

"esm": {
  "JSON6": true
}

[d3x0r]

Okay.

I updated my one-shot test batch file to do this..

mocha --require chai/register-expect --require ../lib/require.js %*

I added tests test/1.09-require.js and test/1.09-require.mjs ; the first works without issue.
the second works when I manually run tests using the above command
the second fails with Unknown file extension error mentioned above, when run through npm...

but it works okay from the command line.

M:\javascript\JSON6\test>mocha --require chai/register-expect --require ../lib/require.js 1.0.9-require.mjs


  Added in 1.0.9 - require(esm)
    √ allows using require on extension


  1 passing (5ms)

I had to rename test/1.0.9-require.mjs to .fail to allow pushing... I'm not sure how to get it to work from npm... from just mocha it seems to work fine.

https://github.com/d3x0r/JSON6/tree/master/test renaming the require test to .mjs instead of .mjs.fail
then npm run test-lite fails.

> json-6@1.0.8 mocha-lite
> mocha --require chai/register-expect --require ./lib/require.js



  Added in 1.0.9 - require(cjs)
    √ allows using require on extension

  Added in 1.0.9 - require(esm)
    √ allows using require on extension
    1) allows using require on extension


  2 passing (14ms)
3:09 PM  1 failing
1) Added in 1.0.9 - require(esm)
     allows using require on extension:
      Uncaught TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".json6" for M:\javascript\JSON6\test\1.0.9-require.json6
    at new NodeError 

[JakobJingleheimer]

Hmm, if you're running it via mocha directly, I presume you've installed mocha globally? That could make a huge difference (and is unlikely to be an uncommon real-world scenario).

I get the same Unknown file extension error via npx mocha btw.

Preliminary results with $> NODE_ENV=test node_modules/mocha/bin/mocha are…promising (though not ideal): at least Mocha is complaining about actual errors now (previously it was erroring out before getting to them).

[d3x0r]

found deprecation
https://nodejs.org/api/all.html#DEP0039

v6.12.0, v4.8.6 | A deprecation code has been assigned.

that was a long time ago... could wish they included 'and has been replaced with....'

[d3x0r]

I'd maybe recommend copying the snippet of code from require.js that loads and parses the file and returns an object.... as a workaround.
I have another project that has a custom import() function that handles it; but I've not found how to hook into the module loader chain of node.

[JakobJingleheimer]

I took a quick gander at @babel/register hoping there was something quick I could glean and incorporate from that, but it's much more involved than I was hoping ☹️

[d3x0r]

This is the module the error is thrown from... specifically the list of extensions internal - with no way to add another list
https://github.com/nodejs/node/blob/master/lib/internal/modules/esm/get_format.js#L17-L22

This has the resolve logic which I think prefilters '.json' ... it looks like it supports file:, data:, and node: URL resolutions for filenames... (not really any help)
https://github.com/nodejs/node/blob/master/lib/internal/modules/esm/resolve.js

[JakobJingleheimer]

I think the way to handle this now is: https://nodejs.org/api/esm.html#esm_hooks

Looking at get_format.js, I found a workaround to avoid the error: get_format.js#L65 add --experimental-specifer-resolution=node

This works the same for me as calling mocha directly (via node_modules/mocha/bin/mocha):

$> npm run test --experimental-specifer-resolution=node

(it also works when appended to the npm script)

[d3x0r]

that doesn't work for me... I didn't see any sign that using the other map would have helped any... that still doesn't relate to 'require.extensions' ?

(you mispelled specifier above :) and for a while node was saying it couldn't take that option)

> SET NODE_OPTIONS=--experimental-specifier-resolution=node && mocha --require chai/register-expect --require ./lib/require.js

node:internal/process/esm_loader:74

    internalBinding('errors').triggerUncaughtException(

                              ^



TypeError [ERR_INVALID_RETURN_PROPERTY_VALUE]: Expected string to be returned for the "format" from the "loader getFormat" function but got type object.

    at new NodeError (node:internal/errors:259:15)

    at Loader.getFormat (node:internal/modules/esm/loader:110:13)

[JakobJingleheimer]

I think it's not working for you because I didn't misspell specifer: it's misspelled in the source code (get_format.js line 65)

      if (experimentalSpeciferResolution === 'node') {

[JakobJingleheimer]

By the way, for the esm require, it looks like a custom transpiler loader is the way to go: https://github.com/nodejs/node/blob/master/doc/api/esm.md#transpiler-loader

That doc explicitly says require.extensions doesn't work for esm / import: https://github.com/nodejs/node/blob/master/doc/api/esm.md#no-requireextensions

[d3x0r]

okay so do I need to do a @rollup/json6-plugin etc ? babel?

[d3x0r]

that doesn't solve everything though - I do still read .jsox(json6) files for configuration files because of human editability... so it should really always be a dynamic sort of thing that wouldn't be included...

[d3x0r]

So I'm about to update the test to be 'success is to throw an error', and update the readme about limitations.

[JakobJingleheimer]

I think neither of those. Basically an esm version of the existing lib/require accomplished via that loader stuff. Then append whatever command with --experimental-loader json-6/lib/es-loader. Ex

// package.json
{
  "scripts": {
    "test": "NODE_ENV=test mocha ./src --experimental-loader json-6/lib/es-loader"
  }
}

[d3x0r]

This should basically work...
https://github.com/d3x0r/JSON6/blob/master/lib/import.mjs
--experimental-loader=./lib/import.mjs

updating the projects own build to include that and require causes it to fail to import 'node_modules/mocha/bin/mocha' because it doesn't have an extension.... but again was able to run the tests as a one-off with the extra flags. and this fails for me

SET NODE_OPTIONS=--experimental-loader=./lib/import.mjs && mocha --require chai/register-expect --require ./lib/require.js

However, the new loader should generally work... it worked as below...

This usage worked.

import {default as config} from './1.0.9-require.json6'
console.log( config );

This method also worked.

return import( "./1.0.9-require.json6" ).then( config=>{
			config.default //...

[d3x0r]

Revised the loader; works more as intended.
bumped version, should publish as 1.1.1; update includes json6 stringifier support...

The first other version took advantage that JSON6 is basically just JS, so I could just emit 'export default '; this uses JSON6 parser properly to decode that; adds JSON6 to the globalThis object though.

[JakobJingleheimer]

This looks great—thanks! I've got an unrelated ESM error currently that's blocking me trying this, but I'll get back as soon as I've cleared that.

[JakobJingleheimer]

Hmm, I'm getting an error:

$> npm run test

NODE_ENV=test mocha './src/test.spec.js' --no-warnings --es-module-specifier-resolution=node --experimental-loader=json-6/lib/import --experimental-loader=./esm-alias-loader.mjs


TypeError [ERR_INVALID_RETURN_PROPERTY_VALUE]: Expected string to be returned for the "format" from the "loader getFormat" function but got type object.

node v15.1.0
npm v7.0.8

json-6 v1.1.1
mocha v8.2.1

(esm-alias-loader.mjs is from ilearnio/module-alias#59)

Strangely, if I change the sequence of loaders (json-6's after alias-loader), json-6's loader appears to stomp alias-loader.

[d3x0r]

Not sure why I would 'stomp' the other... every operation checks for extension of '.json6' before doing anything custom, otherwise it calls the default functions passed.

experimental-loader=./esm-alias-loader.mjs

don't know where esm-alias-loader comes from... it doesn't seem to be part of module-alias

Edit:
Also none of the methods return just a string... return { format: 'module' }; is the return from getFormat...

[d3x0r]

You might check your version... I did bump the version to 1.1.1 (from 1.0.5) so the 1.0/1 change might be keeping you on a prior version.... that might have stmped everything.

[JakobJingleheimer]

@d3x0r don't know where esm-alias-loader comes from... it doesn't seem to be part of module-alias

me: (esm-alias-loader.mjs is from ilearnio/module-alias#59)

You might check your version... I did bump the version to 1.1.1

I did check the json-6 version (and posted it with a few other potentially relevant ones) 🙂
I used npm list --depth=0 json-6 to confirm (instead of just checking my package.json). I did add it as an edit to my post though, so it might not have been in the email GitHub set you.

[JakobJingleheimer]

Expected string to be returned for the "format" from the "loader getFormat" function but got type object.

The error says it's expecting a string but got an object.

I compared your getFormat function with the example from esm's doc, and they both return an object like { format: someString }. The main difference that seems potentially important is line 26:

	return defaultGetFormat(url,context );

it's missing a 3rd argument which is present in the example:

  // Let Node.js handle all other URLs.
  return defaultGetFormat(url, context, defaultGetFormat);

Your other functions include the 3rd default… argument and I see your code as a trailing space character after the 2nd arg, so I'm thinking you intended to include it?

RE esm-alias-loader getting stomped: the first thing that comes to mind is the json-6 loader doesn't have/export a resolve function (which would pass along whatever it doesn't understand, and is the only hook esm-alias-loader uses). Maaaaybe that's causing the stomping?

EDIT: I just noticed in the esm loader doc it says a resolve is required when using transformSource (which json-6's loader uses):

If this hook is used to convert unknown-to-Node.js file types into executable JavaScript, a resolve hook is also necessary in order to register any unknown-to-Node.js file extensions. See the transpiler loader example below.

[JakobJingleheimer]

Adding a resolve to json-6's import loader didn't address the potential stomping, and adding that 3rd defaultGetFormat argument to getFormat's else return didn't appear to affect anything either

code added

import fs from "fs"; // same as before
import url, { URL, pathToFileURL } from 'url';
import path from "path"; // same as before

const baseURL = pathToFileURL(`${process.cwd()}/`).href;

export function resolve(specifier, context, defaultResolve) {
	const { parentURL = baseURL } = context;

	const ext = path.extname(specifier);

	// Node.js normally errors on unknown file extensions, so return a URL for
	// specifiers ending in the json-6 file extension.
	if (ext === '.json6') return { url: new URL(specifier, parentURL).href };

	// Let Node.js handle all other specifiers.
	return defaultResolve(specifier, context, defaultResolve);
}

[JakobJingleheimer]

I added some logging to ems-alias-loader, and its resolve doesn't get called at all when json-6's is listed after it in the command (--experimental-loader=./esm-alias-loader.mjs --experimental-loader=json-6/lib/import.mjs), and they appear to get called in reverse order:

1. I found v14 regresses #16605: node --experimental-modules does not allow running files without an extension nodejs/node#33259 which seemed to suggest that --experimental-specifier-resolution=node causes the getFormat error
2. I removed that flag, which caused:
   • the --experimental-loader=json-6/lib/import to fail (it needed .mjs appended)
   • esm-alias-loader imports package.json, and that started failing (Unknown file extension ".json")
3. I updated json-6's loader to include .json extensions since it can technically work for that too (and it did)
4. json-6's loader was listed first, so I expected the above to JustWork, but no change: 🤯
5. I switched the sequence so json-6's loader came after, and now only json-6's loader executed (I put a throw at the top of esm-alias-loader.mjs and it didn't trigger)

I also tried adding --experimental-json-modules, but no change.

It can't be that only 1 loader is allowed because when esm-alias-loader is after json-6's, they both run.

[d3x0r]

Well I did ignore(remove) the resolve because it just ended up calling the default resolve - I want basically the default node file resolution... nothing custom to do with the path. (though I suppose that mod-alias uses that)

I wondered about that default function parameter... it could be that it's only allowed to have 1, and you need one that merges both together...

[d3x0r]

from node\lib\internal\modules\esm\loader.js

  async getFormat(url) {
    const getFormatResponse = await this._getFormat(
      url, {}, defaultGetFormat);
    if (typeof getFormatResponse !== 'object') {
      throw new ERR_INVALID_RETURN_VALUE(
        'object', 'loader getFormat', getFormatResponse);
    }

getting the format doesn't chain... the defaultGetFormat is just the internal version, not some other loader...

[d3x0r]

node module: ESM loaders next steps
nodejs/node#36396

This is an experimental state, so probably it can be fixed.

[JakobJingleheimer]

@d3x0r so actually, you comment above about how json6 is effectively just javascript gave me an idea, which I just verified—this is actually the only necessary piece for importing json6 files:

// jsonlike-loader.mjs

import path from 'path';

const moduleLikeExts = [
	'.json',
	'.json5',
	'.json6',
];

export function getFormat(url, context, defaultGetFormat) {
	const ext = path.extname(url);

	if (moduleLikeExts.includes(ext)) return { format: 'module' };

	return defaultGetFormat(url, context, defaultGetFormat);
};

$> NODE_ENV=test mocha './test.spec.js' --no-warnings --es-module-specifier-resolution=node --experimental-loader=./jsonlike-loader.mjs

This tells node to handle the files as if they were normal modules, and voila! No getSource, transformSource, etc. You only need to tell Node the file is a module.

[d3x0r]

Okay1) I'm not sure why that would really behave any different; other than not intercepting the getSource or Transform calls and doing things with them.... they would only be used for .json6 files (I could add .json5 too).
Could probably use format:'js' also... but that might not enable import/export ability in the loaded modules.

But, that doesn't help https://github.com/d3x0r/jsox which deviates from JS and requires its own transform...

[d3x0r]

https://github.com/nodejs/node/blob/master/lib/internal/process/esm_loader.js#L42

node only uses one of the command line options (first? last?) It's right now even if you specify more than one it only uses one...

this routine should at least be updated to use all --experimental-loader options.... would seem to me, that even if this was the simplest method, it still wouldn't work with other loaders....