Bug report: Master branch fails to build with fresh pull

[dgotrik]

Doing a fresh clone of the repository, the application fails to generate OperationConfig.json and OpModules.mjs and does not serve content.

Example

[GCHQ77703]

Can't reproduce on a Mac, following commands were run:

$ git clone https://github.com/gchq/CyberChef NewCyberChef
$ cd NewCyberChef
$ npm i
$ grunt dev

Which worked. That looks suspiciously like Cmder though, so are you on Windows?

[dgotrik]

I am on Windows, node version is 8.9.1. I haven't tried it on Linux or Mac yet.

The commands I ran were the exact same as yours, and I made sure write permissions were set correctly.

[mattnotmitt]

Could you try the solution to #335? Looks somewhat similar to what was going on there.

[GCHQ77703]

Ah yes, just tried it out. Can indeed reproduce the issue on Windows using both Node v8.9.1 and v10.4.0. It appears the Gruntfile was specifically built for Linux systems only, noting the following lines:

https://github.com/gchq/CyberChef/blob/master/Gruntfile.js#L379-L388

@artemisbot tried that, doesn't seem to fix the issue. I think the reason it's getting confused is because the configuration file just isn't there, the generation command isn't working.

[mattnotmitt]

Ah right. In the meantime perhaps you could try using the LXSS/WSL? I haven't worked on CyberChef on Windows in a while, and when I have in the past it's been using that.

[GCHQ77703]

You can get past the initial compilation issue by just doing it yourself, running the following commands from the command line:

mkdir src/core/config/modules
echo export default {};\n > src/core/config/modules/OpModules.mjs
echo []\n > src/core/config/OperationConfig.json
node --experimental-modules --no-warnings --no-deprecation src/core/config/scripts/generateOpsIndex.mjs
node --experimental-modules --no-warnings --no-deprecation src/core/config/scripts/generateConfig.mjs

And starting it with:

grunt clean:dev
grunt concurrent:dev

Then you run into two build issues:

Error: Couldn't find preset "es2015" relative to directory (root/node_modules/estraverse)
SyntaxError: root/node_modules/chi-squared/cdf.js: 'with in strict mode (7:2)

And also the fact that you have to run those commands every time you change the configuration. The latter is fairly easy to solve by chucking it in a batch file. The former issues, more unknown. I'm assuming some of our required modules might, in some way, require Linux themselves.

[dgotrik]

I can reproduce the same issue you're having @GCHQ77703 and attempted to get through it.

I can get past the first one (es2015) error by running
npm install --save babel-preset-es2015 babel-preset-stage-2

I got past the other error relating to strict mode by adding a plugin to babelrc (not sure if this is the correct route to take).
npm install --save @babel/plugin-transform-strict-mode
In .babelrc:
"plugins": [ "@babel/plugin-transform-strict-mode", ...

Then it complained that I needed Babel ^7.0.0.0, so I upgraded my babel.
npx babel-upgrade --write --install
Now it's a new error:

[GCHQ77703]

@dgotrik I wonder if, in your uplift to 7.0.0, you ran into some incompatibilities based on changing env presets:

babel/babel#6830
babel/babel#8195

But when I look at the migration documentation I don't see anything that looks a likely candidate to cause this issue and both of the issues linked above both seem open and in the RFC stage.

What version of Babel did you upgrade to? Is it a general availability release or a release candidate?

[GCHQ77703]

In .babelrc you can likely just change useBuiltIns from true to false without doing anything else. It's only used as an optimisation to speed up some browser load times by extending existing features instead of recreating them from scratch.

[dgotrik]

I am still working on getting things to work correctly in Windows. I verified it builds on Mac, but on linux (Ubuntu) I also had to do npm install babel-preset-es2015 babel-preset-es2016 babel-preset-stage-2 babel-preset-env

[animatorz]

In the Gruntfile.js, the generateConfig script is not running correctly on Windows due to the "\n" and the way the path is formatted.

As suggested in earlier replies you have to manually run the commands from generateConfig on the command line.
make the directory src/core/config/modules
create src/core/config/modules/OpModules.mjs and add the text export default {}; and add a carriage return
create src/core/config/OperationConfig.json" and add the text [ ] and add a carriage return
node --experimental-modules --no-warnings --no-deprecation src/core/config/scripts/generateOpsIndex.mjs",
"node --experimental-modules --no-warnings --no-deprecation src/core/config/scripts/generateConfig.mjs

You also need to remove the "clean:config" calls in the Grunt tasks for Prod and Dev so that when you build the project, you don't undo everything that you manually did.

After doing all that, you will get an error which was shown in an earlier post:
SyntaxError: root/node_modules/chi-squared/cdf.js: 'with in strict mode (7:2)

This can be fixed by opening node_modules/chi-squared/cdf.js and removing the "with" blocks in functions Gcf and Gser and specifying the "Math" library directly:

function Gcf(X,A) {        // Good for X>A+1
  {
    var A0=0;
    var B0=1;
    var A1=1;
    var B1=X;
    var AOLD=0;
    var N=0;
    while (Math.abs((A1-AOLD)/A1)>.00001) {
      AOLD=A1;
      N=N+1;
      A0=A1+(N-A)*A0;
      B0=B1+(N-A)*B0;
      A1=X*A0+N*A1;
      B1=X*B0+N*B1;
      A0=A0/B1;
      B0=B0/B1;
      A1=A1/B1;
      B1=1;
    }
    var Prob=Math.exp(A*Math.log(X)-X-LogGamma(A))*A1;
  }
  return 1-Prob
}
function Gser(X,A) {        // Good for X<A+1.
    {
    var T9=1/A;
    var G=T9;
    var I=1;
    while (T9>G*.00001) {
      T9=T9*X/(A+I);
      G=G+T9;
      I=I+1;
    }
    G=G*Math.exp(A*Math.log(X)-X-LogGamma(A));
    }
    return G
}

I hope that works for everyone.

[cnotin]

Same issue here on Windows.
Can we maybe add a note on the wiki page explaining that building on Windows is not exactly supported (except through WSL)?
https://github.com/gchq/CyberChef/wiki/Getting-started

[Wingless-Archangel]

same problem here when try to build on 6-AUG-2019, OSX

Please find the error below

$ grunt prod
Running "eslint:configs" (eslint) task

Running "eslint:core" (eslint) task

Running "eslint:web" (eslint) task

Running "eslint:node" (eslint) task

Running "eslint:tests" (eslint) task

Running "clean:prod" (clean) task
>> 1 path cleaned.

Running "clean:config" (clean) task
>> 1 path cleaned.

Running "exec:generateConfig" (exec) task

--- Regenerating config files. ---
Written operation index.
>> internal/modules/esm/default_resolve.js:59
>>   let url = moduleWrapResolve(specifier, parentURL);
>>             ^
>>
>> Error: Cannot find module '/Users/xxx/pentest/CyberChef/node_modules/crypto-api/src/hasher/has160' imported from /Users/xxx/pentest/CyberChef/node_modules/crypto-api/src/crypto-api.mjs
>>     at Loader.resolve [as _resolve] (internal/modules/esm/default_resolve.js:59:13)
>>     at Loader.resolve (internal/modules/esm/loader.js:73:33)
>>     at Loader.getModuleJob (internal/modules/esm/loader.js:149:40)
>>     at ModuleWrap.<anonymous> (internal/modules/esm/module_job.js:43:40)
>>     at link (internal/modules/esm/module_job.js:42:36) {
>>   code: 'ERR_MODULE_NOT_FOUND'
>> }
--- Config scripts finished. ---


Running "webpack:web" (webpack) task
 94% after sealError parsing bundle asset "/Users/xxx/pentest/CyberChef/build/prod/assets/main.js": no such file
Error parsing bundle asset "/Users/xxx/pentest/CyberChef/build/prod/sitemap.js": no such file

No bundles were parsed. Analyzer will show only original module sizes from stats file.

Webpack Bundle Analyzer saved report to /Users/xxx/pentest/CyberChef/build/prod/BundleAnalyzerReport.html
Hash: 8e1491340bdd07f1838a
Version: webpack 4.35.2
Time: 15306ms
Built at: 08/06/2019 5:12:36 PM
 10 assets
Warning:  Use --force to continue.

Aborted due to warnings.

[n1474335]

We currently support Node v10 which is the LTS version (see Node support in the README).

There are some issues with the crypto-api library that cause problems with some other versions of Node. We are working to get these fixed. The issue is being tracked in #596.