Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug report: Cannot build on n1474335 #1162

Closed
remygrandin opened this issue Jan 30, 2021 · 9 comments
Closed

Bug report: Cannot build on n1474335 #1162

remygrandin opened this issue Jan 30, 2021 · 9 comments
Labels
bug

Comments

@remygrandin
Copy link

remygrandin commented Jan 30, 2021 •
edited
Loading

Describe the bug
Cannot build on n1474335

To Reproduce
download (as of now) current repo (commit n1474335 )
Running grunt prod
build fail at webpack:web task

Expected behaviour
Expected build to pass (same docker build worked about 6 month ago, dockerfile at the bottom of the post)

Screenshots

Running "eslint:configs" (eslint) task

Running "eslint:core" (eslint) task

Running "eslint:web" (eslint) task

Running "eslint:node" (eslint) task

Running "eslint:tests" (eslint) task

Running "clean:prod" (clean) task
>> 0 paths cleaned.

Running "clean:config" (clean) task
>> 0 paths cleaned.

Running "exec:generateConfig" (exec) task

--- Regenerating config files. ---
Written operation index.
Written OperationConfig.json
Written Ciphers module
Written Default module
Written Image module
Written Crypto module
Written Serialise module
Written Hashing module
Written Bletchley module
Written Compression module
Written Code module
Written Encodings module
Written Diff module
Written Shellcode module
Written Charts module
Written Regex module
Written PGP module
Written PublicKey module
Written OCR module
Written URL module
Written UserAgent module
Written Yara module
Written OpModules.mjs
--- Config scripts finished. ---


Running "findModules" task
Found 19 modules.

Running "webpack:web" (webpack) task
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/assets/main.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Encodings.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Code.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Yara.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/PGP.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Ciphers.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Crypto.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Image.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Hashing.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Serialise.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Compression.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Charts.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/PublicKey.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Regex.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Shellcode.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Bletchley.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/Diff.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/UserAgent.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/URL.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/modules/OCR.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/ChefWorker.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/DishWorker.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/InputWorker.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/LoaderWorker.js": no such file
Error parsing bundle asset "/tmp/cyberchef/CyberChef/build/prod/ZipWorker.js": no such file

No bundles were parsed. Analyzer will show only original module sizes from stats file.

Webpack Bundle Analyzer saved report to /tmp/cyberchef/CyberChef/build/prod/BundleAnalyzerReport.html
assets by status 33.1 MiB [cached] 42 assets

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 5:0-37
Module not found: Error: Can't resolve './hasher/has160' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'has160.mjs'?
BREAKING CHANGE: The request './hasher/has160' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 6:0-31
Module not found: Error: Can't resolve './hasher/md2' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'md2.mjs'?
BREAKING CHANGE: The request './hasher/md2' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 7:0-31
Module not found: Error: Can't resolve './hasher/md4' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'md4.mjs'?
BREAKING CHANGE: The request './hasher/md4' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 8:0-31
Module not found: Error: Can't resolve './hasher/md5' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'md5.mjs'?
BREAKING CHANGE: The request './hasher/md5' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 9:0-37
Module not found: Error: Can't resolve './hasher/ripemd' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'ripemd.mjs'?
BREAKING CHANGE: The request './hasher/ripemd' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 10:0-33
Module not found: Error: Can't resolve './hasher/sha0' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'sha0.mjs'?
BREAKING CHANGE: The request './hasher/sha0' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 11:0-33
Module not found: Error: Can't resolve './hasher/sha1' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'sha1.mjs'?
BREAKING CHANGE: The request './hasher/sha1' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 12:0-37
Module not found: Error: Can't resolve './hasher/sha256' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'sha256.mjs'?
BREAKING CHANGE: The request './hasher/sha256' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 13:0-37
Module not found: Error: Can't resolve './hasher/sha512' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'sha512.mjs'?
BREAKING CHANGE: The request './hasher/sha512' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 14:0-37
Module not found: Error: Can't resolve './hasher/snefru' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'snefru.mjs'?
BREAKING CHANGE: The request './hasher/snefru' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 15:0-43
Module not found: Error: Can't resolve './hasher/whirlpool' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'whirlpool.mjs'?
BREAKING CHANGE: The request './hasher/whirlpool' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 16:0-40
Module not found: Error: Can't resolve './encoder/utf' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'utf.mjs'?
BREAKING CHANGE: The request './encoder/utf' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 17:0-57
Module not found: Error: Can't resolve './encoder/array-buffer' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'array-buffer.mjs'?
BREAKING CHANGE: The request './encoder/array-buffer' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 18:0-38
Module not found: Error: Can't resolve './encoder/hex' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'hex.mjs'?
BREAKING CHANGE: The request './encoder/hex' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 19:0-44
Module not found: Error: Can't resolve './encoder/base64' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'base64.mjs'?
BREAKING CHANGE: The request './encoder/base64' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

ERROR in ./node_modules/crypto-api/src/crypto-api.mjs 20:0-30
Module not found: Error: Can't resolve './mac/hmac' in '/tmp/cyberchef/CyberChef/node_modules/crypto-api/src'
Did you mean 'hmac.mjs'?
BREAKING CHANGE: The request './mac/hmac' failed to resolve only because it was resolved as fully specified
(probably because the origin is a '*.mjs' file or a '*.js' file where the package.json contains '"type": "module"').
The extension in the request is mandatory for it to be fully specified.
Add the extension to the request.
 @ ./src/core/lib/Hash.mjs 10:0-54 23:15-34 25:9-32
 @ ./src/core/operations/RIPEMD.mjs 17:0-42 60:13-20
 @ ./src/core/config/modules/Crypto.mjs 37:0-49 79:12-18

webpack 5.10.1 compiled with 16 errors in 89295 ms
(node:7) [DEP_WEBPACK_STATS_WARNINGS_FILTER] DeprecationWarning: config.stats.warningsFilter is deprecated in favor of config.ignoreWarnings
Warning:  Use --force to continue.

Aborted due to warnings.
The command '/bin/sh -c grunt prod' returned a non-zero code: 3

Desktop (if relevant, please complete the following information):

  • OS: Ubuntu 20.04 docker image
  • Browser: N/A
  • CyberChef version: commit n1474335

For ref : full docker file

# ==== BUILDER ====
FROM ubuntu:latest AS builder

WORKDIR /tmp/cyberchef

# Installing NodeJS V10
RUN apt update
RUN apt -y install curl dirmngr apt-transport-https lsb-release ca-certificates
RUN curl -sL https://deb.nodesource.com/setup_10.x | bash
RUN apt-get install nodejs -y

# Installing Git
RUN apt -y install git

# Build Cyberchef
RUN npm install -g grunt-cli
RUN git clone https://github.com/gchq/CyberChef.git
WORKDIR /tmp/cyberchef/CyberChef
RUN npm install
RUN grunt prod

# ==== RUNNER ====
FROM ubuntu:latest

## Application Port
EXPOSE 80

# Installing Nginx
RUN apt update
ENV DEBIAN_FRONTEND=noninteractive
RUN apt -y install nginx ca-certificates

COPY --from=builder /tmp/cyberchef/CyberChef/build/prod /opt/cyberchef

RUN rm /etc/nginx/sites-available/default && rm /etc/nginx/sites-enabled/default
COPY cyberchef.conf /etc/nginx/sites-available/cyberchef.conf
RUN ln -s /etc/nginx/sites-available/cyberchef.conf /etc/nginx/sites-enabled/cyberchef.conf

# Run the specified command within the container.
CMD ["nginx", "-g", "daemon off;"]
@sebastianganson
Copy link

sebastianganson commented Jan 30, 2021 via email

@remygrandin
Copy link
Author

The build was happening from scratch, with cache disabled, so technically it run each time as if it was the first.

I have reproduced the issue with the exact same error on another machine.

Here is the full build log :
https://pastebin.pl/view/c41b7c23

@tabic
Copy link

tabic commented Jan 31, 2021

I can reproduce the bug with node:10 and node:11 in docker

@tabic tabic mentioned this issue Jan 31, 2021
Merged
@n1474335
Copy link
Member

n1474335 commented Feb 1, 2021

I have not tried running this in docker before, but I have just set up a bunch of GitHub Actions configs which work. I believe GitHub Actions uses docker behind the scenes, so this seems like a good sign! Perhaps trying modifying the dockerfile to match this build process: https://github.com/gchq/CyberChef/blob/master/.github/workflows/pull_requests.yml

Also, FYI, n1474335 is my username, not a commit hash, although I can see where the confusion comes from!

@tabic
Copy link

tabic commented Feb 1, 2021

@n1474335 thanks for looking into this. GitHub Actions run on a virtual machine. I tried to use npx in the container like you do in the action but it still fails with the same error, the build log is available here: https://github.com/rxcket/docker-cyberchef/runs/1808724385?check_suite_focus=true

@n1474335
Copy link
Member

n1474335 commented Feb 1, 2021

The problem is that the postinstall.sh script is not running. It sorts out the import file extension issues in crypto-api.

The error is here: https://github.com/rxcket/docker-cyberchef/runs/1808724385?check_suite_focus=true#step:4:247

Can you not run bash in docker? I can modify the postinstall npm script if there is a better solution.

@tabic
Copy link

tabic commented Feb 1, 2021

Thanks. I actually figured it out a few minutes ago too and tested both workarounds. I could reproduce the bug on a fresh ubuntu 18.04 / 20.04 it failed with the same error. I noticed GitHub actions are executed as runner user: https://github.com/gchq/CyberChef/runs/1808392346?check_suite_focus=true#step:7:57
The actual issue is that npm install does not run any scripts if executed as root user.

Following two workarounds are available:

  1. Run npm install with --unsafe-perm
  2. Run npm install with a non-root in Docker

Example with node base image:

FROM node:10
RUN chown -R node:node /srv
USER node
WORKDIR /srv
RUN git clone -b master --depth=1 https://github.com/gchq/CyberChef.git .
RUN npm install # postinstall is now executed

@remygrandin
Copy link
Author

@n1474335 : my appologies about the mixup with your display name, I shall read better next time :-)
@rxcket : I used the --unsafe-perm as a quick patch, I confirm it work too on my dockerfile, thanks for the finding !

@n1474335
Copy link
Member

n1474335 commented Feb 2, 2021

I would very much like to get rid of the postinstall script, but unfortunately it is required until crypto-api sort out their imports. I have raised an issue on their repo and someone has submitted a PR. Once it is merged, this will be sorted out. In the mean time, these workarounds will be required. Given that, I'll close this.

@n1474335 n1474335 closed this as completed Feb 2, 2021
@sergiogval sergiogval mentioned this issue Apr 14, 2024
Open
Open
@Croat79 Croat79 mentioned this issue Apr 14, 2024
Open
@Shadow0ps Shadow0ps mentioned this issue Apr 14, 2024
Open
@deepcodeg deepcodeg mentioned this issue Apr 14, 2024
Open
@ssmbct-netops ssmbct-netops mentioned this issue Apr 14, 2024
Open
@masifpak masifpak mentioned this issue Apr 14, 2024
Open
@ekmixon ekmixon mentioned this issue Apr 15, 2024
Open
@rx007 rx007 mentioned this issue Apr 15, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@remygrandin @sebastianganson @n1474335 @tabic