Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: enable loading internal modules from disk #31321

Merged
merged 1 commit into from
Jan 31, 2020

Conversation

devsnek
Copy link
Member

@devsnek devsnek commented Jan 11, 2020

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • commit message follows commit guidelines

@nodejs-github-bot nodejs-github-bot added build Issues and PRs related to build files or the CI. c++ Issues and PRs that require attention from people who are familiar with C++. labels Jan 11, 2020
@devsnek devsnek requested a review from joyeecheung January 11, 2020 22:04
@nodejs-github-bot
Copy link
Collaborator

Copy link
Member

@addaleax addaleax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still sad that this isn’t a runtime option, but thanks for picking this up again.

src/node_native_module.cc Outdated Show resolved Hide resolved
@devsnek
Copy link
Member Author

devsnek commented Jan 12, 2020

@addaleax I'd be happy to make it a runtime option. I'm not sure how I'd get data from the options parser to the internal module loader though.

@tniessen
Copy link
Member

This is great! I am fine with it being a configure option (maybe just for now), given that we probably wouldn't want people to use it outside of development. (For example, the current PR seems to resolve the path relative to the current working directory, which seems dangerous outside of the repo.)

@BridgeAR BridgeAR added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jan 13, 2020
@devsnek devsnek removed the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Jan 13, 2020
src/node_native_module.cc Outdated Show resolved Hide resolved
src/node_native_module.cc Outdated Show resolved Hide resolved
src/node_native_module.cc Outdated Show resolved Hide resolved
configure.py Outdated Show resolved Hide resolved
@Trott
Copy link
Member

Trott commented Jan 13, 2020

Should this print a warning telling end users not to use it?

@jasnell
Copy link
Member

jasnell commented Jan 13, 2020

+1 to emitting a warning on this.

@addaleax
Copy link
Member

I think if we turn this into a runtime option, the warning may be appropriate, but for a compile time option I don’t quite see the point – whoever built the Node.js binary explicitly opted into a developer feature, and adding the warning is going to make running the test suite infeasible, which makes it less useful as a developer feature.

@targos
Copy link
Member

targos commented Jan 13, 2020

Maybe James was thinking about a build time warning?

@jasnell
Copy link
Member

jasnell commented Jan 13, 2020

Maybe James was thinking about a build time warning?

Yes, thank you. That's what I meant.

Regarding the idea of a runtime option: I see the benefit but I also see significant risk. The fact that we have to expose --expose-internals is bad enough. I'd prefer we start with the configuration option and go from there.

Copy link
Member

@joyeecheung joyeecheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of non-blocking nits. Happy to see this happening!

I think it's possible to add a test for it (at least by testing that adding a new module works), to feature-detect I think it makes sense to just use process.features. But that can be added later since this is just a developer feature after all.

src/node_native_module.cc Outdated Show resolved Hide resolved
configure.py Outdated Show resolved Hide resolved
@devsnek
Copy link
Member Author

devsnek commented Jan 14, 2020

I'm not sure how to test this cc @nodejs/build

@richardlau
Copy link
Member

I'm not sure how to test this cc @nodejs/build

If you want to do a one-off build you can run https://ci.nodejs.org/job/node-test-commit-custom-suites-freestyle/ with custom args passed to configure.
image

@devsnek
Copy link
Member Author

devsnek commented Jan 14, 2020

@richardlau i mean continuously. I guess we'd need to add a new box/configuration?

@devsnek devsnek force-pushed the load-internal-modules-from-disk branch from 52a14a6 to c44133d Compare January 14, 2020 21:00
configure.py Outdated Show resolved Hide resolved
@richardlau
Copy link
Member

@richardlau i mean continuously. I guess we'd need to add a new box/configuration?

You probably wouldn't need a new box but probably would need a new job since you'd have to build Node.js with different options passed to configure. The nearest to that we currently have are the various sharedlibs jobs but we do not test every option that can be passed to configure.

The Build WG is currently very stretched though so there may not be much enthusiasm for adding another build configuration for a developer feature.

@richardlau
Copy link
Member

re. new build configurations nodejs/build#1982 / #30057 would be ideal for this sort of thing but unfortunately it's stuck (nodejs/build#1982 (comment)).

@devsnek devsnek force-pushed the load-internal-modules-from-disk branch 2 times, most recently from 6ecfc90 to dcc29cb Compare January 14, 2020 22:17
@devsnek
Copy link
Member Author

devsnek commented Jan 14, 2020

cctest failures when this is enabled:

[----------] 6 tests from BaseObjectPtrTest
[ RUN      ] BaseObjectPtrTest.ScopedDetached
../test/cctest/test_base_object_ptr.cc:50: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
../test/cctest/test_base_object_ptr.cc:53: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:55: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.ScopedDetached (33 ms)
[ RUN      ] BaseObjectPtrTest.ScopedDetachedWithWeak
../test/cctest/test_base_object_ptr.cc:66: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
../test/cctest/test_base_object_ptr.cc:70: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:73: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.ScopedDetachedWithWeak (21 ms)
[ RUN      ] BaseObjectPtrTest.Undetached
../test/cctest/test_base_object_ptr.cc:87: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:83: Failure
Expected equality of these values:
  static_cast<Environment*>(arg)->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.Undetached (20 ms)
[ RUN      ] BaseObjectPtrTest.GCWeak
../test/cctest/test_base_object_ptr.cc:104: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:111: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
[  FAILED  ] BaseObjectPtrTest.GCWeak (20 ms)
[ RUN      ] BaseObjectPtrTest.Moveable
../test/cctest/test_base_object_ptr.cc:129: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:140: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 2
  1
../test/cctest/test_base_object_ptr.cc:145: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.Moveable (30 ms)
[ RUN      ] BaseObjectPtrTest.NestedClasses
../test/cctest/test_base_object_ptr.cc:175: Failure
Expected equality of these values:
  env->base_object_count()
    Which is: 4
  3
../test/cctest/test_base_object_ptr.cc:167: Failure
Expected equality of these values:
  static_cast<Environment*>(arg)->base_object_count()
    Which is: 1
  0
[  FAILED  ] BaseObjectPtrTest.NestedClasses (19 ms)
[----------] 6 tests from BaseObjectPtrTest (143 ms total)

@rvagg
Copy link
Member

rvagg commented Jan 14, 2020

I don't mind putting in a bit of work to extend the sharedlibs set to include new build configurations if the build configuration is above some poorly defined level of significance. Basically, is this anticipated to be a commonly used build option? Do we think this is going to grow into a more significant thing? If it's a key feature then I could do some work with some pointers on how to compile and run the tests with this option unless this is just some obscure thing in the corner that serves the use of a couple of people with special needs.

@devsnek
Copy link
Member Author

devsnek commented Jan 14, 2020

I'd imagine people working on node core would just run with this option enabled (unless they were working specifically on the internal module loader itself)

I don't think it's a huge priority to get testing infra for it though.

@rvagg
Copy link
Member

rvagg commented Jan 14, 2020

(missed the button, sorry)

@mmarchini
Copy link
Contributor

Can this be tested on GH Actions instead? It would give some coverage and should be lower maintenance than doing it on Jenkins right now.

src/node_native_module.cc Outdated Show resolved Hide resolved
src/node_native_module.cc Show resolved Hide resolved
@devsnek devsnek force-pushed the load-internal-modules-from-disk branch from dcc29cb to fb0f519 Compare January 16, 2020 19:16
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@devsnek
Copy link
Member Author

devsnek commented Jan 17, 2020

this is not ready to land, see #31321 (comment)

@devsnek
Copy link
Member Author

devsnek commented Jan 29, 2020

@nodejs/collaborators re #31321 (comment)

@nodejs-github-bot
Copy link
Collaborator

Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

PR-URL: nodejs#31321
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
@devsnek devsnek force-pushed the load-internal-modules-from-disk branch from fb0f519 to 43fb6ff Compare January 31, 2020 09:23
@devsnek devsnek merged commit 43fb6ff into nodejs:master Jan 31, 2020
@devsnek devsnek deleted the load-internal-modules-from-disk branch January 31, 2020 09:24
@devsnek
Copy link
Member Author

devsnek commented Jan 31, 2020

landed in 43fb6ff

codebytere pushed a commit that referenced this pull request Feb 17, 2020
PR-URL: #31321
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
@codebytere codebytere mentioned this pull request Feb 17, 2020
@codebytere
Copy link
Member

@devsnek should this go into v12.x, and could you please manually backport it if so? if not, feel free to update the label.

targos pushed a commit to targos/node that referenced this pull request Apr 25, 2020
PR-URL: nodejs#31321
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
targos pushed a commit that referenced this pull request Apr 28, 2020
PR-URL: #31321
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Issues and PRs related to build files or the CI. c++ Issues and PRs that require attention from people who are familiar with C++.
Projects
None yet
Development

Successfully merging this pull request may close these issues.