Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up require(), phase 1 and 2 #1801

Merged
merged 2 commits into from
May 27, 2015

Conversation

bnoordhuis
Copy link
Member

Replace calls to fs.statSync() with an internal variant that does not
create Error or Stat objects that put strain on the garbage collector.

Replace calls to fs.readFileSync() with an internal variant that does
not create Error objects on failure and is a bit speedier in general.

A secondary benefit is that it improves start-up times in the debugger
because it no longer emits thousands of exception debug events.

On a medium-sized application[0], this commit and its predecessor reduce
start-up times from about 1.5s to 0.5s and reduce the number of start-up
exceptions from ~6100 to 32, half of them internal to the application.

CI: https://jenkins-iojs.nodesource.com/view/iojs/job/iojs+any-pr+multi/712/

R=@trevnorris?

var json = fs.readFileSync(jsonPath, 'utf8');
} catch (e) {
var jsonPath = path.resolve(requestPath, 'package.json');
var json = internalModuleReadFile(jsonPath);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two other places where fs.readFileSync could be replaced but that would alter the stack trace in case of error.

I hope and assume no one writes code that depends on the exact stack trace but I decided to let it be for now, just in case.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that should be a concern, especially not when there's potential perf gains. Nobody should depend on the exact stack trace.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really disagree but the other two call sites don't seem to be bottlenecks like the package.json load step is.

@Fishrock123 Fishrock123 added the module Issues and PRs related to the module subsystem. label May 26, 2015
@ofrobots
Copy link
Contributor

For posterity, can you provide the missing link for "medium-sized application[0]"?

}
}

fclose(stream);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally ridiculous case, but what would happen if the file were deleted while being read? I assume fread() would simply return 1, then fclose() would return EBADF?

Likelihood of this happening is near zero.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On UNIX systems, unlinking the file doesn't delete it until the last file descriptor is closed. If another process truncates it, fwrite() simply returns with a small (possibly zero) read count.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bnoordhuis
And if the fs gets unmounted or the connection the the remote fs gets broken?
Probablilty = 0, but still?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR won't change the observable behavior of require(), if that is what you're asking.

EDIT: Oh wait, I think I get you now. Yes, fclose() can return an error.

@trevnorris
Copy link
Contributor

Just a couple questions, but LGTM.

@bnoordhuis
Copy link
Member Author

@ofrobots Sorry, it's in the commit log but I forgot to paste it into the description. It's https://github.com/strongloop/loopback-sample-app

@bnoordhuis
Copy link
Member Author

Incorporated feedback, PTAL.

@trevnorris
Copy link
Contributor

LGTM

Replace calls to fs.statSync() with an internal variant that does not
create Error or Stat objects that put strain on the garbage collector.

A secondary benefit is that it improves start-up times in the debugger
because it no longer emits thousands of exception debug events.

PR-URL: nodejs#1801
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Replace calls to fs.readFileSync() with an internal variant that does
not create Error objects on failure and is a bit speedier in general.

A secondary benefit is that it improves start-up times in the debugger
because it no longer emits thousands of exception debug events.

On a medium-sized application[0], this commit and its predecessor reduce
start-up times from about 1.5s to 0.5s and reduce the number of start-up
exceptions from ~6100 to 32, half of them internal to the application.

[0] https://github.com/strongloop/loopback-sample-app

PR-URL: nodejs#1801
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
@bnoordhuis bnoordhuis closed this May 27, 2015
@bnoordhuis bnoordhuis deleted the speed-up-require branch May 27, 2015 19:22
@bnoordhuis bnoordhuis merged commit 1bbf8d0 into nodejs:master May 27, 2015
@ghost
Copy link

ghost commented May 29, 2015

Woh ,iojs just got faster?

@rvagg rvagg mentioned this pull request May 30, 2015
rvagg added a commit to rvagg/io.js that referenced this pull request May 31, 2015
Notable Changes:

* node: Speed-up require() by replacing usage of fs.statSync() and
  fs.readFileSync() with internal variants that are faster for this use-case
  and do not create as many objects for the garbage collector to clean up.
  The primary two benefits are: significant increase in application start-up
  time on typical applications and better start-up time for the debugger by
  eliminating almost all of the thousands of exception events.
  (Ben Noordhuis) nodejs#1801.
* node: Resolution of pre-load modules (-r or --require) now follows the
  standard require() rules rather than just resolving paths, so you can now
  pre-load modules in node_modules. (Ali Ijaz Sheikh) nodejs#1812.
* npm: Upgraded npm to v2.11.0. New hooks for preversion, version, and
  postversion lifecycle events, some SPDX-related license changes and license
  file inclusions. See the release notes for full details.
rvagg added a commit to rvagg/io.js that referenced this pull request May 31, 2015
PR-URL: nodejs#1808

Notable Changes:

* node: Speed-up require() by replacing usage of fs.statSync() and
  fs.readFileSync() with internal variants that are faster for this use-case
  and do not create as many objects for the garbage collector to clean up.
  The primary two benefits are: significant increase in application start-up
  time on typical applications and better start-up time for the debugger by
  eliminating almost all of the thousands of exception events.
  (Ben Noordhuis) nodejs#1801.
* node: Resolution of pre-load modules (-r or --require) now follows the
  standard require() rules rather than just resolving paths, so you can now
  pre-load modules in node_modules. (Ali Ijaz Sheikh) nodejs#1812.
* npm: Upgraded npm to v2.11.0. New hooks for preversion, version, and
  postversion lifecycle events, some SPDX-related license changes and license
  file inclusions. See the release notes for full details.
andrewdeandrade pushed a commit to andrewdeandrade/node that referenced this pull request Jun 3, 2015
Replace calls to fs.statSync() with an internal variant that does not
create Error or Stat objects that put strain on the garbage collector.

A secondary benefit is that it improves start-up times in the debugger
because it no longer emits thousands of exception debug events.

PR-URL: nodejs/node#1801
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
andrewdeandrade pushed a commit to andrewdeandrade/node that referenced this pull request Jun 3, 2015
Replace calls to fs.readFileSync() with an internal variant that does
not create Error objects on failure and is a bit speedier in general.

A secondary benefit is that it improves start-up times in the debugger
because it no longer emits thousands of exception debug events.

On a medium-sized application[0], this commit and its predecessor reduce
start-up times from about 1.5s to 0.5s and reduce the number of start-up
exceptions from ~6100 to 32, half of them internal to the application.

[0] https://github.com/strongloop/loopback-sample-app

PR-URL: nodejs/node#1801
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
@ChALkeR
Copy link
Member

ChALkeR commented Jun 16, 2015

It looks stat at least one module out there was monkey-patching fs and relied on require using that: tschaub/mock-fs#43

But I guess monkey-patching was never supported, because supporting it means freezing everything.

targos added a commit to targos/node that referenced this pull request Jun 16, 2015
nodejs#1801 introduced internal fs methods to speed up require.
The methods do not call path._makeLong like their counterpart
from the fs module. This brings back the old behaviour.

Fixes: nodejs#1990
Fixes: nodejs#1980
Fixes: nodejs#1849
piscisaureus pushed a commit that referenced this pull request Jun 16, 2015
#1801 introduced internal fs methods
to speed up require. The methods do not call path._makeLong like their
counterpart from the fs module. This brings back the old behaviour.

Fixes: #1990
Fixes: #1980
Fixes: #1849

PR-URL: https://github.com/nodejs/io.js/pull/1991/files
Reviewed-By: Bert Belder <bertbelder@gmail.com>
rvagg added a commit that referenced this pull request Jun 23, 2015
PR-URL: #1996

Notable changes

* module: The number of syscalls made during a require() have been
  significantly reduced again (see #1801 from v2.2.0 for previous
  work), which should lead to a performance improvement
  (Pierre Inglebert) #1920.
* npm:
  - Upgrade to v2.11.2 (Rebecca Turner) #1956.
  - Upgrade to v2.11.3 (Forrest L Norvell) #2018.
* zlib: A bug was discovered where the process would abort if the
  final part of a zlib decompression results in a buffer that would
  exceed the maximum length of 0x3fffffff bytes (~1GiB). This was
  likely to only occur during buffered decompression (rather than
  streaming). This is now fixed and will instead result in a thrown
  RangeError (Michaël Zasso) #1811.
rvagg added a commit that referenced this pull request Jun 23, 2015
PR-URL: #1996

Notable changes

* module: The number of syscalls made during a require() have been
  significantly reduced again (see #1801 from v2.2.0 for previous
  work), which should lead to a performance improvement
  (Pierre Inglebert) #1920.
* npm:
  - Upgrade to v2.11.2 (Rebecca Turner) #1956.
  - Upgrade to v2.11.3 (Forrest L Norvell) #2018.
* zlib: A bug was discovered where the process would abort if the
  final part of a zlib decompression results in a buffer that would
  exceed the maximum length of 0x3fffffff bytes (~1GiB). This was
  likely to only occur during buffered decompression (rather than
  streaming). This is now fixed and will instead result in a thrown
  RangeError (Michaël Zasso) #1811.
@ChALkeR ChALkeR added the performance Issues and PRs related to the performance of Node.js. label Feb 16, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
module Issues and PRs related to the module subsystem. performance Issues and PRs related to the performance of Node.js.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants