fs.promises.readFile is 40% slower than fs.readFile

[Jarred-Sumner]

• Version: 15.10.0
• Platform: macOS
• Subsystem: 11.2.1

What steps will reproduce the bug?

Run this benchmark on a 1 MB file (big.file):

const Benchmark = require("benchmark");
const fs = require("fs");
const path = require("path");
const chalk = require("chalk");
const util = require("util");
const promisifed = util.promisify(fs.readFile);

const bigFilepath = path.resolve(__dirname, "./big.file");

const suite = new Benchmark.Suite("fs");

suite
  .add(
    "fs.readFileSync",
    (defer) => {
      fs.readFileSync(bigFilepath, "utf8");
      defer.resolve();
    },
    { defer: true }
  )
  .add(
    "fs.readFile",
    (defer) => {
      fs.readFile(bigFilepath, (err, data) => {
        defer.resolve();
      });
    },
    { defer: true }
  )
  .add(
    "fs.promises.readFile",
    (defer) => {
      fs.promises.readFile(bigFilepath).then(() => {
        defer.resolve();
      });
    },
    { defer: true }
  )
  .add(
    "util.promisify(fs.readFile)",
    (defer) => {
      promisifed(bigFilepath).then(() => {
        defer.resolve();
      });
    },
    { defer: true }
  )
  .on("cycle", function (event) {
    console.log(String(event.target));
  })
  .on("complete", function () {
    console.log(
      "Fastest is " + chalk.green(this.filter("fastest").map("name"))
    );
    console.log("Slowest is " + chalk.red(this.filter("slowest").map("name")));
  })
  .run({ defer: true });

To create a 1 MB file (~40% slower):

dd if=/dev/zero of=big.file count=1024 bs=1024

To create a 20 KB file (~55% slower):

dd if=/dev/zero of=big.file count=20 bs=1024

How often does it reproduce? Is there a required condition?

Always.

What is the expected behavior?

fs.promises.readFile should perform similarly to fs.readFile

What do you see instead?

Additional information

I suspect the cause is right here: https://github.com/nodejs/node/blob/master/lib/internal/fs/promises.js#L319-L339

Instead of creating a new Buffer for each chunk, it could allocate a single Buffer and write to that buffer. I don't think Buffer.concat or temporary arrays are necessary.

[aduh95]

Would you be interested in sending a PR to improve the implementation?

[TheRamann]

But no one said fs.promises.readFile should perform similarly to fs.readFile 🤔

[Shigetorum635]

Well that's right, but no one said it had to be slower either. I think all code should always be as fast as possible.

[justinvanwinkle]

My guess is that the promise version is idling when IO capacity is available due to scheduling? Is it implemented in an event driven way internally, and if it is (for example using epoll) is the epoll event able to interrupt and cause the promise to continue to be evaluated or does it have to wait for the vm to get around to scheduling it again?

[vigneshtdev]

What about other resources(CPU/memory) consumption?

[Shigetorum635]

What about other resources(CPU/memory) consumption?

I dont think it's that, but cant be discarded.

The best option here it's to reproduce.

[Linkgoron]

Note that it looks like there are some other differences as well, although I'm not sure how much they affect performance:

• It looks like the fs/promises version does an extra read even if it has already read size amount of bytes (it doesn't keep track of how much it has read in total), and only finishes reading when it reads 0 bytes. The fs version checks if it has read size amount and ends the read (readFileAfterRead in read_file_context.js)
• chunk sizes are a bit different (2**14 vs 512*1024)
• Buffer.alloc vs Buffer.allocUnsafeSlow

Also, the 'utf8' in your readFileSync is really slowing it down, on my machine if you remove it the sync version is the fastest by far (and the utils.promisify(fs.readFile) is usually faster than fs.readFile).

[medikoo]

(and the utils.promisify(fs.readFile) is usually faster than fs.readFile).

This signals the benchmark flaw. as it's same to stating that decorate(fn)() is faster than fn() (ofc assuming that in both cases fn() does exactly same job).

Promises naturally will always be slower than callbacks (extra objects to garbage collect and artificial ticks are not free)

on my machine if you remove it the sync version is the fastest

I can expect it when running 1 to 1, but I'd expect callback version to be more efficient (faster) if we start to process multiple files and take advantage of parallel execution. If that's not the case, I'd see it as other flaw on Node.js side

[thelebdev]

This is an amazing finding, @Jarred-Sumner
I have a quick question, what did you use to benchmark it and come up with the tests?

[Shigetorum635]

This is an amazing finding, @Jarred-Sumner
I have a quick question, what did you use to benchmark it and come up with the tests?

Quite sure it's npm package "benchmark", sorry if I did not understand your question and you meant something else

[MoritzLoewenstein]

Hello, is there something I can do to get the changes for this into node v14, or is this blocked by something else?

[Linkgoron]

@targos I thought that this was already backported to Node 14? If not, I'll reopen my backport PR that I closed

[targos]

@Linkgoron the original PR is marked backport-requested. In that case, there are two options:

• remove the label if the commit lands cleanly. it should then be picked up by the releaser
• open a backport PR if it doesn't land cleanly or if you want to be 100% sure it is not forgotten

[Linkgoron]

@targos Thanks, originally it didn't land cleanly because of the AbortSignal stuff, but I think that now it should as you backported the AbortSignal stuff (that's why I thought that it was automatically merged). I'll check that it works out, and if it doesn't I'll reopen my backport PR.

[Mesteery]

I guess it's fixed? Feel free to (or ask to) reopen if it isn't.

[sanjarcode]

@Mesteery can you get the docs updated. #39847

[SyedMSawaid]

bro fixed it in bun

[benjamingr]

Hey I'm locking this thread for non contributors leaving off topic comments.

I also blocked the last person who left an off topic comment.