Skip to content
This repository has been archived by the owner on Nov 28, 2020. It is now read-only.

Regressions in buffer operations - 4.4.0 #38

Closed
gareth-ellis opened this issue Mar 18, 2016 · 10 comments
Closed

Regressions in buffer operations - 4.4.0 #38

gareth-ellis opened this issue Mar 18, 2016 · 10 comments

Comments

@gareth-ellis
Copy link
Member

Hello,

One of the benchmarks we run contains what is in effect a number of microbenchmarks. We do a number of different actions with buffers, and I noticed in 4.4.0 we have seen a drop in some of these buffer tests.
One of these tests is looking at how long it takes to create a new buffer from an array of data.

var harness = require('./harness.js');

var ARRAY = [1, 2, 23829, 4, 5, 7, 12312321, 2131, 434832, 43792, 23421, 65345, 132210, 77777, 322131, 1, 2, 23829, 4, 5, 7, 12312321, 2131, 434832, 43792, 23421, 65345, 132210, 77777, 322131, 1, 2, 23829, 4, 5, 7, 12312321, 2131, 434832, 43792, 23421, 65345, 132210, 77777, 322131, 1, 2, 23829, 4, 5, 7, 12312321, 2131, 434832, 43792, 23421, 65345, 132210, 77777, 322131];
var ITERATIONS = 300000;
var result;

function test() {
    for(var i=0;i<ITERATIONS;i++) {
        result = new Buffer(ARRAY);
    }
}
harness.run_test(test);

Our harness repeats the test function a number of times, either until we get good consistency in scores, or until a max number of attempts. We then report a number of operations per second.

Comparing 4.3.2 and 4.4.0 we saw quite a drop in the ops/sec for this test. Looking at the changes between the builds, I saw that 4.4.0 had some changes to lib/buffer.js as per nodejs/node#4886 .

I rebuilt node 4.4.0 with the buffer.js from 4.3.2 and saw the performance recover:

4.4.0 with the buffer.js from 4.3.2:
sdk4.4.0_oldbuffer/bin/node new-buffer-from-array-test.js
total time: 5.093s -- iterations: 43 -- ops/sec: 8.44 -- average time: 0.12s -- variance: 2.19%
total time: 5.077s -- iterations: 43 -- ops/sec: 8.47 -- average time: 0.12s -- variance: 0.22%
total time: 5.076s -- iterations: 43 -- ops/sec: 8.47 -- average time: 0.12s -- variance: 0.32%
total time: 5.076s -- iterations: 43 -- ops/sec: 8.47 -- average time: 0.12s -- variance: 0.18%

4.4.0 unmodified
sdk4.40/bin/node new-buffer-from-array-test.js
total time: 5.003s -- iterations: 26 -- ops/sec: 5.2 -- average time: 0.19s -- variance: 4.42%
total time: 5.059s -- iterations: 26 -- ops/sec: 5.14 -- average time: 0.19s -- variance: 7.04%
total time: 5.149s -- iterations: 27 -- ops/sec: 5.24 -- average time: 0.19s -- variance: 0.24%
total time: 5.149s -- iterations: 27 -- ops/sec: 5.24 -- average time: 0.19s -- variance: 0.24%

Looking at the changes in buffer.js, they mainly seem to be changing var to either const or let.

I reverted these ones, and saw the performance improve:

4.4.0 with the const & let changes switched back to var
sdk4.4.0-testfix/bin/node new-buffer-from-array-test.js
total time: 5.062s -- iterations: 42 -- ops/sec: 8.3 -- average time: 0.12s -- variance: 2.11%
total time: 5.039s -- iterations: 42 -- ops/sec: 8.33 -- average time: 0.12s -- variance: 0.13%
total time: 5.045s -- iterations: 42 -- ops/sec: 8.33 -- average time: 0.12s -- variance: 1.09%
total time: 5.039s -- iterations: 42 -- ops/sec: 8.33 -- average time: 0.12s -- variance: 0.29%

Granted not quite to 100%, but much closer. Any ideas why this would cause a regression?

@targos
Copy link
Member

targos commented Mar 18, 2016

cc @nodejs/lts @Trott @trevnorris

@gareth-ellis
Could you make another try by only changing let to var here ?

@gareth-ellis
Copy link
Member Author

Looks good -

4.4.0
sdk4.40/bin/node new-buffer-from-array-test.js
total time: 5.032s -- iterations: 27 -- ops/sec: 5.37 -- average time: 0.19s -- variance: 4.58%
total time: 5.175s -- iterations: 28 -- ops/sec: 5.41 -- average time: 0.18s -- variance: 0.26%
total time: 5.172s -- iterations: 28 -- ops/sec: 5.41 -- average time: 0.18s -- variance: 0.25%
total time: 5.174s -- iterations: 28 -- ops/sec: 5.41 -- average time: 0.18s -- variance: 0.23%

4.4.0 with old buffer
sdk4.4.0_oldbuffer/bin/node new-buffer-from-array-test.js
total time: 5.058s -- iterations: 42 -- ops/sec: 8.3 -- average time: 0.12s -- variance: 2.05%
total time: 5.047s -- iterations: 42 -- ops/sec: 8.32 -- average time: 0.12s -- variance: 0.31%
total time: 5.044s -- iterations: 42 -- ops/sec: 8.33 -- average time: 0.12s -- variance: 0.25%
total time: 5.045s -- iterations: 42 -- ops/sec: 8.33 -- average time: 0.12s -- variance: 0.27%

4.4.0 with let changed to var as per request from @targos
sdk4.4.0_justlet/bin/node new-buffer-from-array-test.js
total time: 5.105s -- iterations: 43 -- ops/sec: 8.42 -- average time: 0.12s -- variance: 2.05%
total time: 5.09s -- iterations: 43 -- ops/sec: 8.45 -- average time: 0.12s -- variance: 0.41%
total time: 5.089s -- iterations: 43 -- ops/sec: 8.45 -- average time: 0.12s -- variance: 0.41%
total time: 5.089s -- iterations: 43 -- ops/sec: 8.45 -- average time: 0.12s -- variance: 0.41%

@MylesBorins
Copy link

Is this due to the version of v8 that we are currently shipping with v4? would v5 be suffering a similar fate? /cc @ofrobots

Based on these bench marks I'd be open to doing a project wide s/for(let/for(var/, at least for lts

thoughts?

/cc @jasnell

@jasnell
Copy link
Member

jasnell commented Mar 18, 2016

It's likely due to some point in time optimization quirk with the version of v8. We can likely get those let/const uses reverted in v4, just need a PR opened against v4.x-staging to do so!

@ofrobots
Copy link
Collaborator

I think it would be best to avoid let in for loops until TurboFan becomes the default optimizer. See also: https://groups.google.com/d/msg/v8-users/hsUrt4I2D98/QpvLs6aFAQAJ.

@MylesBorins
Copy link

@ofrobots when is the timeline for TurboFan becoming the default optimizer? Should we be updating master / v5 as well?

@gareth-ellis
Copy link
Member Author

Once we know where the PR should just be targeted, I can raise it

@MylesBorins
Copy link

@gareth-ellis I think that this should likely be targeted at master if TurboFan is not coming with v8 v5.x

If we do target master we will likely want to include a rule for eslint to enforce that we don't regress here

@trevnorris
Copy link
Collaborator

v4 is when we switched to using the typed array API. I believe the mechanics are getting faster since then, but there was a known performance drop when moving over. Nothing we can do since the API previously relied on was removed.

@gareth-ellis
Copy link
Member Author

I think can close this too, the PR has landed

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants