Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mutate labels to avoid excessive object cloning. #220

Merged
merged 1 commit into from
Sep 21, 2018

Conversation

nowells
Copy link
Contributor

@nowells nowells commented Sep 20, 2018

I discovered another performance optimization we can perform, which is to mutate the labels and avoid Object.assign where we can in hot paths.

❯ node benchmarks/index.js
Progress:
published#registry#getMetricsAsJSON x 1,819 ops/sec ±4.51% (80 runs sampled)
published#registry#metrics x 1,301 ops/sec ±5.13% (76 runs sampled)
local#registry#getMetricsAsJSON x 9,025 ops/sec ±4.71% (82 runs sampled)
local#registry#metrics x 2,968 ops/sec ±3.59% (85 runs sampled)

Results:
╔══════════════════╤════════════════════════════╤═══════════════════════════╗
║ registry         │ published                  │ local                     ║
╟──────────────────┼────────────────────────────┼───────────────────────────╢
║ getMetricsAsJSON │ 1819.383794048214 ops/sec  │ 9024.761535386682 ops/sec ║
╟──────────────────┼────────────────────────────┼───────────────────────────╢
║ metrics          │ 1300.6959959677145 ops/sec │ 2968.460595395247 ops/sec ║
╚══════════════════╧════════════════════════════╧═══════════════════════════╝

Fastest is local#registry#getMetricsAsJSON
Fastest is local#registry#metrics

@SimenB
Copy link
Collaborator

SimenB commented Sep 20, 2018

Thanks!

I also decided to include a basic benchmarking suite. If you want me to improve the suite, change it, or remove it entirely let me know, but seems like it might be useful to compare/see performance over time.

That's awesome, we've long wanted a benchmarking suite! But if we add it, I think it should be written in JS using e.g. https://github.com/bestiejs/benchmark.js/.

@nowells
Copy link
Contributor Author

nowells commented Sep 20, 2018

@SimenB happy to do it in JS, I only used the one I added here because I already had it sitting around in another project, so just copy and pasted. I'll take on creating the benchmark suite using that library (or something similar). I will rip out the current suite and leave just the performance changes (you can feel free to test with the test script locally to confirm)

@SimenB
Copy link
Collaborator

SimenB commented Sep 20, 2018

@KevinAMurray linked to a suite written in the framework I suggested in https://github.com/KevinAMurray/prom-client/tree/master/benchmarks. Might be a good starting point 🙂

lib/registry.js Outdated
})
);
for (const val of item.values) {
for (const label of Object.keys(this._defaultLabels)) {
Copy link
Collaborator

@SimenB SimenB Sep 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is a perf PR, we can lift Object.keys(this._defaultLabels) out of the inner loop to avoid iterating through that object every time

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely

@nowells
Copy link
Contributor Author

nowells commented Sep 20, 2018

With the latest benchmark suite and changes:

❯ node benchmarks/index.js
published#getMetricsAsJSON x 20.24 ops/sec ±7.67% (38 runs sampled)
local#getMetricsAsJSON x 72.32 ops/sec ±5.77% (61 runs sampled)
Fastest is local#getMetricsAsJSON

@SimenB
Copy link
Collaborator

SimenB commented Sep 20, 2018

Oooh, I love the idea of requiring our own latest publish for comparison

lib/histogram.js Outdated
createBucketValues(bucketData, histogram)
);
const buckets = [];
let acc = 0;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I inlined this to avoid creating a function on a big loop, also made performance profiling cleaner. If you don't like it I can revert, but I did get a performance gain by inlining the method.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with this, especially if we now have benchmarks where it shows improvement

@SimenB SimenB requested review from siimon and zbjornson September 20, 2018 13:30
lib/histogram.js Outdated
@@ -280,12 +280,13 @@ function convertLabelsAndValues(labels, value) {
function extractBucketValuesForExport(histogram) {
return bucketData => {
const buckets = [];
const bucketLabels = Object.entries(bucketData.labels);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Object.entries is not available in Node 6 :(

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops! I'll switch back to just shifting Object.keys up. Sorry!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no worries! I thought we had the linter set up to yell, apparently not...

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#221 fwiw 🙂

@nowells
Copy link
Contributor Author

nowells commented Sep 20, 2018

I switched to benchtable (which just extends benchmark internally) for better test setup as well as reporting. If you would rather it just be vanilla benchmark it should be easy to switch back.

❯ node benchmarks/index.js
getMetricsAsJSON for inputs published x 17.88 ops/sec ±7.60% (35 runs sampled)
getMetricsAsJSON for inputs local x 68.51 ops/sec ±8.18% (58 runs sampled)
Fastest is getMetricsAsJSON for inputs local
+-----------+------------------+
|           │ getMetricsAsJSON |
+-----------+------------------+
| published │ 17.88 ops/sec    |
+-----------+------------------+
| local     │ 68.51 ops/sec    |
+-----------+------------------+

@SimenB
Copy link
Collaborator

SimenB commented Sep 20, 2018

benchtable looks nice!

@nowells
Copy link
Contributor Author

nowells commented Sep 20, 2018

benchtable looks nice!

It is not maintained (2016 was last commit) but it just extends benchtable@^2 so it seemed reasonable. But I would understand if you would rather not pull in a seemingly unmaintained library, but I figure that it probably "just works" and it is just an extension of a library that is actively maintained.

@SimenB
Copy link
Collaborator

SimenB commented Sep 20, 2018

Yeah, no worries. I'd rather think of it as "complete" and not "unmaintained" 😀

@SimenB
Copy link
Collaborator

SimenB commented Sep 20, 2018

Mind updating the changelog?

@nowells
Copy link
Contributor Author

nowells commented Sep 20, 2018

Mind updating the changelog?

Sure thing!

@nowells
Copy link
Contributor Author

nowells commented Sep 20, 2018

> node ./benchmarks/index.js

getMetricsAsJSON for inputs published x 1,627 ops/sec ±5.03% (78 runs sampled)
getMetricsAsJSON for inputs local x 8,694 ops/sec ±4.96% (82 runs sampled)
metrics for inputs published x 1,252 ops/sec ±6.36% (76 runs sampled)
metrics for inputs local x 2,250 ops/sec ±5.84% (81 runs sampled)
Fastest is getMetricsAsJSON for inputs local
+------------------+---------------+---------------+
|                  │ published     │ local         |
+------------------+---------------+---------------+
| getMetricsAsJSON │ 1,627 ops/sec │ 8,694 ops/sec |
+------------------+---------------+---------------+
| metrics          │ 1,252 ops/sec │ 2,250 ops/sec |
+------------------+---------------+---------------+

@siimon
Copy link
Owner

siimon commented Sep 20, 2018

Great work! I'll try and have a closer look tonight

lib/registry.js Outdated
valAcc += '\n';
return valAcc;
}, '');
values += line.join(' ').trim();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

values += line.join(' ').trim() + '\n'; to save an assignment

}
})
.on('complete', () => {
// eslint-disable-next-line no-console
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of this, just add it as exception here:

prom-client/.eslintrc

Lines 64 to 69 in 25255c3

{
"files": ["example/**/*.js"],
"rules": {
"no-console": "off"
}
}

@nowells
Copy link
Contributor Author

nowells commented Sep 20, 2018

After Array.join removal.

❯ node benchmarks/index.js
getMetricsAsJSON for inputs published x 1,764 ops/sec ±4.77% (82 runs sampled)
getMetricsAsJSON for inputs local x 9,399 ops/sec ±3.89% (84 runs sampled)
metrics for inputs published x 1,386 ops/sec ±3.71% (83 runs sampled)
metrics for inputs local x 3,016 ops/sec ±4.65% (85 runs sampled)
Fastest is getMetricsAsJSON for inputs local
+------------------+---------------+---------------+
|                  │ published     │ local         |
+------------------+---------------+---------------+
| getMetricsAsJSON │ 1,764 ops/sec │ 9,399 ops/sec |
+------------------+---------------+---------------+
| metrics          │ 1,386 ops/sec │ 3,016 ops/sec |
+------------------+---------------+---------------+

@KevinAMurray
Copy link

(not come across BenchTable -- very nice!)

Benchmark is a great addition. And loving the performance increases.

package.json Outdated
@@ -39,6 +41,7 @@
"lint-staged": "^7.0.0",
"lolex": "^2.1.3",
"prettier": "1.14.2",
"prom-client": "^11.1.2",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love testing against a published version. 👍

This might be a non-issue, but we'll have to update this (and the lock files) after every release (or at least when we care about benchmarks). I don't see a way to exclude a module from the lock files, which would give some options for improving that situation. Not listing it as a dependency at all could be another option, and then we could put the require for it in a try/catch?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could add a CI check or something? npm show prom-client version returns latest published, and if the local version does not match, throw?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that is interesting. We could have a postinstall step that only runs locally that installs the latest version? But yeah, I hadn't thought of the lockfile issue. Let me know what you prefer.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oo, using a package.json script is another good idea. Wouldn't want it to run all the time though, so if it's in postinstall, would have to check somehow if it's being installed in a git checkout or as a dependency, I think?

(Might not be worth the trouble to make this "nice" -- could just manually bump it as necessary.)

@SimenB
Copy link
Collaborator

SimenB commented Sep 20, 2018

Could you add some prose about the benchmarks to a CONTRIBUTING.md or something? Not much, just the fact that they exist and how to run them 🙂

lib/registry.js Outdated
const values = (item.values || []).reduce((valAcc, val) => {
const merged = Object.assign({}, this._defaultLabels, val.labels);
help = `# HELP ${name} ${help}`;
const type = `# TYPE ${name} ${item.type}`;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏 that was long overdue!

lib/registry.js Outdated
@@ -26,42 +26,51 @@ class Registry {
}

getMetricAsPrometheusString(metric, conf) {
const opts = Object.assign({}, defaultMetricsOpts, conf);
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point with using Object.assign like this was both a good way to extend and overwrite the default settings but also a way to make sure a specific configuration value always was there. I know that you changed on how you call this method from the metrics function but this function is public available so it is a breaking change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can revert this change, not the biggest perf win. Good call on the public api change potential.

package.json Outdated
"lolex": "^2.1.3",
"prettier": "1.14.3",
"prom-client": "^11.1.2",
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you could wildcard it? Doesn't solve the lock problem though.

@nowells
Copy link
Contributor Author

nowells commented Sep 20, 2018

Could you add some prose about the benchmarks to a CONTRIBUTING.md or something? Not much, just the fact that they exist and how to run them 🙂

I was planning on just making the benchmarking part of the test phase, and failing if the performance was not within a standard deviation. Would you rather have just the CONTRIBUTING, just the automatic failures, or both?

@SimenB
Copy link
Collaborator

SimenB commented Sep 20, 2018

I think HW varies too much for it to make sense to run as part of the test build. But I think we can paste the results of whatever we end up merging into a doc as a baseline

@nowells
Copy link
Contributor Author

nowells commented Sep 20, 2018

I think HW varies too much for it to make sense to run as part of the test build.

The build will run not with the results of a previous run, but will run the two jobs (with published and local) so I don't think the HW matters, it will all be relative to the current HW compute power. If we were going based on previous results where it was on different hardware I would agree. Thoughts?

@SimenB
Copy link
Collaborator

SimenB commented Sep 20, 2018

Oh, like run against latest published, then current? That makes sense 🙂

@nowells
Copy link
Contributor Author

nowells commented Sep 21, 2018

  1. I added the automatic check to fail if the installed version of prom-client is not equal to the latest on npm.
  2. I setup the benchmark suite to fail the build if the performance of the local changes are worse than the published (we might want to tweak in the future to give it some wiggle room)
  3. I may have gone a little overboard on fleshing out the benchmarks suite.

Success

prom-client benchmark results success

Failure

prom-client benchmark results failure

@SimenB
Copy link
Collaborator

SimenB commented Sep 21, 2018

I may have gone a little overboard on fleshing out the benchmarks suite.

Haha, this is awesome! 😀 You might want to consider creating a module we can install for the setup part, but this looks really good. Thank you so much for working on it!

@SimenB
Copy link
Collaborator

SimenB commented Sep 21, 2018

Also, would you mind creating a separate PR for the benchmark (once fully iterated)? That will keep this PR more focused on perf improvements


set -e

INSTALLED_VERSION=`node -p -e "require('prom-client/package.json').version"`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need the -e, -p is enough 🙂

@nowells
Copy link
Contributor Author

nowells commented Sep 21, 2018

would you mind creating a separate PR for the benchmark

😆I had a feeling I was approaching that boundary. Makes total sense, I just couldn't help myself. I will update this PR to just be the perf changes, and move the benchmark suite to a new PR.

You might want to consider creating a module we can install for the setup part

Great idea! Once I open a PR with it separately, and land it, I will take on extracting it out into it's own package.

Thank you so much for working on it!

This is an awesome package. Thanks for helping to make it.

@nowells
Copy link
Contributor Author

nowells commented Sep 21, 2018

I extracted the benchmark suite to #222

@siimon siimon merged commit 6674ada into siimon:master Sep 21, 2018
@nowells
Copy link
Contributor Author

nowells commented Sep 21, 2018

Thanks for the great review everyone. And thanks for the merge @siimon! Will you be releasing this as a new version, or waiting to batch it up with other changes? Thanks!!!

@SimenB
Copy link
Collaborator

SimenB commented Sep 22, 2018

prom-client 11.1.3 published 🎉

doochik added a commit to doochik/prom-client that referenced this pull request Sep 20, 2019
Bug introduced at siimon#220 and fixed for .getMetricAsPrometheusString() at siimon#273
doochik added a commit to doochik/prom-client that referenced this pull request Nov 13, 2019
Bug introduced at siimon#220 and fixed for .getMetricAsPrometheusString() at siimon#273
zbjornson pushed a commit that referenced this pull request Nov 13, 2019
Bug introduced at #220 and fixed for .getMetricAsPrometheusString() at #273
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants