feat: support aggregating metrics across workers in a Node.js worker_… #403

rafixer · 2020-10-15T13:26:12Z

…threads

this PR is connected with this issue: #401 (comment)

zbjornson

Thanks for opening this and sorry it took forever to get reviewed. A few comments below, plus:

Looks like Node.js 10 breaks - drop support
Needs a changelog update
Please add tests

lib/cluster.js

bgalek · 2021-05-05T13:54:23Z

@zbjornson node 10 met its EOL https://nodejs.org/en/about/releases/ maybe we should not test against it anymore? worker features are available from node 11.0.0

SimenB · 2021-05-12T06:24:27Z

Dropping node 10 must be done in a major release, but we can probably do one of those?

alec-benson · 2021-07-23T13:42:01Z

any chance we could push this along? I am using worker threads on a project and this would help a lot.

bgalek · 2021-07-23T13:43:52Z

@rafixer @zbjornson? :)

zbjornson · 2022-01-01T20:19:44Z

This looks good.

We have two other PRs staged for a semver-major release. Let's drop v10 support in that release and land this PR with it.

(I can add the changelog and fix the merge conflict if @rafixer doesn't get to it.)

bgalek · 2022-02-18T17:24:32Z

@zbjornson any chances for new major? :) deprecation fix for node 16+ is also "ready to release" - it would be great if new release comes some time soon ;)

bgalek · 2022-02-28T08:05:37Z

@zbjornson? :)

Zagorodnyi · 2022-06-02T16:05:19Z

@zbjornson Any news about this feature?

rafixer · 2022-06-02T17:19:13Z

Sorry for no action about this PR. I will back to this on next week!

SimenB

sorry about the slow response! can you update the documentation?

lib/workerThreads.js

rafixer · 2023-01-17T14:02:16Z

sorry about the slow response! can you update the documentation?

also, I will check why currently tests cases are failing

SimenB · 2023-01-17T14:04:43Z

there's a syntax error. Miiight be the merge? ee05476 (#403). I didn't get any merge conflicts, tho

SimenB · 2023-01-17T14:12:34Z

Fixing the syntax error gives a bunch of lint errors, so I dunno 😅

rafixer · 2023-01-17T14:17:03Z

as I remember, I tried to refactor the implementation - I will bring back the previous one

rafixer · 2023-01-17T14:39:47Z

but still, I'm worried that implementation might not be efficient - because it depends on communication between parent < - > child, especially when using a nested tree of workers/processes. Maybe the better idea will be to create a separate process and try to communicate with him via IPC

SimenB · 2023-01-17T14:59:41Z

thread communication shouldn't be any slower than separate process, should it?

SimenB · 2023-03-13T08:09:32Z

@rafixer could you target master with this (and resolve conflicts)?

rafixer · 2023-03-13T14:46:45Z

@rafixer could you target master with this (and resolve conflicts)?

Target changed , resolving conflicts in progress!

…threads

infloop · 2023-06-26T11:52:51Z

@SimenB @zbjornson Are there any updates yet? Does this need any help?

iamyamakin · 2024-02-05T22:02:47Z

@SimenB @zbjornson are you waiting for tests and changes for readme.md? How can we help you?

matthieu-beteille · 2024-07-08T19:47:04Z

@SimenB @zbjornson Any update on this? Anything we can do to push this through?

nrathi · 2024-07-25T16:58:38Z

+1, this would really help me as well!

SimenB

is it possible to write tests for this? Probably by spinning up some process which emulates this and making requests against it?

I can't speak to the feature itself as I've never used the cluster mode

SimenB · 2024-07-25T17:05:01Z

lib/cluster.js

@@ -170,6 +182,7 @@ function addListeners() {
 		cluster().on('message', (worker, message) => {
 			if (message.type === GET_METRICS_RES) {
 				const request = requests.get(message.requestId);
+				request.pending += message.workerRequests || 0;


Suggested change

request.pending += message.workerRequests || 0;

if (message.workerRequests) {

request.pending += message.workerRequests;

}

SimenB · 2024-07-25T17:05:32Z

lib/cluster.js

@@ -196,12 +209,38 @@ function addListeners() {
 		// Respond to master's requests for worker's local metrics.
 		process.on('message', message => {
 			if (message.type === GET_METRICS_REQ) {
+				let workerRequests = 0;
+				workersQueue.forEach(worker => {
+					if (worker && worker instanceof Worker) {


Suggested change

if (worker && worker instanceof Worker) {

if (worker instanceof Worker) {

should we check this when instantiating/pushing into workersQueue rather than when receiving messages? We should probably throw

SimenB · 2024-07-25T17:06:28Z

lib/cluster.js

@@ -196,12 +209,38 @@ function addListeners() {
 		// Respond to master's requests for worker's local metrics.
 		process.on('message', message => {
 			if (message.type === GET_METRICS_REQ) {
+				let workerRequests = 0;
+				workersQueue.forEach(worker => {


use for-of instead of forEach

SimenB · 2024-07-25T17:07:14Z

lib/cluster.js

+		parentPort.on('message', ({ type, requestId, port } = {}) => {
+			if (type === GET_METRICS_REQ) {
+				Promise.all(registries.map(r => r.getMetricsAsJSON()))
+					.then(metrics => {


can we use async-await rather than .then and .catch?

zbjornson linked an issue Jan 23, 2021 that may be closed by this pull request

Support for worker_threads module #401

Open

zbjornson requested changes Jan 23, 2021

View reviewed changes

lib/cluster.js Outdated Show resolved Hide resolved

lib/cluster.js Outdated Show resolved Hide resolved

zbjornson added the waiting on author label Jan 23, 2021

rafixer force-pushed the worker_threads_support branch from f9dd057 to ef9e9f7 Compare March 18, 2021 02:13

zbjornson added semver-major and removed waiting on author labels Jan 1, 2022

rafixer force-pushed the worker_threads_support branch from c3a1be5 to 2bb230a Compare October 14, 2022 10:17

SimenB requested changes Jan 17, 2023

View reviewed changes

lib/workerThreads.js Outdated Show resolved Hide resolved

lib/workerThreads.js Outdated Show resolved Hide resolved

SimenB changed the base branch from master to next January 17, 2023 13:58

SimenB force-pushed the next branch 6 times, most recently from 3f25c32 to 47441a0 Compare March 9, 2023 11:52

SimenB force-pushed the next branch from 47441a0 to 84fae84 Compare March 9, 2023 11:57

rafixer changed the base branch from next to master March 13, 2023 14:45

rafixer force-pushed the worker_threads_support branch from 6aad6b5 to e2e185b Compare March 14, 2023 09:17

feat: support aggregating metrics across workers in a Node.js worker_…

7e1fc1f

…threads

rafixer force-pushed the worker_threads_support branch from e2e185b to 7e1fc1f Compare March 14, 2023 09:25

all tests suits passed

c85db46

bgalek approved these changes Mar 15, 2023

View reviewed changes

quixote15 mentioned this pull request Sep 11, 2023

Metrics not being computed when the application is forked into a child process willsoto/nestjs-prometheus#1889

Closed

Merge branch 'siimon:master' into worker_threads_support

f8cf3ad

SimenB requested changes Jul 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support aggregating metrics across workers in a Node.js worker_… #403

feat: support aggregating metrics across workers in a Node.js worker_… #403

rafixer commented Oct 15, 2020 •

edited

Loading

zbjornson left a comment •

edited

Loading

bgalek commented May 5, 2021 •

edited

Loading

SimenB commented May 12, 2021

alec-benson commented Jul 23, 2021

bgalek commented Jul 23, 2021

zbjornson commented Jan 1, 2022

bgalek commented Feb 18, 2022 •

edited

Loading

bgalek commented Feb 28, 2022

Zagorodnyi commented Jun 2, 2022 •

edited

Loading

rafixer commented Jun 2, 2022

SimenB left a comment •

edited

Loading

rafixer commented Jan 17, 2023

SimenB commented Jan 17, 2023

SimenB commented Jan 17, 2023

rafixer commented Jan 17, 2023

rafixer commented Jan 17, 2023

SimenB commented Jan 17, 2023

SimenB commented Mar 13, 2023

rafixer commented Mar 13, 2023

infloop commented Jun 26, 2023

iamyamakin commented Feb 5, 2024

matthieu-beteille commented Jul 8, 2024

nrathi commented Jul 25, 2024

SimenB left a comment

SimenB Jul 25, 2024

SimenB Jul 25, 2024

SimenB Jul 25, 2024

SimenB Jul 25, 2024

SimenB Jul 25, 2024

-				request.pending += message.workerRequests || 0;
+				if (message.workerRequests) {
+					request.pending += message.workerRequests;
+				}

	if (worker && worker instanceof Worker) {
	if (worker instanceof Worker) {

feat: support aggregating metrics across workers in a Node.js worker_… #403

Are you sure you want to change the base?

feat: support aggregating metrics across workers in a Node.js worker_… #403

Conversation

rafixer commented Oct 15, 2020 • edited Loading

zbjornson left a comment • edited Loading

Choose a reason for hiding this comment

bgalek commented May 5, 2021 • edited Loading

SimenB commented May 12, 2021

alec-benson commented Jul 23, 2021

bgalek commented Jul 23, 2021

zbjornson commented Jan 1, 2022

bgalek commented Feb 18, 2022 • edited Loading

bgalek commented Feb 28, 2022

Zagorodnyi commented Jun 2, 2022 • edited Loading

rafixer commented Jun 2, 2022

SimenB left a comment • edited Loading

Choose a reason for hiding this comment

rafixer commented Jan 17, 2023

SimenB commented Jan 17, 2023

SimenB commented Jan 17, 2023

rafixer commented Jan 17, 2023

rafixer commented Jan 17, 2023

SimenB commented Jan 17, 2023

SimenB commented Mar 13, 2023

rafixer commented Mar 13, 2023

infloop commented Jun 26, 2023

iamyamakin commented Feb 5, 2024

matthieu-beteille commented Jul 8, 2024

nrathi commented Jul 25, 2024

SimenB left a comment

Choose a reason for hiding this comment

SimenB Jul 25, 2024

Choose a reason for hiding this comment

SimenB Jul 25, 2024

Choose a reason for hiding this comment

SimenB Jul 25, 2024

Choose a reason for hiding this comment

SimenB Jul 25, 2024

Choose a reason for hiding this comment

SimenB Jul 25, 2024

Choose a reason for hiding this comment

rafixer commented Oct 15, 2020 •

edited

Loading

zbjornson left a comment •

edited

Loading

bgalek commented May 5, 2021 •

edited

Loading

bgalek commented Feb 18, 2022 •

edited

Loading

Zagorodnyi commented Jun 2, 2022 •

edited

Loading

SimenB left a comment •

edited

Loading