-
Notifications
You must be signed in to change notification settings - Fork 7.3k
(cluster) --debug-port is forced #8159
Comments
What do you mean by "forced"? node was called with the --debug flag, debug was explicitly requested, not "forced". In v0.11, if you call In v0.10, calling node with Why does the v0.11 behaviour cause a memory leak, and what do you mean by debug port "going crazy"? |
First, I'm saying forced because without adding a I could try to find a proper way of reproducing this, but if the master has no |
Yes the --debug flag is set by default (https://github.com/joyent/node/blob/master/lib/cluster.js#L291). Concerning the debug port, the cluster module increment the id every time a new worker is created. So assigning .debugPort + worker_id (https://github.com/joyent/node/blob/master/lib/cluster.js#L276), cause at a certain point to assign a port that is outside the port limit and so I get this:
So the worker is directly killed, then spawned back, then killed, till V8 throw strange errors (EMFILE...). |
I don't see the I agree that with enough workers, or a high-enough base port, eventually the debug-port strategy of "base+worker ID" won't work. That is a problem. But the "memory leak" and "debug forced" still isn't making sense to me. Missing from this report is how node is being called. |
@sam-github with regards to the "base+worker ID" problem - how do you feel about letting the user pass in their own |
+1 @cjihrig, we already hack the Linear port scan is not really great and should not be needed only for debugging tasks. |
User doesn't have access to the That said, since it means cluster debugging wouldn't work out of the box with node 0.12, much like it doesn't for v0.10, it gives me a feature to add to strong-supervisor, where I'd implement a debug port allocation strategy for all workers. To see how unworkable debugging clustered apps was in v0.10, check out the workarounds in this blog: http://strongloop.com/strongblog/whats-new-nodejs-v0-12-debugging-clusters/. That blog is pretty much a description of the problem to be solved. One thing here is that cluster IDs always go up, workers don't get the "lowest possible" ID. That make sense, I don't propose changing it. But for port, we could perhaps always allocate "debug ports" as the lowest possible above the base debug port. Until workers exit, the effect would be that the port allocation would be exactly as it is now, where worker debug port === base port + worker ID. But after workers start to exit, they would re-use the lowest available port. This would mean that the debugPort for a worker should be exposed as a property on a Worker, since its not necessarily derivable from simply knowing the worker ID. If As a no-one-has-time-for-this idea... it might be possible to ONLY have the master open a debug TCP port, and extend the debug protocol to include selection of a worker ID. The master would then use node cluster IPC to forward the debug commands to the currently selected woker. This would be much more like debugging "threaded" apps in other languages. Since the debug agent is getting re-written anyhow for v0.12 (having gotten deleted from v8), maybe this isn't out of the question. @bnoordhuis and @bajtos, what do you think? |
The V8 debugger protocol is close to being deprecated and will probably get dropped by upstream sometime in the next 6-12 months. So, while your suggestion would work as an interim solution, that means we'll have to revisit it in the not too distant future. (In case you're wondering what the deal is with the debugger protocol: the plan is to replace it with Chrome's remote debugging protocol. I don't know if your suggestion would work with that, too early to tell.) |
I got the impression from the v8 list that the new chrome remote debugging protocol is implemented still by using V8's My thinking was that if node is forced to implement its own debug protocol (or reimplement and maintain itself the old debug protocol), then it makes it easier for us to extend it with cluster-specific commands (before, we would have had to maintain a fork to v8, which is uncomfortable). Are you anticipatining that Node will eventually move over to a version of Chrome's socket.io-based debug protocol? Implementing it anew, or perhaps ripping it out of Chrome? |
That's the plan for v0.12. But for v0.13...
...the plan is to copy Chrome's implementation (where the word "copy" is part literal, part conceptual.) Chrome's debugger is a mix of C++ and JS. The C++ glue will have to be redone for node.js but the JS code can probably be used as-is. |
Still, the Could we only make this call when a |
Famous last words. If you're looking for a workaround for that error, couldn't you just ensure that the port is in the valid range instead of completely removing the flag altogether? |
What are you talking about? I'm just saying that Ofc it should be in a valid range, and that's the main part of the issue because after a while it raises over the 65635 limit and we can't do anything about it. But as @sam-github mentions the debug part of node will be rewritten so we might find a better solution to those cluster ports later. In the mean time just left this option to the user choice and if port is already taken throw an error. |
@joyent/node-coreteam nominate:0.12.3 This is a nasty surprise for callers of cluster.fork(). |
See: nodejs#8159 Per above discussion, cluster.fork() currently sends --debug-port to forked processes regardless of whether debug is enabled, and more importantly, without any bounds checking. This will rather unexpectedly crash and Node process that forks enough times. This patch simply bounds checks --debug-port and doesn't set arg if out of bounds (V8 requires 1024 < debug-port < 65535). This will prevent surprises to callers of cluster.fork() while waiting for the debug part of node to be rewritten as mentioned in issue discussion above.
See: nodejs#8159 Per above discussion, cluster.fork() currently sends --debug-port to forked processes regardless of whether debug is enabled, and more importantly, without any bounds checking. This will rather unexpectedly crash any Node process that forks enough times. This patch simply bounds checks --debug-port and doesn't set arg if out of bounds (V8 requires 1024 < debug-port < 65535). This will prevent surprises to callers of cluster.fork() while waiting for the debug part of node to be rewritten as mentioned in issue discussion above.
See: nodejs#8159 Per above discussion, cluster.fork() currently sends --debug-port to forked processes regardless of whether debug is enabled, and more importantly, without any bounds checking. This will rather unexpectedly crash any Node process that forks enough times. This patch simply bounds checks --debug-port and doesn't set arg if out of bounds (V8 requires 1024 < debug-port < 65535). This will prevent surprises to callers of cluster.fork() while waiting for the debug part of node to be rewritten as mentioned in issue discussion above.
I'm not positive I understand all of the above posts, but on node v0.12.2 my worker processes are being forked with debug set:
Is there a way to stop this from happening in this version? This app is in production and even though these ports are closed, I'm concerned about the performance implications of this, and the fact that this port cycling could eventually overflow if enough workers are [re]started |
Debug port is no longer set by default as of nodejs/node@309c0f4. |
@cjihrig Can nodejs/node@309c0f4 be backported to joyent/node in some form, or is everything in there a breaking change? |
@misterdjules conservatively, I would say not to backport it. It's only related to debugging, and has a very simple workaround to get the old behavior (pass a debug flag to the cluster master), but it's technically still a behavior change. Your call :-) |
It is a behaviour change, but the existing behaviour causes problems with 0.12. Also, the debug port passing to cluster workers was a feature introduced in 0.12, that started to cause problems when it was used in production, so fixing this kind of thing will make 0.12 more stable, I think. |
Specifically, what are these production problems? Isn't this simply setting the port but not actually putting each cluster worker into full debug mode? |
My main concern about --debug in production is that there might be a lot more plumbing wired up inside node when the debug flag is set that causes more overhead than necessary |
@nikmartin there is not |
@euskode https://github.com/joyent/node/blob/master/src/node.cc#L2903, node fails to start if port is out-of-bounds, even if debugging is never turned on and the port is never used. |
My issue here was a memory leak, consequence of the debug port increasing
|
Currently, each cluster worker is assigned an ever increasing --debug-port argument. A long running cluster application that does not use the debugger can run into errors related to the port range. This commit mitigates the problem by only setting the debug port if the master is started with debug arguments, or the user explicitly defines debug arguments for the worker. This commit also adds a new debug port offset counter that is only incremented when a worker is created that utilizes debugging. Fixes: nodejs/node-v0.x-archive#8159 Refs: #1524 PR-URL: #1949 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Oleg Elifantiev <oleg@elifantiev.ru>
Currently, each cluster worker is assigned an ever increasing --debug-port argument. A long running cluster application that does not use the debugger can run into errors related to the port range. This commit mitigates the problem by only setting the debug port if the master is started with debug arguments, or the user explicitly defines debug arguments for the worker. This commit also adds a new debug port offset counter that is only incremented when a worker is created that utilizes debugging. Fixes: nodejs/node-v0.x-archive#8159 Refs: #1524 PR-URL: #1949 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Oleg Elifantiev <oleg@elifantiev.ru>
Currently, each cluster worker is assigned an ever increasing --debug-port argument. A long running cluster application that does not use the debugger can run into errors related to the port range. This commit mitigates the problem by only setting the debug port if the master is started with debug arguments, or the user explicitly defines debug arguments for the worker. This commit also adds a new debug port offset counter that is only incremented when a worker is created that utilizes debugging. Fixes: nodejs/node-v0.x-archive#8159 Refs: #1524 PR-URL: #1949 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Oleg Elifantiev <oleg@elifantiev.ru>
Currently, each cluster worker is assigned an ever increasing --debug-port argument. A long running cluster application that does not use the debugger can run into errors related to the port range. This commit mitigates the problem by only setting the debug port if the master is started with debug arguments, or the user explicitly defines debug arguments for the worker. This commit also adds a new debug port offset counter that is only incremented when a worker is created that utilizes debugging. Fixes: nodejs/node-v0.x-archive#8159 Refs: #1524 PR-URL: #1949 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Oleg Elifantiev <oleg@elifantiev.ru>
Currently, each cluster worker is assigned an ever increasing --debug-port argument. A long running cluster application that does not use the debugger can run into errors related to the port range. This commit mitigates the problem by only setting the debug port if the master is started with debug arguments, or the user explicitly defines debug arguments for the worker. This commit also adds a new debug port offset counter that is only incremented when a worker is created that utilizes debugging. Fixes: nodejs/node-v0.x-archive#8159 Refs: #1524 PR-URL: #1949 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Oleg Elifantiev <oleg@elifantiev.ru>
Currently, each cluster worker is assigned an ever increasing --debug-port argument. A long running cluster application that does not use the debugger can run into errors related to the port range. This commit mitigates the problem by only setting the debug port if the master is started with debug arguments, or the user explicitly defines debug arguments for the worker. This commit also adds a new debug port offset counter that is only incremented when a worker is created that utilizes debugging. Fixes: nodejs/node-v0.x-archive#8159 Refs: #1524 PR-URL: #1949 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl> Reviewed-By: Oleg Elifantiev <oleg@elifantiev.ru>
Hi,
https://github.com/joyent/node/blob/master/lib/cluster.js#L291
Why is debugging forced in the cluster module? Is it a wanted behavior? Did someone forgot to remove it?
It's breaking some code of ours (Unitech/pm2#610 (comment)) and leading to a memory leak:
May I PR on this to remove those debug forcing stuff?
The text was updated successfully, but these errors were encountered: