child_process: truncate output when maxBuffer is exceeded #24951

Fishrock123 · 2018-12-11T01:14:41Z

Preserves truncated output for child_process.exec() when maxBuffer is exceeded.

This is particularly useful for commands which have indistinguishable
error codes for what output they produce.

I am presently running into this and just vendoring exec with a similar change into some code because... well, I need it, and don't want to re-write all of exec by hand.

~~Done as an added option / semver-minor because I want it sooner. IMO we should make this the default behavior but I think there's a chance something relies on the old behavior.~~
Done as patch as per discussion.

The old behavior was also completely untested so... commit 1 fixes that.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

nodejs-github-bot · 2018-12-11T01:14:42Z

@Fishrock123 build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/1891/pipeline

BridgeAR · 2018-12-11T19:03:46Z

@Fishrock123 I would argue this is a kind of bug fix and we do not need the option for it. We could just run CITGM and see if anyone relies on this behavior but I actually doubt that a lot of code relies on the specific error code and that the output is empty in such a case.

Fishrock123 · 2018-12-12T00:06:22Z

@nodejs/child_process, and also @nodejs/citgm I suppose. (Since there's no way I can decipher output even if I start a run...)

BridgeAR · 2018-12-12T14:31:52Z

I am happy to look into the CITGM result if you refactor the code / open an alternative one which adds the functionality without option.

Fishrock123 · 2018-12-12T19:03:08Z

@BridgeAR updated as per discussion

CI: https://ci.nodejs.org/job/node-test-pull-request/19457/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1682/

test/parallel/test-child-process-exec-maxBuffer.js

addaleax · 2018-12-15T20:43:00Z

Fresh CI: https://ci.nodejs.org/job/node-test-pull-request/19566/

Trott · 2018-12-15T21:44:42Z

Fresh CI: https://ci.nodejs.org/job/node-test-pull-request/19566/

Windows test failures look relevant.

Trott

Blocking on relevant CI failures on Windows. Feel free to dismiss once those are addressed. I'm just blocking to make sure no one accidentally lands this before it's fixed.

Fishrock123 · 2018-12-19T02:32:18Z

🤨(wat.)

(Why did 4 of the same test case run in fanned?)

anyways:

assert.js:86
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected

+ '"hell'
- 'hello'
    at runChecks (c:\workspace\node-test-binary-windows\test\parallel\test-child-process-exec-maxBuffer.js:10:10)
    at cp.exec.common.mustCall (c:\workspace\node-test-binary-windows\test\parallel\test-child-process-exec-maxBuffer.js:31:7)
    at c:\workspace\node-test-binary-windows\test\common\index.js:335:15
    at ChildProcess.exithandler (child_process.js:301:5)
    at ChildProcess.emit (events.js:189:13)
    at maybeClose (internal/child_process.js:978:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:265:5)

So.... on windows (cmd.exe?) echo "x" returns "x" and not x?...

Fishrock123 · 2018-12-19T02:42:04Z

New CI: https://ci.nodejs.org/job/node-test-pull-request/19669/

Trott · 2018-12-19T04:57:07Z

Resume Build CI: https://ci.nodejs.org/job/node-test-pull-request/19672/ ✔️

test no longer fails

Trott · 2018-12-31T06:46:52Z

No one has reviewed this since the last commit was pushed. @addaleax @BridgeAR Can you confirm that this still looks good to you? Anyone else want to review it? @nodejs/child_process

@Fishrock123 Are you still in Camp "I Like To Land My Own PRs, Please Don't Land Them For Me, KThxBai"? Or have you moved over to Camp "Sure, Whatevs, Land My PRs If You Want, Save Me The Trouble, Kewl"?

Fishrock123 · 2019-01-04T18:52:24Z

I would prefer to land them myself to avoid surprises.

Anyways this should be ready to land? It's not clear to me if people are actually ok with this considering no one from the relevant domain has ever commented.

Trott · 2019-01-04T19:35:34Z

Anyways this should be ready to land?

Technically, yes, I believe so.

It's not clear to me if people are actually ok with this considering no one from the relevant domain has ever commented.

If you'd be more comfortable getting the approval of someone specific, please do, obviously. (As you know, GitHub notifications can be overwhelming so it's sometimes a good idea to ping folks via other channels.)

addaleax

Yes, still LGTM

BridgeAR

Still LGTM

Fishrock123 · 2019-01-12T21:52:32Z

https://ci.nodejs.org/job/node-test-commit-arm/21379/nodes=ubuntu1604-arm64/testReport/(root)/test/known_issues_test_vm_timeout_escape_queuemicrotask/

test.known_issues/test-vm-timeout-escape-queuemicrotask
fail (0)
(node:54495) ExperimentalWarning: queueMicrotask() is experimental.

Expands this test case to check what happens to stdout/stderr when maxBuffer is exceeded. Also changes how cases are checked so that assertion stacks are tracable to their test case, aka 'make it actually debuggable'. PR-URL: #24951 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Anna Henningsen <anna@addaleax.net>

Preserves truncated output for `child_process.exec()` when `maxBuffer` is exceeded. This is particularly useful for commands which have indistinguishable error codes for what output they produce. PR-URL: #24951 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Anna Henningsen <anna@addaleax.net>

Fix test/pummel/test-exec.js which broke as a result of e47f972 (nodejs#24951). (Until very recently, pummel tests were not run at all in CI and currently only run nightly on master.)

Expands this test case to check what happens to stdout/stderr when maxBuffer is exceeded. Also changes how cases are checked so that assertion stacks are tracable to their test case, aka 'make it actually debuggable'. PR-URL: #24951 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Anna Henningsen <anna@addaleax.net>

Preserves truncated output for `child_process.exec()` when `maxBuffer` is exceeded. This is particularly useful for commands which have indistinguishable error codes for what output they produce. PR-URL: #24951 Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de> Reviewed-By: Anna Henningsen <anna@addaleax.net>

Fix test/pummel/test-exec.js which broke as a result of e47f972 (nodejs#24951). (Until very recently, pummel tests were not run at all in CI and currently only run nightly on master.) PR-URL: nodejs#25677 Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Anto Aravinth <anto.aravinth.cse@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>

Fix test/pummel/test-exec.js which broke as a result of e47f972 (#24951). (Until very recently, pummel tests were not run at all in CI and currently only run nightly on master.) PR-URL: #25677 Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Anto Aravinth <anto.aravinth.cse@gmail.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>

Notable Changes: * events: * For unhandled `error` events with an argument that is not an `Error` object, the resulting exeption will have more information about the argument. #25621 * child_process: * When the `maxBuffer` option is passed, `stdout` and `stderr` will be truncated rather than unavailable in case of an error. #24951 * policy: * Experimental support for module integrity checks through a manifest file is implemented now. #23834 * n-api: * The `napi_threadsafe_function` feature is now stable. #25556 * report: * An experimental diagnostic API for capturing process state is available as `process.report` and through command line flags. #22712 * tls: * `tls.connect()` takes a `timeout` option analogous to the `net.connect()` one. #25517 * worker: * `process.umask()` is available as a read-only function inside Worker threads now. #25526 * An `execArgv` option that supports a subset of Node.js command line options is supported now. #25467 PR-URL: #25687