-
Notifications
You must be signed in to change notification settings - Fork 490
refactor(@embark/blockchain_process): swallow errors, revise streams #1181
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
This PR works together with #1166 to prevent hangs and crashes when running / stopping embark processes. |
| setImmediate(destroy); | ||
| }; | ||
| req.on('error', swallowECONNRESET); | ||
| socket.on('error', swallowECONNRESET); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can this cause problem for other types of errors?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like I made a mistake last night when rebasing and used an older commit... dangers of develping / testing on 3 different computers at the same time. I'm going to sort that out now.
Here's what that part will look like:
if (ws) {
server.on('upgrade', (msg, socket, head) => {
const swallowError = (err) => {
console.error(`Proxy: Network error '${err.message}'`);
};
socket.on('error', swallowError);
proxy.upgrade(msg, socket, head);
});
}I checked yesterday and here's what the docs say about net.Socket's 'error' event:
Emitted when an error occurs. The
'close'event will be called directly following this event.
Since the socket will be closed, no need to destroy it manually and there's not much more we can / should do at that point other than report with console.error().
Also, the variable I had named req previously is actually an http.IncomingMessage instance. It lacks an 'error' event and its destroy() method just destroys the underlying socket (msg.socket, alias msg.connection), which is identity with the object in the socket variable, so we don't need to do anything with it.
For reasons unknown, `ECONNRESET` errors on websocket connections to embark's blockchain proxy are not automatically handled on Windows as they are on macOS and Linux (or those errors aren't happening on those platforms, it's difficult to determine). Explicitly swallow such errors so the blockchain process doesn't crash. Prior to this PR, the crash-behavior can be reproduced on Windows by running `embark blockchain` and `embark run` in separate terminals and quitting `embark run` while `embark blockchain` is still running. Consistently use the `simples` package's `WsParser` to process websocket traffic instead of using `WsParser` for requests and the `ws` package's `Websocket.Receiver` for responses. Consistently use `pump` to connect parser streams instead of using `pump` in some places and `chain` in others. Drop use of `cloneable` (and the package dependency) since it was used previously in hopes it would fix the errors, but it's unnecessary and didn't fix them.
f1ae969 to
5ec9d0c
Compare
The problems described in embark PR #1166 can be resolved by implementing the blockchain proxy with `http-proxy` directly instead of using `express` together with `http-proxy-middleware`. The ultimate cause of the buggy behavior (the "stuck sockets" problems described in #1166) is unknown. The need to swallow some errors as described in embark PR #1181 is also eliminated by dropping `http-proxy-middleware` and `express`.
The problems described in embark PR #1166 can be resolved by implementing the blockchain proxy with `http-proxy` directly instead of using `express` together with `http-proxy-middleware`. The ultimate cause of the buggy behavior (the "stuck sockets" problems described in #1166) is unknown. The need to swallow some errors as described in embark PR #1181 is also eliminated by dropping `http-proxy-middleware` and `express`.
This PR replaces #1166. The "stuck sockets" bug is addressed in #1195 so there is no longer a need to use timeouts. However a few aspects of the original PR are still useful, and lessons learned from #1166, #1181, and #1195 can be put to good use. Use a websocket client from the `ws` package when pinging websocket endpoints instead of manually building a request header. The `'upgrade'` event being listened for was never actually firing; and though a response was received for those pings, the response messages indicated problems with those requests. It seems cleaner to use a proper websocket client and callback success upon the `ws` client's `'open'` event. Abstract error and success handling across websocket and http pings. Report network errors other than `ECONNREFUSED`. Only `ECONNREFUSED` is expected, as that genuinely indicates an endpoint isn't accepting connections at the specified host and port. If other kinds of network errors are occurring, it will be helpful to have a visual indicator to prompt investigation. After success or the first error, cleanup the ping's request/connection immediately since we're not awaiting `'data'` events on an http request and we don't want to leave a websocket connection open. Don't callback any additional `'error'` events that might fire after the first `'error'` event, but do report them.
This PR replaces #1166. The "stuck sockets" bug is addressed in #1195 so there is no longer a need to use timeouts. However a few aspects of the original PR are still useful, and lessons learned from #1166, #1181, and #1195 can be put to good use. Use a websocket client from the `ws` package when pinging websocket endpoints instead of manually building a request header. The `'upgrade'` event being listened for was never actually firing; and though a response was received for those pings, the response messages indicated problems with those requests. It seems cleaner to use a proper websocket client and callback success upon the `ws` client's `'open'` event. Abstract error and success handling across websocket and http pings. Report network errors other than `ECONNREFUSED`. Only `ECONNREFUSED` is expected, as that genuinely indicates an endpoint isn't accepting connections at the specified host and port. If other kinds of network errors are occurring, it will be helpful to have a visual indicator to prompt investigation. After success or the first error, cleanup the ping's request/connection immediately since we're not awaiting `'data'` events on an http request and we don't want to leave a websocket connection open. Don't callback any `'error'` events that might fire after the first `'error'` event or a success event, but do report them.
The problems described in embark PR #1166 can be resolved by implementing the blockchain proxy with `http-proxy` directly instead of using `express` together with `http-proxy-middleware`. The ultimate cause of the buggy behavior (the "stuck sockets" problems described in #1166) is unknown. The need to swallow some errors as described in embark PR #1181 is also eliminated by dropping `http-proxy-middleware` and `express`.
This PR replaces #1166. The "stuck sockets" bug is addressed in #1195 so there is no longer a need to use timeouts. However a few aspects of the original PR are still useful, and lessons learned from #1166, #1181, and #1195 can be put to good use. Use a websocket client from the `ws` package when pinging websocket endpoints instead of manually building a request header. The `'upgrade'` event being listened for was never actually firing; and though a response was received for those pings, the response messages indicated problems with those requests. It seems cleaner to use a proper websocket client and callback success upon the `ws` client's `'open'` event. Abstract error and success handling across websocket and http pings. Report network errors other than `ECONNREFUSED`. Only `ECONNREFUSED` is expected, as that genuinely indicates an endpoint isn't accepting connections at the specified host and port. If other kinds of network errors are occurring, it will be helpful to have a visual indicator to prompt investigation. After success or the first error, cleanup the ping's request/connection immediately since we're not awaiting `'data'` events on an http request and we don't want to leave a websocket connection open. Don't callback any `'error'` events that might fire after the first `'error'` event or a success event, but do report them.
The problems described in embark PR #1166 can be resolved by implementing the blockchain proxy with `http-proxy` directly instead of using `express` together with `http-proxy-middleware`. The ultimate cause of the buggy behavior (the "stuck sockets" problems described in #1166) is unknown. The need to swallow some errors as described in embark PR #1181 is also eliminated by dropping `http-proxy-middleware` and `express`.
The problems described in embark PR #1166 can be resolved by implementing the blockchain proxy with `http-proxy` directly instead of using `express` together with `http-proxy-middleware`. The ultimate cause of the buggy behavior (the "stuck sockets" problems described in #1166) is unknown. The need to swallow some errors as described in embark PR #1181 is also eliminated by dropping `http-proxy-middleware` and `express`.
This PR replaces #1166. The "stuck sockets" bug is addressed in #1195 so there is no longer a need to use timeouts. However a few aspects of the original PR are still useful, and lessons learned from #1166, #1181, and #1195 can be put to good use. Use a websocket client from the `ws` package when pinging websocket endpoints instead of manually building a request header. The `'upgrade'` event being listened for was never actually firing; and though a response was received for those pings, the response messages indicated problems with those requests. It seems cleaner to use a proper websocket client and callback success upon the `ws` client's `'open'` event. Abstract error and success handling across websocket and http pings. Report network errors other than `ECONNREFUSED`. Only `ECONNREFUSED` is expected, as that genuinely indicates an endpoint isn't accepting connections at the specified host and port. If other kinds of network errors are occurring, it will be helpful to have a visual indicator to prompt investigation. After success or the first error, cleanup the ping's request/connection immediately since we're not awaiting `'data'` events on an http request and we don't want to leave a websocket connection open. Don't callback any `'error'` events that might fire after the first `'error'` event or a success event, but do report them.
For reasons unknown,
ECONNRESETerrors on websocket connections to embark's blockchain proxy are not automatically handled on Windows as they are on macOS and Linux (or those errors aren't happening on those platforms, it's difficult to determine). Explicitly swallow such errors so the blockchain process doesn't crash. Prior to this PR, the crash-behavior can be reproduced on Windows by runningembark blockchainandembark runin separate terminals and quittingembark runwhileembark blockchainis still running.Consistently use the
simplespackage'sWsParserto process websocket traffic instead of usingWsParserfor requests and thewspackage'sWebsocket.Receiverfor responses.Consistently use
pumpto connect parser streams instead of usingpumpin some places andchainin others. Drop use ofcloneable(and the package dependency) since it was used previously in hopes it would fix the errors, but it's unnecessary and didn't fix them.