Unhandled promise on the monitor caused 'out of memory'

[akolotov]

Here is report from the bridge monitor validating the POA-POA20 Bridge:

(node:27816) UnhandledPromiseRejectionWarning: Error: Invalid JSON RPC response: ""
    at Object.InvalidResponse (/home/bridgemon/bridge-monitor/node_modules/web3-core-helpers/src/errors.js:42:16)
    at XMLHttpRequest.request.onreadystatechange (/home/bridgemon/bridge-monitor/node_modules/web3-providers-http/src/index.js:73:32)
    at XMLHttpRequestEventTarget.dispatchEvent (/home/bridgemon/bridge-monitor/node_modules/xhr2/lib/xhr2.js:64:18)
    at XMLHttpRequest._setReadyState (/home/bridgemon/bridge-monitor/node_modules/xhr2/lib/xhr2.js:354:12)
    at XMLHttpRequest._onHttpRequestError (/home/bridgemon/bridge-monitor/node_modules/xhr2/lib/xhr2.js:544:12)
    at ClientRequest.<anonymous> (/home/bridgemon/bridge-monitor/node_modules/xhr2/lib/xhr2.js:414:24)
    at emitOne (events.js:116:13)
    at ClientRequest.emit (events.js:211:7)
    at TLSSocket.socketOnEnd (_http_client.js:423:9)
    at emitNone (events.js:111:20)
    at TLSSocket.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
(node:27816) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:27816) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
out of memory
FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
 1: node::Abort() [node]
 2: 0x11e7fec [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 6: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: v8::internal::Factory::NewOneByteInternalizedString(v8::internal::Vector<unsigned char const>, unsigned int) [node]
 8: v8::internal::OneByteStringKey::AsHandle(v8::internal::Isolate*) [node]
 9: v8::internal::StringTable::LookupKey(v8::internal::Isolate*, v8::internal::StringTableKey*) [node]
10: v8::internal::Factory::InternalizeOneByteString(v8::internal::Vector<unsigned char const>) [node]
11: v8::String::NewFromOneByte(v8::Isolate*, unsigned char const*, v8::NewStringType, int) [node]
12: node::IsolateData::IsolateData(v8::Isolate*, uv_loop_s*, unsigned int*) [node]
13: node::Start(uv_loop_s*, int, char const* const*, int, char const* const*) [node]
14: node::Start(int, char**) [node]
15: __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
16: 0x899ab1 [node]
Killed
Aborted (core dumped)

The config file:

HOME_RPC_URL=https://core.poa.network
FOREIGN_RPC_URL=<url to the eth mainnet>
HOME_BRIDGE_ADDRESS=0xB87b6077D59B01Ab9fa8cd5A1A21D02a4d60D358
FOREIGN_BRIDGE_ADDRESS=0xd819E948b14cA6AAD2b7Ffd333cCDf732b129EeD
POA20_ADDRESS=0x6758B7d441a9739b98552B373703d8d3d14f9e62
HOME_DEPLOYMENT_BLOCK=2477327
FOREIGN_DEPLOYMENT_BLOCK=5578725
GAS_PRICE_SPEED_TYPE=fast
GAS_LIMIT=300000
GAS_PRICE_FALLBACK=21
LEFT_TX_THRESHOLD=100

[patitonar]

Which version of bridge-monitor is POA-POA20 Bridge using?

[akolotov]

Here is answer from @phahulin: 8f0eeb577c033235c69da4ce39b80151f01a3a55. So, it means that it is on the release 1.0.0

[patitonar]

Are we using an Infura url for FOREIGN_RPC_URL? When trying to run 1.0.0 with an Infura url for foreign I'm getting the error Returned error: query returned more than 1000 results. This issue was solved for v2 on PR omni/bridge-monitor#27. Should we also apply that fix for monitor v1?

[akolotov]

No, a private node is being used there. I did not put the address in the configuration above in order to not reveal it to public.

[patitonar]

Do we have some more logs before the last error? I'm not able to reproduce the issue. Error Error: Invalid JSON RPC response: "" suggest that rpc call failed. From my testing calls to home rpc and foreign node seems to work OK, in some cases calls to get events on foreign private node take a little of time, but this is not happening every time for the same calls. Is it possible that home rpc or foreign private node were not available for a period of time? Perhaps several failed calls caused memory leak and finally out of memory error

[phahulin]

Here's stderr output from checkWorker job
cronWorker.err.gz

Most of errors are Invalid JSON RPC response: "", it is quite possible that some rpc endpoints were unavailable during checks. Also, in this log file there are a few invalid responses that contain huge json-like outputs - this might also cause a memory issue.

It looks like by default node.js doesn't terminate if there was uncaught exception in a promise, but just hangs 🤷‍♂️ Maybe we could somehow add a catch-all clause that would log the error and terminate?

[patitonar]

Thanks for the logs. We should improve error handling. Right now if an error is thrown by web3 call, it is being catched, logged and re-thrown 2 times; the last time by the catch-all in checkWorker so that's why the warning UnhandledPromiseRejectionWarning is showing up and could be causing the memory issue.

We should apply those fixes to the current version and for v1. The following improvements can be also applied to v1 version:

• Update package-lock.json to work with node.js 10 (for security and performance improvements)
• Fix missing reference error to GAS_PRICE_FALLBACK displayed on the logs: [validators] ReferenceError: GAS_PRICE_FALLBACK is not defined at getGasPrices (/home/bridgemon/bridge-monitor/validators.js:44:5) (for v1)