Error: Multiplexer is destroyed` on the server when libp2p circuit is doing something while the client crashes just after connecting

[mkg20001]

• Version: v18
• Platform: Linux - 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux, node -v: v8.9.4
• Subsystem: muxer

Type: bug

Severity: high

Description: Error: Multiplexer is destroyed on the server when libp2p floodsub is doing something while the client crashes just after connecting

Steps to reproduce the error:

Run a server

Run a client

Connect the client to the server and dial a protocol

The server should send some data and the client should throw when it receives it.

The server will now crash with Error: Multiplexer is destroyed

(Once again discovered this while writing libp2p-nodetrust - btw, here is the server-log.txt)

Note that this error does not occur 100% of the time.

[mkg20001]

Edit: Turns out circuit and not floodsub is causing this: at Dialer.canHop (/home/maciej/Projekte/libp2p/libp2p-nodetrust/server/node_modules/libp2p-circuit/src/circuit/dialer.js:81:7)

[daviddias]

@dryajov ^^

[dryajov]

@mkg20001 It seems to happen when circuit is trying to send the canHop message to the other peer, that happens to be the first thing that circuit will do when another peer just connected, but I don't see much going on besides that from the attached log.

One thing that comes to mind is, are you running on the same machine or are your nodes on different machines, if so, it might be a connection drop. Could you run this with debug='libp2p*' please?

[mkg20001]

@dryajov This was already run with DEBUG=libp2p*. What I was saying is that the server is crashing when the connection gets dropped due to the other peer crashing. I'm reporting this issue because this could easily be abused to crash important nodes maliciously.

[mkg20001]

Edit: Also note that in the log the error below "this is the actual crash" is server side and not on the client (the client was supposed to crash after receiving the servers response)

[dryajov]

Got you, thanks for clarifying.

Does something similar happen when circuit is disabled? I think canHop just happens to be the next thing that happens in the sequence of events? We should definitely handle muxer errors more graciously.

[mkg20001]

Yes. This time it took 2 runs to crash the server.

server-logs.txt

[dryajov]

Ah, this is it -

/home/maciej/Projekte/libp2p/libp2p-nodetrust/server/node_modules/libp2p-mplex/src/internals/index.js:95
      throw new Error('Multiplexer is destroyed')

We're mixing callbacks and throws....

[dryajov]

Here is the line - https://github.com/libp2p/js-libp2p-mplex/blob/bfc6b113e0026618cb72899b39d162eddeb42b3f/src/muxer.js#L54

We're never handling the throw anywhere.

[dryajov]

I think this should help - https://github.com/libp2p/js-libp2p-mplex/pull/75/files

Note that there are other throws, but they don't seem to be used anywhere. Specifically in -https://github.com/libp2p/js-libp2p-mplex/blob/bfc6b113e0026618cb72899b39d162eddeb42b3f/src/internals/index.js#L106 which gets called by https://github.com/libp2p/js-libp2p-mplex/blob/bfc6b113e0026618cb72899b39d162eddeb42b3f/src/internals/index.js#L136, which doesn't get used anywhere.

[dryajov]

Here is the required change in libp2p-switch libp2p/js-libp2p-switch#245

[daviddias]

@jacobheun was this handled by your other PRs?

[jacobheun]

@diasdavid this was ultimately resolved by libp2p/js-libp2p-mplex#79, which catches the error and returns it via the callback. This should be closable. This fix is available in libp2p-mplex 0.8.0.

[daviddias]

RAD! :D Thanks for double checking :)