Handle web socket disconnects

[mikedpid]

Im making an app that monitors a specific addresses' balance, im using Infura's public websocket endpoint and web3@1.0
I often get disconnected and get "Connection not open on send()"
I tried using a set interval that polls web3.eth.net.isListening() that resets the web3 provider, making it to reconnect to the node but in some cases i get Unhandled rejection Error: CONNECTION ERROR: Couldn't connect to node on IPC. and the app crashes. How am i supposed to handle the disconnects and make my app reconnect to the node?

setInterval(function() { web3.eth.net.isListening() .then() .catch(e => { console.log('[ - ] Lost connection to the node, reconnecting'); web3.setProvider(config.infura_ws); app(); // this just subscribes to newBlockHeaders event }) }, interval);

[mikedpid]

Still looking for help, im checking for an address' balance every 5seconds and after a couple minutes i keep getting connection not open on send()

[7Ds7]

I have this problem as well although, i am not subscribing on a setInterval, my workaround is catching the event promise error and when it happens, i just reconnect it again with a new Web3(infura_ws_url) replacing the old one

Not sure if the problem is from web3 or infura (ws)

[mikedpid]

I believe its the infura ws node thats crashing, i tried a try catch block but the app still crashes sometimes. Im trying to subscribe to new pending txs but its unstable af and after a couple seconds i get loads of promise errors and more than half of the transactions are not being checked. Do i fix this issue if i host myself a node that will be accessed only by my app?

[7Ds7]

Using my own geth instance for testing purposes i did not get this error, although you might try not having it run on a setInterval, you can check the subscribed event error, i am doing something along these lines

function listenToEvent() {}
    contract.events.SomeEvent({
          toBlock: 'latest'
        }, (error, event) => { console.log(event); })
          .on('data', (event) => {
            console.log('--contract.events.SomeEvent--');
          })
          .on('changed', (event) => {
           console.log('--SomeEvent--Changed');
         })
          .on('error', (e) => {
            console.log('--SomeEvent--Error');
            console.log(e);
           your_web3_instance = new Web3('wss://infura_ws_url') });
           wait_for_your_web3_instance_to_be_connected
           listenToEvent()
      });

[mikedpid]

Im subscribing to the new pending txs and checking each transaction’s “to” and “value” fields, its for a payment system but as i said, using infura and it crashes after a couple seconds. Will try using my own parity node to see if the connection still crashes, will let you know!

[jamboj]

same issue here.. even if I subscribe to new blocks/subscribe to events and poll a contract calling an immulable method periodically, sending a transaction/calling a payable method would often result in immediate Connection not open on send() error

[Plinpod]

I am running into the exact same issue. Did anyone have a fix for this?

[BransonGitomeh]

i'm on parity and i've been getting disconnections too

[cryptophonic]

this is also happening trying to extend web3 1.0 to handle debug_traceTransaction over websocket. wsapi is set to admin,debug,..

web3.extend({
  property: 'debug',
  methods: [{
    name: 'traceTransaction',
    call: 'debug_traceTransaction',
    params: 1
  }]
])

await web3.debugIt.traceTransaction('0xe09418975ecca6dfb43aa9ee4ac64597f84b4dcc920e6872d2fb9fd592671f35')
Error: CONNECTION ERROR: Couldn't connect to node on IPC.

[Plinpod]

Anyone have a solution to this?

[kingjerod]

Having the same issue. Connected to Infura Rinkeby websocket endpoint and it disconnects within minutes.

[damozhang]

Same issue here.

connection not open on send()
Unhandled rejection Error: connection not open

I'm using public websocket endpoint to check the latest block height 'web3.eth.getBlock("latest") ' every 5 seconds.

Does Infura's api have any limitations?

[elie222]

Same issue here. Surely there's a way to reconnect on error?

[elie222]

Resetting the provider upon disconnect semi works:

web3.setProvider(NODE_URL)

The only problem is if that if you were listening to events you need to start listening again.

[BransonGitomeh]

@elie222 how to know disconnection? if there was an event released on disconnection then we could use that to restart listening, but i cant find that in docs

[dmihal]

You can listen to events from the WebSocketProvider to detect disconnects.

  const provider = new Web3.providers.WebsocketProvider(path);
  provider.on('error', e => console.error('WS Error', e));
  provider.on('end', e => console.error('WS End', e));

[Plinpod]

@dmihal This is exactly what I needed. Can't believe I haven't been able to find this code all this time. Thank You!

[kingjerod]

I have noticed even when running my own private geth instance that the websocket will disconnect after almost exactly 1 minute, even with reconnects. The only way around this is to send a request like the previously mentioned web3.eth.net.isListening() call.

Edit: In the options for the WebsocketProvider I'm using {timeout: 30000}. Not sure if that somehow corresponds to 1 minute, but I did try upping it to 120000 just to see what happens. Still disconnects after ~1 minute.

[BransonGitomeh]

i used to get disconnections with parity running on a really small server, i dont get it as often now with a larger ram capacity

…

On Sat, Apr 28, 2018 at 1:53 AM, Jerod King ***@***.***> wrote: I have noticed even when running my own private geth instance that the websocket will disconnect after almost exactly 1 minute, even with reconnects. The only way around this is to send a request like the previously mentioned web3.eth.net.isListening() call. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1354 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKqeD-IaZ6U-glekTox_wuoq6GntwJaCks5ts6FNgaJpZM4R4h01> .

-- *Much Regards* Branson G Kuria Systems Developer Sabek Systems P.O Box 78024 Nairobi. www.sabek.co.ke +254 711 657108

[mkhraisha]

I tried using a snippet like the following but i could never get it to reconnect, i keep getting a CONNECTION ERROR: Couldn't connect to node on WS.

const provider = web3.currentProvider;
provider.on("error",e=> handleDisconnects(e));
provider.on("close",e=> handleDisconnects(e));
	
	function handleDisconnects(e) {
		console.log("error",e);
		web3.setProvider('wss://ropsten.infura.io/ws')
		
	}

[dmihal]

@mkhraisha I haven't been able to test this, but I think I had a similar issue that I fixed using ws: instead of wss:

[mkhraisha]

@dmihal trying to connect to Web3.providers.WebsocketProvider('ws://ropsten.infura.io/ws') wouldn't connect even if i try a simple
var web3 = new Web3(Web3.givenProvider || new Web3.providers.WebsocketProvider('ws://ropsten.infura.io/ws'));

[mkhraisha]

are there any updates on a fix for this issue?

[anhnt]

This works for me

const RINKEBY_WSS = "wss://rinkeby.infura.io/ws";
var provider = new Web3.providers.WebsocketProvider(RINKEBY_WSS);
var web3 = new Web3(provider);

provider.on('error', e => console.log('WS Error', e));
provider.on('end', e => {
    console.log('WS closed');
    console.log('Attempting to reconnect...');
    provider = new Web3.providers.WebsocketProvider(RINKEBY_WSS);

    provider.on('connect', function () {
        console.log('WSS Reconnected');
    });
    
    web3.setProvider(provider);
});

[cmaliwal]

@anhnt , I tried your solution. It's just shows the Wss Reconnected in console but it don't reconnect to wss .
so sorry ! This solution is not working.

[cmaliwal]

@dmihal , As per your solution if we use Ws instead of Wss, it will not work with HTTPS.

Error : Failed to construct 'WebSocket': An insecure WebSocket connection may not be initiated from a page loaded over HTTPS.

so sorry! This solution is not working.

[anhnt]

@cmaliwal are you using it on browser. My solution works great on nodejs but havent tested it on browser just yet, I think there is issue with ssl certificate not web3

[woodydeck]

@melnikaite

Thank you. I was afraid of that. :/ I am using web3@1.0.0-beta.55 . Should I be switching to 2.0?

[tsujp]

This issue should be reopened.

Watching the provider for IPC, or WebSocket does work on either 2.0.0-alpha or beta.55 @melnikaite. Furthermore, intentionally crashing a geth node also does not trigger these events.

At the moment it seems impossible to detect any kind of websocket close. The only event that works is 'connect'. 'error' and 'end' are not detected at all.

Sample:

const Web3 = require('web3')
const net = require('net')

const web3 = new Web3('/path/to/geth.ipc', net)
const provider = web3.currentProvider

// works
provider.on('connect', () => {
  console.log('connected')
})

// doesn't work
provider.on('error', e => {
  console.error('error', e)
})

// doesn't work
provider.on('end', e => {
  console.error('end', e)
})

// EDIT: THIS WORKS use the close event instead
provider.on('close', e => {
  console.error('close', e)
})

EDIT: I couldn't find any information about this in the documentation, so I inspected the prototype of the provider and it emits a close event as per standard. Watching for close handles crashes at the node. There are also events ready and timeout. All is fine now – unsure why the focus on end versus close. As far as I am aware end is not part of the WebSocket spec so when I decided to sanity check the prototype and try close it worked.

[woodydeck]

@tsujp

You are correct, only connect works. @melnikaite is correct as well it seems. Even if you reconnect, the subscription/getPastEvents no longer listens for incoming events. You have to nuke the whole script, which is not a workaround for production.

[ivanvpan]

There are different reasons for disconnect. When I shutdown my local node manually the provider does get an end request. When I drop the internet connection on purpose (shut off wifi) then no events fire at all (no end, close, etc.). Perhaps in some instance close does fire, but I have not seen it yet. I'm using web3 1.2.1, I couldn't get 2-alpha to work correctly and quickly gave up.

[naddison36]

For future devs looking for a solution to keeping a WebSocket connection to Infura alive, this worked for me

const provider = new Web3.providers.WebsocketProvider(
            config.PROVIDER_WS,
            {
                // @ts-ignore
                clientConfig: {
                    keepalive: true,
                    keepaliveInterval: 60000
                }
            }
        );

@ts-ignore is required for TypeScript as the Web3js typings is expecting a string for the clientConfig option which is wrong. The doco for the WebSocketClient used by Web3js also does not properly cover the keepalive config. https://github.com/theturtle32/WebSocket-Node/blob/master/docs/WebSocketClient.md#client-config-options

The WebSocketConnection code with the keepalive config is at https://github.com/theturtle32/WebSocket-Node/blob/574b703ca978709832c6c2f67d6c1de376d432b5/lib/WebSocketConnection.js#L111

[naddison36]

I should add keepaliveInterval in my previous post is in milliseconds. So the above will ping Infura via the WebSocket every minute.

[joZephhh]

Thanks for your solution @naddison36, seems to resolve the auto idle disconnect problem for me.

Otherwise, it seems it does not prevent other disconnect events. To resolve them and thus ensure a connection stability, I'm personally adding reconnect option on the WebsocketProvider :

reconnect: {
  auto: true,
  delay: 1000,
  maxAttempts: 10,
},

which is set to falseby default : https://github.com/ethereum/web3.js/blob/1.x/packages/web3-providers-ws/src/index.js#L45

[naddison36]

Thanks @joZephhh. I've added the reconnect config to my listener service.

I was trapping the close event on the provider and crashing my process as per the following. Since my process was running on Kubernetes, it would be restarted but your approach is much better.

provider.on("close", err => {
	logger.error(`WebSocket connection closed. Error code ${err.code}, reason "${err.reason}"`);
	process.exit(1);
});

[1baga]

@joZephhh and @naddison36 Thank you so much... I am really grateful

[snitovets]

I looking for workaround how to handle this error within my app. It was a surprise why websocket not store pending request to resolve after reconnection.

/app/node_modules/web3-core-helpers/lib/errors.js:63
return new Error('CONNECTION ERROR: Provider started to reconnect before the response got received!');
Error: CONNECTION ERROR: Provider started to reconnect before the response got received!
at Object.PendingRequestsOnReconnectingError (/app/node_modules/web3-core-helpers/lib/errors.js:63:16)
at /app/node_modules/web3-providers-ws/lib/index.js:332:37
at Map.forEach ()
at WebsocketProvider.reconnect (/app/node_modules/web3-providers-ws/lib/index.js:331:28)
at WebsocketProvider._onClose (/app/node_modules/web3-providers-ws/lib/index.js:149:14)
at W3CWebSocket._dispatchEvent [as dispatchEvent] (/app/node_modules/yaeti/lib/EventTarget.js:115:12)
at W3CWebSocket.onClose (/app/node_modules/websocket/lib/W3CWebSocket.js:228:10)
at WebSocketConnection. (/app/node_modules/websocket/lib/W3CWebSocket.js:201:17)
at WebSocketConnection.emit (node:events:379:20)
at WebSocketConnection.EventEmitter.emit (node:domain:470:12)
at WebSocketConnection.handleSocketClose (/app/node_modules/websocket/lib/WebSocketConnection.js:389:14)
at TLSSocket.emit (node:events:391:22)
at TLSSocket.EventEmitter.emit (node:domain:470:12)
at node:net:667:12
at TCP.done (node:_tls_wrap:573:7)

[lucai11]

Error: CONNECTION ERROR: Provider started to reconnect before the response got received!

also getting this one..did you manage to find a solution?

[snitovets]

@sloambbr Unfortunately, not yet

[GregTheGreek]

Would you mind opening a new issue if you believe there to be one.

Thank you!

[ko0f]

For future reference, there are some extra parameters required for full auto-reconnect websocket provider -

            const provider = new Web3.providers.WebsocketProvider(url, {
                reconnect: {
                    auto: true,
                    delay: 1000, // ms
                    onTimeout: false,
                    // maxAttempts: 
                },
                timeout: 5000, // ms
                clientConfig: {
                    maxReceivedFrameSize: 10000000000,
                    maxReceivedMessageSize: 10000000000,
                    keepalive: true,
                    keepaliveInterval: 1000, // ms
                    dropConnectionOnKeepaliveTimeout: true,
                    keepaliveGracePeriod: 4000, // ms
                }
            });

[LukeSamkharadze]

This works for me

const RINKEBY_WSS = "wss://rinkeby.infura.io/ws";
var provider = new Web3.providers.WebsocketProvider(RINKEBY_WSS);
var web3 = new Web3(provider);

provider.on('error', e => console.log('WS Error', e));
provider.on('end', e => {
    console.log('WS closed');
    console.log('Attempting to reconnect...');
    provider = new Web3.providers.WebsocketProvider(RINKEBY_WSS);

    provider.on('connect', function () {
        console.log('WSS Reconnected');
    });
    
    web3.setProvider(provider);
});

What if it gets end second time? @anhnt

[LukeSamkharadze]

reconnect: {
  auto: true,
  delay: 1000,
  maxAttempts: 10,
},

@joZephhh does this fix all disconnects or do we still have to listen on end events and create new provider?

[joZephhh]

reconnect: {
  auto: true,
  delay: 1000,
  maxAttempts: 10,
},

@joZephhh does this fix all disconnects or do we still have to listen on end events and create new provider?

As far as I remember it resolved all my disconnects issues! 💪

[georgiod9]

reconnect: {
  auto: true,
  delay: 1000,
  maxAttempts: 10,
},

@joZephhh does this fix all disconnects or do we still have to listen on end events and create new provider?

This does in fact help reconnect to the websocket provider, however if your code relies on a constant connection to the rpc node and some specific setup of listening to a contract address or wallet, then aside from the reconnecting the provider, you would also need to update your contract or wallet object.

For those still encountering problems, it could be deeper that just the reconnect configuration option.
The way I handled it is as follows:

Since you configured the provider to restart automatically on drop, emit a "reconnect" event to catch a connection drop

 network.provider.on('connect', () => {
                console.log(`NetworkHandler: Chain ${network.name} Websocket connect!`);
               
                this.eventsEmitter.emit('networkHandler-reconnect', { chainName: network.name, chainId: network.chainId }); //catch a connection drop and reconnection from provider here
                this.eventsEmitter.emit("networkHandler-connect", this.logger.types.info, { chainId: network.chainId, message: `NetworkHandler: Chain ${network.name} Websocket connect!` });
            });

and then in your code where you used the provider instance to instantiate your contract object:
Catch this event in your code where you instantiated the contract object, and trigger a reset of the listener

// Internal listener for provider connection drop
    setupReconnectListeners() {
        this.networkHandler.eventsEmitter.on('networkHandler-reconnect', (data) => {
            console.log(`NetworkHandler: Handling provider reconnect...`);

            // Reattach contract listeners on this chain
            this.contractsHandler.handleReconnect(data.chainId, this.setupListener.bind(this)); //Here im binding the setup listener function to reset the contract listeners
        })
    }

This way, when the provider reconnects automatically, you can reattach the listeners of your contracts.

Assuming you have a listener to a contract/address for some specific "transfer" event, you would recall your function to setup the listener again with the new provider.

Without doing so, your contract object will keep listening to events from the old dropped provider.

In my code, im resetting the listener when the connection drops:

//Setup an "allEvents" listener for a smart contract
    setupListener(deployedAddress, abi) {
        console.log(` > Setup listener for contract ${deployedAddress}`)
        const contract = this.contractsHandler.addressToContract.get(deployedAddress); //

        const chainId = this.contractsHandler.getChainIdFromAddress(deployedAddress);

        contract.events.allEvents(async (error, incomingEvent) => { // Reset the listeners of your contract

Hope this helps!

[cheildo]

@georgiod9 Thanks, this solved my problem. I had the same issue, even tho i handled the reconnect issue, my code coudn't catch the new events that was emitted after the reconnection.

If your contract is listening to events, reconnecting to the websocket provider alone is not enough. You need to reattach the event listeners

provider.on("error", (error) => logger.error("web3 WebSocket error"));
provider.on("end", (error) => logger.info("web3: WebSocket connection ended"));
provider.on("timeout", () => logger.error("Web3: WebSocket connection timed out"));
provider.on("close", () => logger.info("Web3: WebSocket connection closed"));
 
provider.on('connect', async () => {
     logger.info('Web3: WebSocket connected ');
     await setupEventListeners(); // // Your function to set up contract event listeners
});