WebSocketConnection object possible memory leak

[johnbrady6]

Hi, I don't know if anyone else has noticed this problem but there seems to be an issue with WebSocketConnection object not being cleaned up after the client connection has closed and gc has ran. Using the "memwatch" and "heapdump" plugins for node I have discovered when connecting 100 clients and then disconnecting 100 it appears that there are about 70 odd "state:closed" WebSocketConnection objects still lingering about. This was found using the example ws server code. As you can imagine after writing a script to repeat these steps over a period of a time these objects build up quite a bit of memory and are never released. If anyone has any recommendations or any insight on this issue it would be greatly appreciated. Thanks.

[chrisabrams]

Do the objects end up disappearing after a second or two, or do the just kinda chill around with a permanent state:closed? I'm curious as I'm wanting to use this lib. but if there is a 70% chance of an object lingering that is not very promising.

[theturtle32]

It's likely that the garbage collection hasn't kicked in yet. A more thorough way to test would be to continually connect and disconnect while watching the memory consumption. You should see a sawtooth pattern on the graph as memory is allocated and then freed once in a while by the garbage collector. Just doing 100 and then sampling afterward will not yield accurate results.

Brian

Sent from my iPhone

On Dec 19, 2013, at 3:09 AM, johnbrady6 notifications@github.com wrote:

Hi, I don't know if anyone else has noticed this problem but there seems to be an issue with WebSocketConnection object not being cleaned up after the client connection has closed and gc has ran. Using the "memwatch" and "heapdump" plugins for node I have discovered when connecting 100 clients and then disconnecting 100 it appears that there are about 70 odd "state:closed" WebSocketConnection objects still lingering about. This was found using the example ws server code. As you can imagine after writing a script to repeat these steps over a period of a time these objects build up quite a bit of memory and are never released. If anyone has any recommendations or any insight on this issue it would be greatly appreciated. Thanks.

—
Reply to this email directly or view it on GitHub.

[chrisabrams]

@theturtle32 That's what I was assuming would happen 👍

[johnbrady6]

I believe that the heapdump plugin manually runs gc before creating a dump file. To be sure, I've also exposed the gc and run it frequently during dump retrievals. @chrisabrams We're currently running tests on it just now but there appears to be a few WebSocketConnection objects remaining with state:closed and they never appear to be cleaned up. I'll keep you informed on how our tests go as we're looking into whats possibly causing it. @theturtle32 I'm not sure if the built xor file is possibly causing it as when using the native js appears to clean up a few and the removal of the deprecated "webSocketVersion" properties seems to help also. We're also looking into native binds and listeners to see if its maybe a timing issue but nothing is for certain as of yet.

[johnbrady6]

As you can see from the screen shot the filename consists of the timestamp followed by the active connection count e.g. "1389614655881_5.heapsnapshot". We can expect to see 7 WebsocketConnection objects, 3 from the WS server, 3 from the WS clients and 1 for the prototype but instead there is 20 WebSocketConnection objects, 6 inaccesible through the code (all have state:closed). This test was ran using the following script:

process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";

var heapdump = require('heapdump');
var memwatch = require('memwatch');
var fs = require('fs');
var WebSocketServer = require('websocket').server;
var WebSocketClient = require('websocket').client
var https = require('https');

var connectionAmount = process.argv[2];
var activeCount = 0;
var deviceList = [];
var maxWaitBeforeFlakeyDisconnect = 120000;
var maxWaitBeforeFlakeyReconnect = 120000;

var config = { 
    key: fs.readFileSync( "privatekey.pem" ), 
    cert: fs.readFileSync( "certificate.pem" )  
};

var server = https.createServer( config );

server.listen(8080, function() {
    console.log((new Date()) + ' Server is listening on port 8080');
});

wsServer = new WebSocketServer({
    httpServer: server,
    autoAcceptConnections: false    
});

wsServer.on('request', function(request) {
    activeCount++;
    console.log("---activeCount---: "+activeCount);
    var connection = request.accept(null, request.origin);
    connection.myUniqueCounter = activeCount;
    console.log((new Date()) + ' Connection accepted.');
    connection.on('message', function(message) {
        if (message.type === 'utf8') {
            console.log('Received Message: ' + message.utf8Data);
            connection.sendUTF(message.utf8Data);
        }       
    });
    connection.on('close', function(reasonCode, description) {
        activeCount--;      
        console.log("---activeCount---: "+activeCount);             
    });
});

setInterval( function(){
    global.gc();
    heapdump.writeSnapshot( './heapdump/'+ new Date().getTime() + '_' + activeCount + '.heapsnapshot' );    
}, 10000 );
memwatch.on('leak', function(info) { console.log(info); });
connectDevices();


function connectDevices() {
    for( var i=0; i < connectionAmount; i++ ){
        connect( i );   
        flake( i );
    }
}

function connect( i ){          
    console.log( "--- Connecting: " + i );
    var client = new WebSocketClient(); 

    client.on('connectFailed', function(error) {
        console.log('Connect Error: ' + error.toString());
    });

    client.on('connect', function(connection) {
        client.connection = connection;
        console.log("Connected!");

        connection.on('error', function(error) {
            console.log("Connection Error: " + error.toString());
        });

        connection.on('close', function() {
            console.log('Connection Closed');
        });

        connection.on('message', function(message) {
            if ( message.type === 'utf8' ) {
                console.log("Received: '" + message.utf8Data + "'");
            }
        });     
    });
    client.connect("wss://localhost:8080");

    deviceList[i] = client; 
}

function disconnect( i ){
    console.log( "--- Disconnecting: " + i );
    deviceList[i].connection.close();   
}

function flake(i) {
    var id = i;
    console.log( "--- flaking " + +i );
    var timeBeforeDisconnect = Math.floor( Math.random() * maxWaitBeforeFlakeyDisconnect );
    console.log( "--- Flakey device " + i + " disconnecting in " + timeBeforeDisconnect + " milliseconds" );
    setTimeout( function() {
        disconnect(id);
        var timeBeforeReconnect = Math.floor( Math.random() * maxWaitBeforeFlakeyReconnect );
        console.log( "--- Flakey device " + id + " reconnecting in " + timeBeforeReconnect + " milliseconds" );
        setTimeout( function() {
            connect( id );
            flake(id);
        }, timeBeforeReconnect);
    }, timeBeforeDisconnect);
}

[chrisabrams]

@johnbrady6 Thanks for writing this up; I hope y'all are able to find what is causing these sockets to not be cleaned up :)

[johnbrady6]

Just a quick update. After finding this blog post http://www.tuicool.com/articles/yUR3q2 we began to look into slab buffers. After adding this line

require('tls').SLAB_BUFFER_SIZE = 100 * 1024 # 100Kb

it made a huge difference to our overall memory consumption as we have went from being able to handle from around 5000 concurrent WS connections to double that value per 2gb flavour servers. It doesn't really solve the leaking socket connection issue but thought it might be of some use to other WebSocket-Node library users.

[theturtle32]

@johnbrady6 I've tried the test code you posted, and found some flaws and bugs with it. I've modified it and split it into a separate client and server process, and found that there doesn't seem to be any leak in WebSocket-Node (at least not with the latest modifications and enhancements on master.) I'm going to go ahead and close this. Once I release 1.0.9, go ahead and check again and let me know if this issue persists for you. I'll be adding my test server/client into the /test directory shortly.