Upgrade to socket.io 1.0 / engine.io

[vojtajina]

Let's use low level https://github.com/LearnBoost/engine.io instead of socket.io

[vojtajina]

Based on discussion in https://github.com/vojtajina/testacular/issues/95

[dignifiedquire]

What is your take on communication? As engine.io doesn't provide the event based interface that is currently used.

[Raynos]

communication is trivial. You send an string (probably JSON encoded) that represents a message. A common example is

send(JSON.stringify({ type: "my event", key: "some value" }))

[dignifiedquire]

@Raynos Thanks but I wasn't talking about the low level communication. I think we need some level of abstraction for the communication. Otherwise we litter our code with switch/case or if/else statements and parsing of the messages.

I've come up with three options so far.

Callback based RPC

Using dnode we can communicate using remote function calls with callback style. For example with the following function definition on one side

var add = function(a,b, callback) {
  callback(a+b);
};

we can execute it on the other side like this

remote.add(1,1, function(result) {
  console.log(result);
  // prints 2
});

Promise based RPC

Using q-connection we can communicate using function calls that return promises instead of using callbacks. Simple example
Function definition

var add = function(a,b) {
  return a + b;
};

Function execution

remote
  .invoke('add', 1, 1)
  .then(function(result) {
    console.log(result);
    // prints 2
  });

Simple EventEmitter

We can write a small wrapper around the message event of engine.io so that we get something like the following
Sending a message:

var add = function (a,b) {
  return a + b;
};
connection.send('added', add(1,1));

Receiving a message

connection.on('added', function(result) {
  console.log(result);
  // prints 2
});

[Raynos]

// client
message("result", resultObj)

// server
dispatch(socket, {
    result: handleResult
})

function message(type, chunk) {
    socket.send(JSON.stringify({ type: type, chunk: chunk }))
}

function dispatch(socket, methods) {
    socket.on("message", function (data) {
        data = JSON.parse(data)
        methods[data.type](data.chunk)
    })
}

Which is effectively the "event emitter" abstraction just inlined in 6 lines of code.

If you want a higher level abstraction generally you want to deal with RPC, invoking remote methods (dnode). Or you want to deal with replicated state like scuttlebutt, crdt or leveldb replication.

State replication may be suited.

If you want tests to be run send state change of the tests to be run to the client which reacts and runs tests and updates the result states which you get as changes to the results on the server and can react to in the terminal GUI.

Probably best modeled as a set of scuttlebutts. One for tests to be run and then one for each browser.

[dignifiedquire]

State replication really sounds like it would fit the bill quite well. Thanks a lot for bringing scuttlebutt etc. to attention. I haven't heard of them before.
Do you have any experience on RPC vs State Replication vs EventEmitter in a real scenario? Especially performance and ease of implementation.

[Raynos]

@dignifiedquire early attempts at state replications using adhoc implementations in production were a disaster.

RPC (dnode) vs EventEmitter isn't really that different, really depends on what you want.

It should also be noted that an alternative to event emitter (which is just namespacing messages) is multiplexing streams (mux-demux) which basically namespacing.

Havn't had a chance to use scuttlebutt in production yet but I have far more trust that is stable.

[Iristyle]

I'm a bit out of the loop on the details here, but I know that I've seen info pop up on Smith.IO.

Its from the Cloud9 IDE guys, so I'd imagine they have some expertise in this area. It builds on engine.io and smith.... and they use it for communication within their virtual file system.

Have a look -- might be useful here.

@Raynos thanks for the scuttlebutt mention -- hadn't seen it. It's particularly interesting to me since we build our back-end on Riak (and we're building a small library to handle CRDT-like data structures)

[dignifiedquire]

@Iristyle Thanks I had heard of smith.io a while ago and the forgot it. Smith looks nice but the API of Smith.IO is somewhat incomplete from what I could tell so not sure on that front.

@ALL I've created a concept that uses multiple streams for communication. You can finde it here. I think this solution ist quite elegant and should perform well. I'm working on getting reconnect to work, which would allow for automatic reconnects.

@vojtajina It would be great to get a little tip about which direction to follow.

[Sate]

What the hell is this discussion even about? @Raynos' comments are spot on. This should just use raw websocket API with an abstraction wrapper. Either way socket.io has way too much overhead and memory leaking, get rid of it asap!

I'd recommend SockJS. It's used in production big time (rabbitmq, meteor, etc.) although it has no abstraction wrapper, I'm working on writing one soon, though someone else may beat me to it.

[Raynos]

@Sate what kind of abstraction wrapper would you want? both shoe and sockjs-stream already exist.

[Sate]

@Raynos I actually meant the event emitter code you posted earlier

// client
message("result", resultObj)

// server
dispatch(socket, {
    result: handleResult
})

function message(type, chunk) {
    socket.send(JSON.stringify({ type: type, chunk: chunk }))
}

function dispatch(socket, methods) {
    socket.on("message", function (data) {
        data = JSON.parse(data)
        methods[data.type](data.chunk)
    })
}

Basically just give socket.io API to sockjs/native WS. Would like to have on/emit in socketObject.prototype instead of dispatch/message functions.

[Raynos]

There's already a module that does that. emit-stream

[dignifiedquire]

@Sate I don't know what your beef with socket.io is but we've already decided not to use sock.js, see #95. As for the discussion here of course this simple solution would work but I'd rather move away from a simple event based system to something more elegant. Why put another layer on streams when we could much more cleanly just use streams. Also this simplistic approach doesn't provide any option for reconnect strategies. Sure we can invent the wheel again but that is exactly what we try not to do.

[Iristyle]

@dignifiedquire swoop in, crap on project, swoop out.. haha 😈

[dignifiedquire]

@Iristyle Yeah we've been trolled

[Iristyle]

I knew there was a better icon in that emoji list somewhere. Even with the autocomplete, there's so many to remember!

[arikon]

Is there any progress on this?

[Phoscur]

Seems to conflict badly when used with grunt and socket.io 1.0 is already present in the project.

[dignifiedquire]

Closing in favor of #1220