Serf Lua RPC Client

[subnetmarco]

Create a Serf client in Lua that follows Serf RPC protocol: https://www.serfdom.io/docs/agent/rpc.html

[Tieske]

some info on arbitrary TCP connections; https://groups.google.com/forum/#!topic/openresty-en/88E7-1CIowM

[Tieske]

Doing this involves setting up an outgoing tcp connection to the serf agent running on the same host.

Basics work; connecting to the agent and sending/receiving some simple commands. One tricky part is the MessagePack library, it might need some changes to support streaming messages.

An issue that is harder to solve (at least for a proper solution) is the synchronisation between worker processes. If implemented in OpenResty worker processes, each worker will have an open TCP connection, so each worker process will also get all events. So for example, 3 worker processes all receive an invalidation event, and they in turn all three invalidate the in memory cache. That would result in 3 expensive DB requests, instead of only one. That is undesirable.

Having only 1 worker setup the tcp connection to the serf agent seems dangerous to me as events might get lost if that worker process crashes.

The issue can be reduced to: how to reduce multiple instances of the same incoming event (one for each worker) to a single one.

Issues to solve;

• streaming support when unpacking messagepack data
• reducing multiple instances of the same event to a single one
• reloading config; when nginx reloads, detect it, and shutdown the tcp connections so the worker process can actually exit

[subnetmarco]

Having only 1 worker setup the tcp connection to the serf agent seems dangerous to me as events might get lost if that worker process crashes.

I think this can be fixed with locks the same way we handle reports and cluster keep-alive pings. A possible implementation will look like a basic leader-election across the workers:

• Every worker does listen on serf, but only one will execute actions.
• Every worker will try to get the lock for a specific key in the shared dict that determines who is going to be the worker that executes the action.
• Only one worker will be able to get the lock (like we do for reports or cluster keep-alive pings), and he will keep renewing it so that the other workers cannot acquire the same lock.
• If the worker crashes, the key will turn to an unlocked because it can't renew the lock anymore (it crashed).
• Another worker takes the lock (aka, becomes the leader), and keeps it. Events are not lost because they are effectively received by every worker, and only then they decide who is going to execute actions.

[Tieske]

closing this in favour of #835 (remove dependencies)