This is a proof-of-concept implementation of Distributed PubSub using Distributed Process Group, GenServer, Consistent Hash Ring, Manifold, and Channels. Scalable PubSub layer is an essential component of message delivery in chat applications and beyond.
Ring used to determine Topic's Node, subscribe to Topic's GenServer, and fan out messages to subscribers, subscriber PIDs are grouped by their remote node to reduce number of send/2 calls.
Direct message publishing helps mitigate the Bandwidth Bisection problem, which the default broadcast PubSub Adapter is more prone to, since number of messages grows quadratically with the number of nodes in the cluster when using broadcast.
Topic is roughly equivalent to a Guild mentioned in How Discord Scaled Elixir to 5,000,000 Concurrent Users.
NB Production ready system would involve dynamic topology management, replication, netsplit recovery, stricter delivery semantics, message queueing, overload protection and other things.
+--------------+
| Message Flow |
+--------------+
Client 1 --(Topic 1)--> WS Server 1 --> Topic Server 1 --------+
|
|
Client 2 <--(Topic 1)-- WS Server 2 <-- Topic 1 (GenServer) ----+
ProcessGroup
+------------------+
+------------+ Topic 1 | +--------------+ | GenServers
| Client 1 |---------| | WS Server 1 | | +-----------------+
+------------+ | +-------|------+ | | +-------------+ |
| | | | | Topic 1 |-----+
| +-------+------+ | | +-------------+ | |
| |Topic Server 1|----+ +-------------+ | |
| +--------------+ | | | Topic X | | |
| | | | | +-------------+ | |
| | | | +-----------------+ |
| +(Hash Ring) + | +-----------------+ |
| | | | | +-------------+ | |
| | | | | | Topic X | | |
| +--------------+ | | +-------------+ | |
| |Topic Server X|----+ +-------------+ | |
| +--------------+ | | | Topic X + 1 | | |
| | | +-------------+ | |
+------------+ Topic 1 | +--------------+ | +-----------------+ |
| Client 2 |+--------|-- WS Server 2 |+-------------------------|
+------------+ | +--------------+ |
+------------------+
# relevant for macos
brew install asdf wxwidgets
# install erlang and elixir
asdf plugin add elixir
asdf install elixir 1.17.1-otp-27
asdf plugin add erlang
asdf install erlang 27.0
# install dependencies
mix deps.get
Application can be started in multiple modes: ws (websocket) and ts (topic server).
# start websocket servers
DPS_PORT=4000 DPS_APP=ws iex --sname dps-a -S mix phx.server
DPS_PORT=4001 DPS_APP=ws iex --sname dps-b -S mix phx.server
# start topic servers
DPS_APP=ts iex --sname dps-ts-a -S mix
DPS_APP=ts iex --sname dps-ts-b -S mix
Ring is static for demonstration purpose, and can be adjusted in config/dev.exs given cluster changes.
# terminal 1
websocat "ws://127.0.0.1:4000/socket/websocket?vsn=2.0.0"
["1", "1", "topics:matrix", "phx_join", {}]
# terminal 2
websocat "ws://127.0.0.1:4001/socket/websocket?vsn=2.0.0"
["1", "1", "topics:matrix", "phx_join", {}]
["1", "1", "topics:matrix", "publish", ["event", { "message": "red pill or blue pill?"}]]
# => the following message should appear in both terminals
["1",null,"topics:matrix","event",{"message":"red pill or blue pill?"}]
# tests
mix test
# quality
mix quality
# debugging
:observer.start()