Modern large scale distributed application architectures, wherein components are distributed across the internet on multiple hosts and multiple CPU cores, are often based on a messaging or event bus that allows the various distributed components to communicate asynchronously with each other. Typically the messaging bus is some form of messaging queue service such as AMQP or ZeroMQ. The message bus supports what is commonly referred to as a publish/subscribe methodology for information exchange.
While there are many advantages to a full featured message queuing service, one of the disadvantagesis the inability to manage performance at scale.
A message queuing service performs two distinct but complementary functions.
- The first is asynchronous transport of messages over the internet.
- The second is message queue management, that is, the identification, tracking, storage, and distribution of messages between publishers and subscribers via queues.
One of the advantages of a message queuing service for many applications is that the service hides behind an API, the complexities of queue management from the clients. The disadvantage is that at scale, where the volume of messages, the timing of messages, and the associated demands on memory, network, and cpu capacity become critical, the client has little ability to tune the service for performance. Often MQ services become bottlenecks for the distributed application. The more complicated MQ services, like AMQP, tend to be unreliable under load.
Separating the function of network transport of asynchrounous event from the function of message queue management allows independant tuning at scale of each function.
Most if not all of the MQ services are based on TCP/IP for transport. TCP/IP adds significant latency to the network communications and is therefore not well suited for the asynchronous nature of distibuted event driven application communications. This is primarily due to the way TCP/IP handles connection setup and teardown as well as failed connections in order to support streams. Fundamentally TCP/IP is optomized for sending large contiguous data streams not many small aynchronous events or messages. While not a problem for small scale systems, the differences in the associated traffic characteristics can become problematic at scale.
Because UDP/IP has lower latency and is connectionless, it is much better suited to many small asynchronous messages and scales better. The drawback of bare UDP/IP is that it is not reliable. What is needed, therefore, is a tuned transport protocol that adds reliability to UDP/IP without sacrificing latency and scalability. A transactioned protocol, is much more appropriate for providing reliablity to asynchronous event transport than a streaming protocol.
Morover, because most MQ services are based on TCP/IP they tend to also use HTTP and therefore TLS/SSL for secure communications. While using HTTP provides easy integration with web based systems, it can become problematic for high performant systems Furthermore, TLS is also problematic as a security system both from performance and vulnerabilty aspects.
Elliptic Curve Cryptography, on the other hand, provides increases in security with lower performance requirements relative to over other approaches. LibSodium provides an open source Elliptic Curve Cryptographic library with support for both authentication and encryption. The CurveCP protocol is based on LibSodium and provides a handshake protocol for bootstrapping secure network exchanges of information.
Finally, one of the best ways to manage and fine tune processor resources (cpu, memory, network) in distributed concurrent event driven applications is to use something called micro-threads. A microthread is typically an in-language feature that allows logical concurrency with no more overhead than a function call. Micro threading uses cooperative multi-tasking instead of threads and/or processes and avoids many of the complexities of resource contention, context switching, and interprocess communications while providing much higher total performance.
Because all the cooperative micro-threads run in one process, a simple micro-threaded application is limited to one CPU core. To enable full utilization of all CPU cores, the application needs to be able to run at least one process per CPU core. This requires same host inter-process communications. But unlike the conventional approach to multi-processing where there is of one process per logical concurrent function, a micro-threaded multi-process application has instead one micro-thread per logical concurrent function and the total number of micro-threads is distributed amoungst a minimal number of processes, no more than the number of cpu cores. This optimizes the use of the cpu power while minimizes the overhead of process context switching.
An example of a framework that uses this type of micro-threaded but multi-process architecture is Erlang. Indeed, the success of the Erlang model provided support for the viability of the RAET approach. Indeed, one might ask, why not use Erlang? Unfortunately, the Erlang ecosystem is somewhat limited in comparison to Python's and the language itself uses what one might describe as a very unfortunate syntax. One of the design objectives behine RAET was to leverage existing Python expertise and the richness of the Python ecosystem but still be able to develop distributed applications using a micro-threaded multi-process architectural model. The goal was to combine the best of both worlds.
RAET is designed to provide secure reliable scalable asynchronous message/event transport over the internet in a micro-threaded multi-process application framework that uses UDP for interhost communication and LibSodium for authentication, encryption and the CurveCP handshake for secure bootstrap.
The queue management and micro-threaded application support is provided by Ioflo. RAET is a complementary project to Ioflo in that RAET enables multiple Ioflo applications to work together over a network as part of a distributed application.
The primary use case and motivating problem that resulted in the development of RAET was the need to enable SaltStack to scale better. SaltStack is a remote execution and configuration management platform written in Python. SaltStack uses ZeroMQ (0MQ) as its message bus or message queuing service. ZeroMQ is based on TCP/IP so suffers from the aforementioned latency and non-asynchronicity issues of TCP/IP based architectures. Moreover because ZeroMQ integrates queue management and transport in a monolithic way with special "sockets", tuning the performance of the queuing independent of the transport at scale becomes problematic. Tracing down bugs can also be problematic.
Current raet is provided as a PyPi package. To install on a unix based system use pip.
$ pip install raet
on OS X
$ sudo pip install raet
Currently RAET supports two types of communication.
- Host to host communication over UDP/IP sockets
- Same host interprocess communication over Unix Domain (UXD) Sockets
The architecture of a RAET based application is shown in the figure below:
##Naming Metaphor for Components
The following naming metaphor is designed to consistent but not conflicting with Ioflo
- The UDP channel is a “Road"
- The members of a Road are “Estates” (as in real estate lots that front the road)
- Each Estate has a unique UDP Host Port address “ha” , a unique string “name” and unique numerical ID “eid".
- One Estate on the Road is the “Main” Estate
- The Main Estate is responsible for allowing other estates to join the Road via the Join (Key Exchange) and Allow (CurveCP) transactions
- The Main Estate is also responsible for routing messages between other Estates
- Within each Estate may be a “Lane”. This is a UXD channel.
- The members of a Lane are “Yards” (as in subdivision plots within the Estate)
- Each Yard on a Lane has a unique UXD file name host address “ha’, and a unique string “name”. There is also a numerical Yard ID “yid" that the class uses to generate yard names but it is not an attribute of the Yard instance.
- The Lane name is also used with the Yard Name to form a unique Filename that is the ha of the UXD
- One Yard on a Lane is the “Main” Yard
- The Main Yard is responsible for forming the Lane and permitting other Yards to be on the lane. There is yet no formal process for this. Currently there is a flag that will drop packets from any Yard that is not already in the list of Yards maintained by the Main yard. Also file permissions can be used to prevent spurious Yards from communicating with the Main Yard.
- The Main Yard is responsible for routing messages between other yards on the Lane
- Each Estate UDP interface is run via a RoadStack (UDP sockets) which is run within the context of an IoFlo House (so think of the House that runs the UDP Stack as the Manor House of the Estate)
- Each Yard UXD interface is run via a LaneStack (Unix domain socket) which is run within the context of an IoFlo House (so think of Houses that run UXD stacks as accessory Houses (Tents, Shacks) on the Estate)
- The "Manor" House is special in that it runs both the UDP stack for the Estate and also runs the UXD Stack for the Main Yard
- The House that runs the Main Estate UDP Stack can be thought of as Mayor’s House
- Within the context of a House is a Data Store. Shares in the Store are Addressed by the unique Share Name which is a dotted path
Given the Ioflo execution architecture described above, routing is performed as follows:
- In order to address a specific Estate, the Estate Name is required
- In order to address a specific Yard within an Estate, the Yard Name is required
- In order to address a specific Queue within a House, the Share Name is required
The UDP stack maps Estate Name to UDP HA and Estate ID The UXD stack maps Yard Name to UXD HA The Store of any IoFlo behavior maps Share Name to Share reference
Therefore Routing from: a source identified by a queue in a source Share, in a source Yard, in a source Estate to: a destination identified by a queue, in a destination Share, in a destination Yard, in a destination Estate
requires two Triples, one for Source and one for Destination
Source (Estate Name, Yard Name, Share Name)
Destination (Estate Name, Yard Name, Share Name)
If any element of the Triple is None or Empty then a Default is used.
Below is an example of a Message Body that has the Routing information it it.
estate = 'minion1'
stack0 = stacking.StackUxd(name='lord', lanename='cherry', yid=0)
stack1 = stacking.StackUxd(name='serf', lanename='cherry', yid=1)
yard = yarding.Yard( name=stack0.yard.name, prefix='cherry')
stack1.addRemoteYard(yard)
src = (estate, stack1.yard.name, None)
dst = (estate, stack0.yard.name, None)
route = odict(src=src, dst=dst)
msg = odict(route=route, stuff="Serf to my lord. Feed me!")
stack1.transmit(msg=msg)
timer = Timer(duration=0.5)
timer.restart()
while not timer.expired:
stack0.serviceAll()
stack1.serviceAll()
lord Received Message
{
'route':
{
'src': ['minion1', 'yard1', None],
'dst': ['minion1', 'yard0', None]
},
'stuff': 'Serf to my lord. Feed me!'
}The UDP Raet protocol is based on a coding metaphor naming convention, that is, of estates attached to a road. The core objects are provided in the following package: raet.road
Manor Estate 4505 Other Estates 4510
The data used to initialize a packet is an ordered dict with several fields most of the fields are shared with the header data format below so only the unique fields are shown here.
sh: source host ip address (ipv4)
sp: source ip port
dh: destination host ip address (ipv4)
dp: destination host ip port
The .data in the packet header is an ordered dict which is used to either create a packet to transmit or holds the field from a received packet. What fields are included in a header is dependent on the header kind.
There are three header encoding formats currently supported.
-
RAET Native. This is an minimized ascii test format that optimizes the tradeoff between easy readability and size. This is the default.
-
JSON. This is the most verbose format but has the advantage of compatibility.
-
Binary. This is not yet implemented. Once the protocol reaches a more mature state and its not likely that there will be any header changes (or very infrequent) then a binary format that minimizes size will be provided.
When the head kind is json = 0, then certain optimizations are used to minimize the header length.
- The header field keys are two bytes long
- If a header field value is the default then the field is not included
- Lengths are encoded as hex strings
- The flags are encoded as a double char hex string in field 'fg'
ri: raet id Default 'RAET'
vn: Version (Version) Default 0
pk: Packet Kind (PcktKind)
pl: Packet Length (PcktLen)
hk: Header kind (HeadKind) Default 0
hl: Header length (HeadLen) Default 0
se: Source Estate ID (SEID)
de: Destination Estate ID (DEID)
cf: Correspondent Flag (CrdtFlag) Default 0
bf: BroadCast Flag (BcstFlag) Default 0
si: Session ID (SID) Default 0
ti: Transaction ID (TID) Default 0
tk: Transaction Kind (TrnsKind)
dt: Datetime Stamp (Datetime) Default 0
oi: Order index (OrdrIndx) Default 0
wf: Waiting Ack Flag (WaitFlag) Default 0
Next segment or ordered packet is waiting for ack to this packet
ml: Message Length (MsgLen) Default 0
Length of message only (unsegmented)
sn: Segment Number (SgmtNum) Default 0
sc: Segment Count (SgmtCnt) Default 1
sf: Segment Flag (SgmtFlag) Default 0
This packet is part of a segmented message
af: All Flag (AllFlag) Default 0
Resend all segments not just one
bk: Body kind (BodyKind) Default 0
ck: Coat kind (CoatKind) Default 0
fk: Footer kind (FootKind) Default 0
fl: Footer length (FootLen) Default 0
fg: flags packed (Flags) Default '00' hs
2 char Hex string with bits (0, 0, af, sf, 0, wf, bf, cf)
Zeros are TBD flags
The Body .data is a Mapping that is serialized using either JSON or MSGPACK
Each packet has 4 parts some of which may be empty. These are:
- Head
- Body
- Coat
- Tail
The Head is manditory and provides the header fields that are needed to process the packet.
The Tail provides the authentication signature that is used to verify the source of the packet and that its contents have not been tampered with.
The Body is the contents of the packet. Some packets such as Acks and Nacks don't need a body. The Body is a serialized Python dictionary typically and ordered dictionary so that parsing and debugging has a consistent view of the ordering of the fields in the body.
The Coat is the encrypted version of the body. The encryption type is CurveCP based. If the Coat is provided then the Body is encapsulated within the Coat Part.
Header is the ASCII Safe JSON encoding of a Python ordered dictionary. Header termination is an empty line given by double pair of carriage-return linefeed characters.
/r/n/r/n 10 13 10 13 ADAD 1010 1101 1010 1101
Carriage-return and newline characters cannot appear in a JSON encoded string unless they are escaped with backslash, so the 4 byte combination is illegal in valid JSON that does not have multi-byte unicode characters so it makes it a uniquely identifiable header termination.
These means the header must be ascii safe so no multibyte utf-8 strings are allowed in the header.
The header consists of newline delimited lines. Each header line consists of a two character field identifier followed by a space followed by the value of the field as ascii hex encoded binary followed by newline. The Header end is indicated by a blank line, that is, a double newline character. Example
The header consists of defined set of fixed length fields
Session is important for security. Want one session opened and then multiple transactions within session.
Session ID SID sid si
OSI Layers
7: Application: Format: Data (Stack to Application interface buffering etc) 6: Presentation: Format: Data (Encrypt-Decrypt convert to machine independent format) 5: Session: Format: Data (Interhost communications. Authentication. Groups) 4: Transport: Format: Segments (Reliable delivery of Message, Transactions, Segmentation, Error checking) 3: Network: Format: Packets/Datagrams (Addressing Routing) 2: Link: Format: Frames ( Reliable per frame communications connection, Media access controller ) 1: Physical: Bits (Transciever communication connection not reliable)
-
Link is hidden from Raet
-
Network is IP host address and UDP Port
-
Transport is Raet transaction and packet authentication vis tail signature that provide reliable transport.
-
Session is session id key exchange for signing. Grouping is Road
-
Presentation is Encrypt-Decrypt Body and Serialize-Deserialize Body
-
Application is Body data dictionary
Packet signing could technically be in either the Transport or Session layers.
RAET UXD Messages are limited in size to the same maximum (pre segmented) RAET UDP message size (about 16 Mb)
UXD Messages have the following Format Header followed by serialized message body dict currently only JSON has been implemented.
-
JSON Header: “RAET\njson\n\n” Followed by a jsonified message body dict
-
msgpack Header: “RAET\npack\n\n” Followed by a msgpackified message body dict