-
Notifications
You must be signed in to change notification settings - Fork 892
Flow and Congestion Control
Aeron uses flow control to keep publishers from sending too fast, ensuring that subscribers do not get overrun. To this end, if a publisher attempts to send data that is too far ahead of where subscribers are consuming data, that publisher's Publication.offer or Publication.tryClaim operation will not be allowed to succeed, thus providing back-pressure. This is a temporary condition and the publisher should attempt the offer again.
From a user's perspective, flow control is based on the subscriber's rate of consumption and the publisher's rate of publication. However, internally, flow control is more complex. In this section, we will try to discuss the intricacies of flow and congestion control/avoidance.
Flow and congestion control in Aeron are operational on a per Stream basis. The operation of one stream SHOULD NOT affect the operation of another.
For background, please see Transport Protocol Specification for basic Aeron terminology. In addition, there is some terminology applied for flow and congestion control.
- Consumption Point: The byte offset within a stream that has been delivered to a consuming entity, such as a driver publication or client application Subscription. Delivery to the application DOES NOT imply successful or complete processing by the application.
- Status Message (SM): A special Aeron frame sent from subscriptions back to publication to inform publication of consumption state and reception window.
- Receiver Window: The window of viable data buffer advertised by a Receiver in Status Messages.
- Flow Control Strategy: The strategy that a driver publication employs in terms of how it treats SMs from multiple subscriptions.
- Publication Term Window Length: The amount of window, or buffer, the client application publication(s) has/have to exceed the driver publication. In other words, how much additional buffering space the client application can be ahead of the driver when sending data.
Low latency applications typically should not leave data in buffers for very long. Hence, the design of Aeron's flow control is to serve as a means to buffer bursts of activity in a safe manner for very high-throughput rates and to provide back-pressure when that buffering is at, or near, exhaustion. To this end, the buffering in use by Aeron is much larger than is typical for some transports that assume file transfer like semantics. However, this is configurable.
There are three points of flow control in an Aeron stream.
-
From client application Publication to driver based Sender: An application is allowed to succeed a call to
offeras long as it will not exceed the consumption point of the driver publication plus the Publication Term Window Length. The Publication Term Window Length is set by the media driver and can be controlled via the property,aeron.publication.term.window.length. When not set, a default value of 1/2 the term buffer length is used. The resulting window will be a minimum of any configured value and 1/2 of the term buffer length. - From Sender to Receiver: A Sender is allowed to immediately send data frames if doing so will not exceed the Consumption Term Offset plus Receiver Window set by the Flow Control Strategy from SM information from the subscriptions.
-
From driver based Receiver to client application Subscription: A Receiver is allowed to receive data and buffer it if the data lies within the last sent SM Consumption Term Offset plus the Receiver Window. The Receiver Window defaults to 128KB and may be controlled by the
aeron.rcv.initial.window.lengthproperty, or by the congestion control strategy. The resulting window will be a minimum of any configured value and 1/2 of the term buffer length.
From an end-to-end perspective, a publisher in a client application may be ahead of a subscriber by
Publication Term Window Length + Receiver Window
Multicast operation subsumes all of unicast operation. In addition, the handling of multiple subscriptions by driver publications is added. These are decoupled via a Flow Control Strategy.
Flow Control Strategy is concerned with how multiple driver receivers feedback, via SMs, is treated to consolidate a single driver publication view of the receivers. By default, Aeron uses a maximum (max) flow control strategy, i.e. the highest known consumption point from all receivers is used. A consequence of this default strategy is that if a slow receiver falls behind, it will eventually stop processing data, let the image go unavailable, then start again at the latest point in the stream.
Custom flow control strategies are supported for both unicast and multicast operation.
The flow control strategy used by the driver Sender is set via the system properties, aeron.unicast.FlowControl.supplier and aeron.multicast.FlowControl.supplier for unicast and multicast/multi-destination channels respectively.
The suppliers are designed so that it is possible to assign different strategies based on channel and/or stream Id. However, convenient suppliers are provided for common strategies that can be applied to all instances.
-
MinMulticastFlowControlSupplier: Track flow control to minimum (min) of all receivers. Slowest receiver sets the pace. -
MaxMulticastFlowControlSupplier: Track flow control to maximum (max) of all receivers. Fastest receiver sets the pace. -
TaggedMulticastFlowControlSupplier: Track flow control to minimum of tagged (tagged) receivers. Slowest tagged receiver sets the pace. -
DefaultMulticastFlowControlSupplier: Used when theaeron.multicast.FlowControl.supplieris not set. First looks into thefcparameter of the channel and if one is set will use its value to create aFlowControlstrategy instance. To that end it recognizes the following valuesmax,minandtagged. If thefcis not defined it returns aFlowControlinstance based on theaeron.multicast.flow.control.strategyproperty value. -
DefaultUnicastFlowControlSupplier: Used when theaeron.unicast.FlowControl.supplieris not set. Does not support anyfcparameter values on the channel and uses theaeron.unicast.flow.control.strategyproperty to configureFlowControlstrategy.
Beyond the suppliers, it is possible to customise a given FlowControl strategy as befits the application semantics desired by implementing the FlowControl interface. FlowControl strategy can be set via aeron.unicast.flow.control.strategy and aeron.multicast.flow.control.strategy properties respectively.
For more information, see the relevant classes and/or interfaces documentation.
Flow control strategies also have the ability to influence the connectivity of the Publication. The min and tagged strategies include support for this (max does not). It is possible to specify a minimum group size for the strategies and until that number of receivers are connected the Publication will indicate that it is not connected. This is orthogonal to the requirement of at least one Subscription to indicate connectivity, therefore the default value for the group minimum size is 0.
Note: If the ssc=true (Spies Simulate Connection) property is set for a publication the publication will be connected regardless of receiver group size if a spy subscription is connected.
SMs are generated by driver subscriptions and sent periodically back to driver publications. They include the following information:
- Term ID: The term ID for the highest consumed byte in the stream.
- Consumption Term Offset: The byte offset within the term of the highest consumed byte in the stream.
- Receiver Window: The advertised window to the driver publication of how much data the driver subscription is willing to receive.
SMs are generated periodically via the following rules:
- Generate an SM if the Term ID increases (i.e. rotation of Term buffers)
- Generate an SM if the difference between the highest seen term offset and the last sent SM Completed Term Offset is greater than 25% of the Receiver Window.
- Generate an SM if an SM not sent in more than a certain configurable timeout
A media driver that has multiple Subscriptions for the same driver subscription will take the minimum of the consumption points for the Subscriptions for use as the Consumption Term Offset and Term ID in the SM when sent.
Flow control is concerned with avoiding a publisher from overrunning a subscriber. Congestion control tries to slow a publication down so that it does not overrun the network.
Congestion control is not mandatory for Aeron implementations and may or may not be used depending on use case.
Congestion control is enforced on a per stream basis and is totally controlled by the driver Receiver as it is the one sending the Receiver Window to the driver publication.
Congestion Control in Aeron is accomplished by manipulating the Receiver Window. To mimic TCP operation, the receiver window may be adjusted as a normal TCP sender would manipulate its own congestion window. Unlike TCP, however, Aeron flow control can deal with window lengths varying and changing. Specifically, when a window shrinks without ACK'ing data.
Aeron is designed to work with low-latency and high-throughput systems in controlled environments where lower latency is traded off against congestion control for burst situations. In such systems, it is unlikely that congestion control is desired or advisable. And if so, it is probably not TCP-friendly congestion control. As such, other manipulation techniques of the Receiver Window probably make more sense than strict TCP fairness.
The driver can be configured to provide various congestion control strategies via the property aeron.CongestionControl.supplier. A congestion control strategy is assigned per Image and can be highly customised for specific channel, stream Id, session Id, term length, etc.
The default congestion control supplier uses a static window, StaticWindowCongestionControl, which effectively disables congestion control.
The following are the provided congestion control suppliers and strategies.
-
StaticWindowCongestionControl: A static window that effectively disables congestion control. -
CubicCongestionControlandCubicCongestionControlSupplier: A CUBIC implementation of receiver window modification with and without TCP Mode support.
Beyond the suppliers, it is possible to customise a given CongestionControl strategy as befits the application semantics by implementing the CongestionControl interface.
For more information, see the relevant classes and/or interfaces documentation.
For more information on concerns with congestion control and TCP, please see the following.
- TCP Reno
- TCP-friend Rate Control
- Multipath TCP Congestion Control
- BIC and CUBIC
- BBR: Congestion-based Congestion Control
IPC operation does not use SMs. Instead the media driver periodically updates the limit that a publication can send based on the state of the Subscriptions. First the minimum of the Subscription consumption points is determined then a window is added and that limit updated for the Publications. This window is controlled by a single property, aeron.ipc.publication.term.window.length. This acts as the window that the Publication can directly use before it is back pressured. When not set, a default value of 1/2 the term buffer length is used. The resulting window will be a minimum of any configured value and 1/2 of the term buffer length.
If more than one subscription matches on the same driver then each can be considered a tether or not. If subscriptions are a tether then they tether the flow control locally, i.e. subscription position's factor in the computation of the flow control window. If a subscription is not a tether and it is in the bottom 1/4th of the window, when another consumer is at the top of the window, for the aeron.untethered.window.limit.timeout then subscription in the bottom 1/4th will be notified that the image is unavailable for them. The subscription then remains a tether on flow control for the same timeout to allow the client to notice the unavailable image, after which the window can move on without that subscription factoring in flow control. At this point the subscription will rest for the period defined by aeron.untethered.resting.timeout after which is can rejoin the stream when it gets notified the image is available once more. If the subscriber does not wish to rest then they can close the subscription and add a new one.
The default for if subscriptions tether or not is controlled by the aeron.tether.subscriptions boolean system property which can be overridden for each subscription with the tether URI channel param, e.g. aeron:ipc?tether=false.
Tethering on IPC or spy channels have the window as defined by aeron.ipc.publication.term.window.length. Tether is also available on UDP subscriptions where the window is the current window as advertised for Status Messages which begins at aeron.rcv.initial.window.length, and aeron.publication.term.window.length for spying on network publications.
A subscription which spies, e.g. aeron-spy:aeron:udp?endpoint=<multicast.address:port>, on an outgoing local UDP publication is not involved in receiver based flow control for min, max, or tagged. It participates in flow control in a similar fashion to IPC and can use tether URI param.