Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Path is not app limited if more data is available to transmit, with limits #1367

Merged
merged 14 commits into from
Jun 23, 2022

Conversation

WesleyRosenblum
Copy link
Contributor

Description of changes:

This change re-introduces the changes from #1326, with additional limits on the maximum size the congestion window can grow to.

From the original description:

When a path is considered "application-limited", meaning the sending rate is constrained by the rate at which the application is providing data to transmit and not constrained by the congestion controller, the CUBIC congestion controller does not grow the congestion window. In the existing implementation, the determination of "application-limited" does not consider whether the application has additional data to transmit and only stopped sending due to the pacer. This results in the possibility of not growing the congestion window when acknowledgements are received after sending is temporarily blocked by the pacer.

This change makes a determination on whether the path is application limited external to the congestion controller. If there is no additional data waiting to be sent and the congestion controller is not blocking further transmission, the path is considered application limited.

The original change would allow the congestion window to grow significantly, even if sending was blocked by flow control. This happened because if the pacer interrupts sending at a moment when sending is momentarily not blocked by flow control (after more flow control credit is received, for example), we will not be considered app-limited, and any acknowledgements received will increase the window.

This change introduces a bytes_in_flight_hi value to the Cubic congestion controller that tracks the highest value of bytes_in_flight seen since the last congestion event. The congestion window is not allowed to grow beyond bytes_in_flight_hi multiplied by a multiplier (2.0 while in Slow Start and 1.5 in Congestion Avoidance).

Testing:

Unit tests, tested on two EC2 hosts:

Before:

./target/debug/s2n-quic-qns perf client --ip 127.0.0.1 --port 4433 --send 10000000000 --local-ip 0.0.0.0 --stats
1.17Mbps	0bps	103.94KB	77.52KB	0	44	292.014ms	226.374ms	204.694264ms	0
2.68Mbps	0bps	103.95KB	77.44KB	0	94	232.98ms	225.348ms	204.654852ms	0
2.36Mbps	0bps	103.96KB	74.49KB	0	80	249.433ms	226.233ms	204.629646ms	0
2.71Mbps	0bps	104.02KB	74.38KB	0	83	20.824ms	203.713ms	204.443565ms	0
2.42Mbps	0bps	104.13KB	74.28KB	0	77	247.882ms	226.1ms	204.774676ms	0
2.57Mbps	0bps	104.26KB	74.54KB	0	95	229.469ms	203.444ms	204.565716ms	0
2.45Mbps	0bps	104.36KB	74.38KB	0	75	229.164ms	203.473ms	202.178425ms	0
2.56Mbps	0bps	104.60KB	74.37KB	0	76	21.921ms	226.533ms	204.539865ms	0
2.54Mbps	0bps	105.12KB	74.28KB	0	76	233.75ms	226.405ms	204.976426ms	0
2.60Mbps	0bps	105.60KB	74.45KB	0	85	19.712ms	226.158ms	205.03833ms	0
2.52Mbps	0bps	106.45KB	74.33KB	0	83	231.009ms	224.645ms	204.64122ms	0
2.47Mbps	0bps	107.38KB	74.32KB	0	81	21.323ms	226.263ms	204.975986ms	0
2.63Mbps	0bps	109.03KB	74.44KB	0	95	240.741ms	226.219ms	206.01122ms	0
2.48Mbps	0bps	110.48KB	74.29KB	0	89	19.392ms	226.234ms	206.231644ms	0
2.65Mbps	0bps	112.57KB	74.29KB	0	102	227.52ms	226.169ms	205.171675ms	0
2.26Mbps	0bps	114.19KB	74.27KB	0	82	138.988ms	226.266ms	207.129585ms	0
2.81Mbps	0bps	116.72KB	77.53KB	0	104	257.697ms	226.313ms	207.40504ms	0
2.49Mbps	0bps	118.52KB	74.33KB	0	87	19.647ms	226.17ms	207.367767ms	0
2.65Mbps	0bps	121.06KB	74.33KB	0	100	256.035ms	226.315ms	207.452165ms	0
2.58Mbps	0bps	123.51KB	77.53KB	0	92	3.897ms	226.304ms	207.405915ms	0
2.50Mbps	0bps	126.16KB	74.43KB	0	98	257.603ms	226.279ms	207.226452ms	0
2.58Mbps	0bps	128.63KB	74.37KB	0	93	12.325ms	226.238ms	207.449185ms	0
2.54Mbps	0bps	131.62KB	74.33KB	0	94	256.284ms	226.228ms	207.470695ms	0
2.81Mbps	0bps	135.59KB	77.49KB	0	102	11.181ms	226.153ms	207.398556ms	0
2.31Mbps	0bps	138.94KB	74.29KB	0	88	255.996ms	226.312ms	207.443956ms	0
2.86Mbps	0bps	143.36KB	74.34KB	0	104	252.14ms	226.091ms	207.454587ms	0
2.27Mbps	0bps	146.59KB	77.49KB	0	85	122.001ms	226.222ms	207.416884ms	0
2.86Mbps	0bps	151.84KB	74.34KB	0	103	253.327ms	226.227ms	207.274836ms	0
2.63Mbps	0bps	155.82KB	74.34KB	0	95	252.594ms	226.246ms	207.446975ms	0
2.51Mbps	0bps	159.91KB	74.33KB	0	97	253.474ms	226.303ms	207.30402ms	0
2.82Mbps	0bps	165.14KB	77.53KB	0	107	253.488ms	227.071ms	207.495796ms	0
2.49Mbps	0bps	168.89KB	77.27KB	0	87	13.514ms	226.244ms	207.462634ms	0
2.65Mbps	0bps	174.39KB	74.33KB	0	99	249.078ms	226.291ms	207.481692ms	0
2.82Mbps	0bps	178.87KB	77.59KB	0	105	236.195ms	226.293ms	207.415056ms	0
2.28Mbps	0bps	182.26KB	77.33KB	0	80	108.271ms	226.208ms	207.038605ms	0
2.84Mbps	0bps	186.27KB	77.27KB	0	98	252.891ms	226.216ms	207.359738ms	0
2.86Mbps	0bps	192.09KB	77.27KB	0	86	8.711ms	226.224ms	207.407899ms	0
2.23Mbps	0bps	195.04KB	77.54KB	0	75	77.14ms	216.648ms	204.797187ms	0
2.85Mbps	0bps	204.15KB	74.33KB	0	85	216.256ms	203.462ms	202.330791ms	0
2.83Mbps	0bps	210.29KB	74.38KB	0	90	215.025ms	203.451ms	202.005385ms	0

After:

./target/debug/s2n-quic-qns perf client --ip 127.0.0.1 --port 4433 --send 10000000000 --local-ip 0.0.0.0 --stats
1.19Mbps	0bps	91.78KB	77.42KB	0	41	288.099ms	203.465ms	202.081315ms	0
2.69Mbps	0bps	121.22KB	77.33KB	0	108	205.755ms	219.14ms	203.603494ms	0
2.65Mbps	0bps	121.22KB	77.56KB	0	109	4.617ms	206.116ms	202.375377ms	0
2.58Mbps	0bps	121.22KB	77.51KB	0	102	246.964ms	226.277ms	205.942144ms	0
2.50Mbps	0bps	121.22KB	74.47KB	0	93	247.464ms	226.326ms	206.723698ms	0
2.63Mbps	0bps	121.22KB	74.35KB	0	102	259.073ms	226.321ms	207.550543ms	0
2.51Mbps	0bps	121.22KB	74.39KB	0	97	253.39ms	226.238ms	207.874545ms	0
2.62Mbps	0bps	121.22KB	74.46KB	0	102	240.077ms	226.298ms	207.641503ms	0
2.51Mbps	0bps	121.22KB	74.39KB	0	99	238.335ms	226.237ms	207.115486ms	0
2.63Mbps	0bps	121.22KB	74.35KB	0	101	258.571ms	226.241ms	207.443034ms	0
2.51Mbps	0bps	121.22KB	74.39KB	0	97	251.243ms	226.317ms	207.893574ms	0
2.61Mbps	0bps	121.22KB	74.46KB	0	101	259.654ms	227.094ms	207.604703ms	0
2.49Mbps	0bps	121.22KB	74.38KB	0	100	202.002ms	226.186ms	207.592277ms	0
2.62Mbps	0bps	121.22KB	74.38KB	0	101	206.773ms	226.292ms	207.903753ms	0
2.68Mbps	0bps	121.22KB	74.45KB	0	100	19.102ms	226.301ms	207.64542ms	0
2.41Mbps	0bps	121.22KB	74.39KB	0	101	183.2ms	226.257ms	207.520342ms	0
2.50Mbps	0bps	121.22KB	74.38KB	0	97	185.947ms	226.234ms	207.382263ms	0
2.81Mbps	0bps	121.22KB	74.38KB	0	105	14.703ms	227.091ms	207.975536ms	0
2.49Mbps	0bps	121.22KB	74.45KB	0	96	15.619ms	227.088ms	207.585839ms	0
2.60Mbps	0bps	121.22KB	74.38KB	0	102	15.654ms	226.177ms	207.521813ms	0
2.50Mbps	0bps	121.22KB	74.39KB	0	97	17.65ms	226.308ms	207.910829ms	0
2.63Mbps	0bps	121.22KB	74.45KB	0	103	250.876ms	226.246ms	207.722119ms	0
2.58Mbps	0bps	121.22KB	74.39KB	0	98	3.286ms	226.265ms	207.152308ms	0
2.52Mbps	0bps	121.22KB	74.38KB	0	102	250.075ms	226.238ms	207.349695ms	0
2.47Mbps	0bps	121.22KB	74.39KB	0	94	15.632ms	227.087ms	207.95339ms	0
2.61Mbps	0bps	121.22KB	74.49KB	0	103	243.998ms	226.243ms	207.618632ms	0
2.59Mbps	0bps	121.22KB	74.38KB	0	97	13.502ms	226.121ms	207.015379ms	0
2.53Mbps	0bps	121.22KB	74.38KB	0	99	241.571ms	226.225ms	207.453032ms	0
2.62Mbps	0bps	121.22KB	74.38KB	0	99	250.493ms	226.232ms	206.949163ms	0
2.50Mbps	0bps	121.22KB	77.50KB	0	97	259.087ms	226.232ms	207.427246ms	0
2.71Mbps	0bps	121.22KB	74.38KB	0	99	15.565ms	226.251ms	207.495323ms	0
2.42Mbps	0bps	121.22KB	74.34KB	0	94	259.472ms	226.306ms	207.497506ms	0
2.81Mbps	0bps	121.22KB	77.50KB	0	100	14.635ms	226.313ms	207.500593ms	0
2.31Mbps	0bps	121.22KB	74.34KB	0	90	259.878ms	226.298ms	207.355353ms	0
2.83Mbps	0bps	121.22KB	74.33KB	0	104	3.812ms	226.303ms	207.473918ms	0
2.31Mbps	0bps	121.22KB	74.33KB	0	93	259.449ms	226.256ms	207.500991ms	0
2.78Mbps	0bps	121.22KB	77.59KB	0	112	12.523ms	226.308ms	207.492882ms	0
2.31Mbps	0bps	121.22KB	74.34KB	0	86	244.032ms	226.232ms	207.459834ms	0
2.82Mbps	0bps	121.22KB	74.38KB	0	105	13.691ms	226.228ms	207.301916ms	0
2.28Mbps	0bps	121.22KB	77.55KB	0	96	117.45ms	226.225ms	207.486817ms	0
2.85Mbps	0bps	121.22KB	74.28KB	0	108	259.892ms	226.354ms	207.404127ms	0
2.28Mbps	0bps	121.22KB	74.33KB	0	86	122.009ms	226.235ms	207.472151ms	0
2.86Mbps	0bps	121.22KB	74.33KB	0	100	258.303ms	227.222ms	207.440549ms	0

Is this a refactor change? If so, how have you proved that the intended behavior hasn't changed? -->

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

self.delivered_bytes > app_limited_bytes
})
{
// Clear app-limited field if bubble is ACKed and gone
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is bubble a term in BBR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It comes from the related delivery rate estimation draft RFC:

An application-limited phase starts when the sending application
requests to send new data, or the connection's retransmission
mechanisms decide to retransmit data, and the connection meets all of
the following conditions ...
If these conditions are all met then the sender has run out of data
to feed the network. This would effectively create a "bubble" of
idle time in the data pipeline.

This comment specifically is from the pseudocode: https://datatracker.ietf.org/doc/html/draft-cheng-iccrg-delivery-rate-estimation#section-3.3

camshaft
camshaft previously approved these changes Jun 20, 2022
…mited

# Conflicts:
#	quic/s2n-quic-transport/src/space/application.rs
@WesleyRosenblum WesleyRosenblum merged commit d080d64 into main Jun 23, 2022
@WesleyRosenblum WesleyRosenblum deleted the WesleyRosenblum/applimited branch June 23, 2022 23:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants