-
Notifications
You must be signed in to change notification settings - Fork 103
HTTP2 streams prioritization
Alexander Krizhanovsky edited this page Oct 3, 2024
·
4 revisions
HTTP/2 is here for quite a while. Cloudflare introduced server-side HTTP/2 streams prioritization back in 2019. That time browsers didn't do a perfect work on assigning priorities to HTTP/2 streams. Now it's 2023 and we decided to explore how the modern popular browsers assign stream priorities and how modern HTTP servers work with the priorities.
Please also reference our recent talk on the Netdev 0x18 conference Scheduling HTTP streams.
Before developing of our own HTTP streams scheduling logic, we analyzed how other open source HTTP servers implement this. We hoped to borrow some existing code, but ended up with our own implementation.
There is RFC 9218, which describes streams prioritization for both the HTTP/2 and HTTP/3, but Chrome and Firefox support the RFC only for HTTP/3 and uses the RFC 7540 scheduling for HTTP/2.
Nginx has it's own HTTP/2 implementation. Ngnix builds stream dependency tree as it's described in the RFC 7540. The main function for adding frame into the send queue is (see nginx/src/http/v2/ngx_http_v2.h)
static ngx_inline void
ngx_http_v2_queue_frame(ngx_http_v2_connection_t *h2c,
ngx_http_v2_out_frame_t *frame)
{
ngx_http_v2_out_frame_t **out;
for (out = &h2c->last_out; *out; out = &(*out)->next) {
if ((*out)->blocked || (*out)->stream == NULL) {
break;
}
if ((*out)->stream->node->rank < frame->stream->node->rank
|| ((*out)->stream->node->rank == frame->stream->node->rank
&& (*out)->stream->node->rel_weight
>= frame->stream->node->rel_weight))
{
break;
}
}
frame->next = *out;
*out = frame;
}A frame is inserted into the sending list according to the rank (the level in the priority tree)
of the stream and weight. But it does not correspond to the RFC: a server should not send data for
a stream which depends on other streams.
Also the algorithm can lead to O(n) complexity (linear scan) if each next frame has higher priority
than the previous one.
Prioritization model in the H2O is much more sophisticated and we considered to reuse it in Tempesta FW (hopefully the MIT license allows to borrow code in a GPL project). There is a tree of streams (each level of this tree contains array with 64 entries). Stream is inserted into this tree according to its dependency and its weight. Also there is a bit array to speed up looking up the most priority stream.
The streams scheduler is implemented in lib/http2/scheduler.c
and uses an O(1) approach described as an Array of Queue in the
presentation.
To estimate the H2O streams scheduler accuracy we wrote a small program evaluating how does it schedule 256 streams, each with a different weight (note that the relatively sophisticated websites, like slack.com, easily employ 100-200 streams). We compared the H2o scheduler with Weighted Fair Queue (WFQ) (page 7), which fairness has been proven.
#!/usr/bin/env python3
# Comparison of H2O and an ideal WFQ schedulers.
# Reference: https://www.mew.org/~kazu/material/2015-http2-priority2.pdf
DEBUG = 0
TOTAL_FRAMES = 10000
CONSTANT = 65536 * 63
# Precomputed values of 63*65536 (the constant) / weight,
# where weight ranges from 1 to 256.
H2O_OFFSET_TABLE = [
4128768, 2064384, 1376256, 1032192, 825754, 688128, 589824,
516096, 458752, 412877, 375343, 344064, 317598, 294912,
275251, 258048, 242869, 229376, 217304, 206438, 196608,
187671, 179512, 172032, 165151, 158799, 152917, 147456,
142371, 137626, 133186, 129024, 125114, 121434, 117965,
114688, 111588, 108652, 105866, 103219, 100702, 98304,
96018, 93836, 91750, 89756, 87846, 86016, 84261, 82575,
80956, 79399, 77901, 76459, 75069, 73728, 72435, 71186,
69979, 68813, 67685, 66593, 65536, 64512, 63520, 62557,
61623, 60717, 59837, 58982, 58152, 57344, 56558, 55794,
55050, 54326, 53620, 52933, 52263, 51610, 50972, 50351,
49744, 49152, 48574, 48009, 47457, 46918, 46391, 45875,
45371, 44878, 44395, 43923, 43461, 43008, 42565, 42130,
41705, 41288, 40879, 40478, 40085, 39700, 39322, 38951,
38587, 38229, 37879, 37534, 37196, 36864, 36538, 36217,
35902, 35593, 35289, 34990, 34696, 34406, 34122, 33842,
33567, 33297, 33030, 32768, 32510, 32256, 32006, 31760,
31517, 31279, 31043, 30812, 30583, 30359, 30137, 29919,
29703, 29491, 29282, 29076, 28873, 28672, 28474, 28279,
28087, 27897, 27710, 27525, 27343, 27163, 26985, 26810,
26637, 26466, 26298, 26131, 25967, 25805, 25645, 25486,
25330, 25175, 25023, 24872, 24723, 24576, 24431, 24287,
24145, 24004, 23866, 23729, 23593, 23459, 23326, 23195,
23066, 22938, 22811, 22686, 22562, 22439, 22318, 22198,
22079, 21962, 21845, 21730, 21617, 21504, 21393, 21282,
21173, 21065, 20958, 20852, 20748, 20644, 20541, 20439,
20339, 20239, 20140, 20043, 19946, 19850, 19755, 19661,
19568, 19475, 19384, 19293, 19204, 19115, 19027, 18939,
18853, 18767, 18682, 18598, 18515, 18432, 18350, 18269,
18188, 18109, 18030, 17951, 17873, 17796, 17720, 17644,
17569, 17495, 17421, 17348, 17275, 17203, 17132, 17061,
16991, 16921, 16852, 16784, 16716, 16648, 16581, 16515,
16449, 16384, 16319, 16255, 16191, 16128]
class Stream:
def __init__(self, id, weight, deficit, sent):
self.id = id
self.weight = weight
self.deficit = deficit
self.sent = sent
def __lt__(self, other):
return self.deficit < other.deficit
h2o_anchors = {}
wfq = []
def h2o_queue_set(stream_id, weight, deficit = 0, sent = 0):
""" queue_set() from h2o/lib/http2/scheduler.c """
offset = H2O_OFFSET_TABLE[weight - 1] + deficit
# 65536 is constant2 from the slides
deficit = offset % 65536
offset = int(offset / 65536)
s = Stream(stream_id, weight, deficit, sent)
if h2o_anchors.get(offset):
h2o_anchors[offset].append(s)
else:
h2o_anchors[offset] = [ s ]
def h2o_queue_pop():
for o in range(256):
if (not o in h2o_anchors) or not h2o_anchors[o]:
continue
return h2o_anchors[o].pop(0)
return None
def wfq_queue_set(stream_id, weight, deficit = 0, sent = 0):
""" WFQ for comparison with H2O """
deficit = CONSTANT / weight
if wfq:
deficit += wfq[0].deficit
s = Stream(stream_id, weight, deficit, sent)
wfq.append(s)
wfq.sort()
def wfq_queue_pop():
return wfq.pop(0)
if __name__ == "__main__":
# Populate the H2O scheduler with streams of all priorities.
for w in range(256):
h2o_queue_set(w, w + 1)
wfq_queue_set(w, w + 1)
# Execute the H2O and ideal schedulers 1000 times.
if DEBUG:
print("H2O STREAMS (WEIGHT DEFICIT) \tWFQ STREAMS (WEIGHT DEFICIT)")
for i in range(TOTAL_FRAMES):
hs = h2o_queue_pop()
h2o_queue_set(hs.id, hs.weight, hs.deficit, hs.sent + 1)
ws = wfq_queue_pop()
wfq_queue_set(ws.id, ws.weight, ws.deficit, ws.sent + 1)
if DEBUG:
print(f"{hs.id} ({hs.weight} {hs.deficit}) \t{ws.id} ({ws.weight} {ws.deficit})")
counters = {}
skew = {}
for o in h2o_anchors:
hl = h2o_anchors[o]
for hs in hl:
for ws in wfq:
if ws.id == hs.id:
diff = abs(ws.sent - hs.sent)
if not diff in counters:
counters[diff] = 0
skew[diff] = 0
counters[diff] += 1
skew[diff] += ws.sent
print(f"{counters[0]} streams sent the same amounts")
for d in sorted(counters):
if not d:
continue
print(f'{counters[d]} streams sent {d} less frames ({round(counters[d] * d * 100 / skew[d])}% inaccuracy)')For 1,000 frames (TOTAL_FRAMES in the code) or only about 4 frames in each of the stream
$ ./h2_stream_prio.py # 1,000 frames
122 streams sent the same amounts
126 streams sent 1 less frames (23% inaccuracy)
8 streams sent 2 less frames (76% inaccuracy)bit less than 50% of streams were scheduled ideally and about 3% of streams were 76% imbalanced. For 10,000 frames we see worse statistic:
$ ./h2_stream_prio.py # 10,000 frames
2 streams sent the same amounts
34 streams sent 1 less frames (3% inaccuracy)
116 streams sent 2 less frames (4% inaccuracy)
52 streams sent 3 less frames (7% inaccuracy)
10 streams sent 4 less frames (17% inaccuracy)
3 streams sent 5 less frames (100% inaccuracy)
4 streams sent 6 less frames (69% inaccuracy)
3 streams sent 7 less frames (100% inaccuracy)
6 streams sent 8 less frames (83% inaccuracy)
5 streams sent 9 less frames (94% inaccuracy)
5 streams sent 10 less frames (76% inaccuracy)
11 streams sent 11 less frames (82% inaccuracy)
5 streams sent 12 less frames (81% inaccuracy)Almost no ideal streams and about 80% of the streams were scheduled with inaccuracy less than 10%. There were 5% of streams with 100% inaccuracy.
We decided this behavior unacceptable and moved with WFQ.
The nghttp2 library implements WFQ in a proper way and the most of HTTP servers (e.g. Envoy or Apache HTTPD) and clients (e.g. curl) use the library for HTTP/2.
Also it seems this is the only implementation supporting RFC 9218 for HTTP/2.
Unfortunately, nghttp2 uses (see for example nghttp2_stream_reschedule() in
nghttp2_stream.c)
plain sorted arrays with ordered insertions (see nghttp2_pq_push() and bubble_up()
in lib/nghttp2_pq.c),
i.e. there are memory reallocations and multiple array traversals with copying and
swapping elements.
While the library provides fair WFQ, it doesn't fit our performance requirements.
Still in 2023 we observed that browsers (e.g. Firefox) may assign the same priority to a bunch of resources. Traditional HTTP/2 (as well as HTTP/3) prioritization algorithms send such resources in a round-robin fashion, so all of them finish deliver at the most late point in time. This is the worst approach in general and only progressive JPEGs benefit from this.
Tempesta FW, just like Cloudflare sends resources (except JPEGs) with the same priority sequentially, one by one, reducing the average load time.
As we saw for other HTTP server examples, it's quite an issue to make the streams scheduler not only fast, but also fair. We chose to use WFQ, like nghttp2, to get fairness, but use a better data structure for the priority queues used by the algorithm. There are a lot of data structures which can be used for this (list, different types of heaps and different types of trees). We analyzed some of them (e.g. Fibonacci heap, RB-tree, insertion sorted array etc) and found that the HAproxy's ebtree provides the best performance (at least x2 faster than the closest in performance Fibonacci heap) on small data (about 100 to 1000 streams in a queue) to pick a minimum item and reinsert it.
At the moment of writing this article the pull request is still in progress, but probably is already merged on the time of reading it :)
We explored how Chromium, Firefox and Opera load a WordPress website. About 50% of websites are developed on WordPress and we used our website as an example of a WordPress application.
- As can be seen below at the Chromium and Opera browsers use exclusive flag for every new created stream. So stream priority tree is a list of streams for these browsers and the weight of streams doesn't matter.
- Firefox build complex priority tree, without exclusive flag and use weights to prioritize streams. CSS and JavaScript code have greater priority than images.
- Quite often responses are sent out of priority, since responses for lower priority streams may arrive earlier from a back-end server.
- Chromium and Opera use priority frames to change stream dependency on the fly.
* Create new stream (id 1 weight 256 exclusive 1 depends from stream with id 0 connection ffff8be8e3869538) (The HTML file for the page itself, h_len 244 b_len 25865)
* finished to send data for stream with id 1
* Create new stream (id 3 weight 256 exclusive 1 depends from stream with id 0 connection ffff8be8e3869538) (css, h_len 79 b_len 2436)
* Create new stream (id 5 weight 256 exclusive 1 depends from stream with id 3 connection ffff8be8e3869538) (css, h_len 33 b_len 8550)
* finished to send data for stream with id 5
* finished to send data for stream with id 3
* Create new stream (id 7 weight 183 exclusive 1 depends from stream with id 0 connection ffff8be8e3869538) (javascript code, h_len 41 b_len 21634)
* Create new stream (id 9 weight 183 exclusive 1 depends from stream with id 7 connection ffff8be8e3869538) (css, h_len 56 b_len 4354)
* Create new stream (id 11 weight 147 exclusive 1 depends from stream with id 9 connection ffff8be8e3869538) (picture *.png format, h_len 41 b_len 6040)
* Create new stream (id 13 weight 147 exclusive 1 depends from stream with id 11 connection ffff8be8e3869538) (picture *.png format, h_len 234 b_len 31288)
* Create new stream (id 15 weight 147 exclusive 1 depends from stream with id 13 connection ffff8be8e3869538) (picture *.png format, h_len 70 b_len 3547)
* Create new stream (id 17 weight 147 exclusive 1 depends from stream with id 15 connection ffff8be8e3869538) (picture *.png format, h_len 32 b_len 2505)
* Create new stream (id 19 weight 147 exclusive 1 depends from stream with id 17 connection ffff8be8e3869538) (picture *.png format, h_len 62 b_len 1776)
* Create new stream (id 21 weight 147 exclusive 1 depends from stream with id 19 connection ffff8be8e3869538) (picture *.png format, h_len 70 b_len 2023)
* Create new stream (id 23 weight 147 exclusive 1 depends from stream with id 21 connection ffff8be8e3869538) (picture *.png format, h_len 32 b_len 2892)
* Create new stream (id 25 weight 147 exclusive 1 depends from stream with id 23 connection ffff8be8e3869538) (picture *.png format, h_len 33 b_len 4451)
* Create new stream (id 27 weight 147 exclusive 1 depends from stream with id 25 connection ffff8be8e3869538) (picture *.png format, h_len 70 b_len 3469)
* Create new stream (id 29 weight 147 exclusive 1 depends from stream with id 27 connection ffff8be8e3869538) (picture *.png format, h_len 37 b_len 14710)
* Create new stream (id 31 weight 147 exclusive 1 depends from stream with id 29 connection ffff8be8e3869538) (picture *.png format, h_len 70 b_len 3962)
* Create new stream (id 33 weight 147 exclusive 1 depends from stream with id 31 connection ffff8be8e3869538) (picture *.png format, h_len 33 b_len 2682)
* Create new stream (id 35 weight 147 exclusive 1 depends from stream with id 33 connection ffff8be8e3869538) (picture *.png format, h_len 37 b_len 12739)
* Create new stream (id 37 weight 147 exclusive 1 depends from stream with id 35 connection ffff8be8e3869538) (picture *.png format, h_len 33 b_len 4462)
* Create new stream (id 39 weight 147 exclusive 1 depends from stream with id 37 connection ffff8be8e3869538) (picture *.png format, h_len 33 b_len 7344)
* Create new stream (id 41 weight 147 exclusive 1 depends from stream with id 39 connection ffff8be8e3869538) (picture *.svg format, h_len 46 b_len 3861)
* Create new stream (id 43 weight 147 exclusive 1 depends from stream with id 41 connection ffff8be8e3869538) (picture *.svg format, h_len 32 b_len 1057)
* Create new stream (id 45 weight 147 exclusive 1 depends from stream with id 43 connection ffff8be8e3869538) (picture *.svg format, h_len 70 b_len 1087)
* Create new stream (id 47 weight 147 exclusive 1 depends from stream with id 45 connection ffff8be8e3869538) (picture *.svg format, h_len 32 b_len 2865)
* Create new stream (id 49 weight 147 exclusive 1 depends from stream with id 47 connection ffff8be8e3869538) (picture *.svg format, h_len 32 b_len 2143)
* Create new stream (id 51 weight 147 exclusive 1 depends from stream with id 49 connection ffff8be8e3869538) (picture *.png format, h_len 84 b_len 376863)
* Create new stream (id 53 weight 147 exclusive 1 depends from stream with id 51 connection ffff8be8e3869538) (picture *.JFIF format, h_len 78 b_len 30950)
* Create new stream (id 55 weight 147 exclusive 1 depends from stream with id 53 connection ffff8be8e3869538) (picture *.png format, h_len 35 b_len 4604)
* Create new stream (id 57 weight 147 exclusive 1 depends from stream with id 55 connection ffff8be8e3869538) (picture *.JFIF format, h_len 49 b_len 11986)
* Create new stream (id 59 weight 147 exclusive 1 depends from stream with id 57 connection ffff8be8e3869538) (picture *.png format, h_len 32 b_len 1574)
* finished to send data for stream with id 15
* finished to send data for stream with id 21
* finished to send data for stream with id 19
* finished to send data for stream with id 23
* finished to send data for stream with id 11
* finished to send data for stream with id 9
* finished to send data for stream with id 17
* finished to send data for stream with id 25
* finished to send data for stream with id 27
* finished to send data for stream with id 31
* finished to send data for stream with id 37
* finished to send data for stream with id 39
* finished to send data for stream with id 41
* finished to send data for stream with id 43
* finished to send data for stream with id 45
* finished to send data for stream with id 47
* finished to send data for stream with id 49
* finished to send data for stream with id 59
* finished to send data for stream with id 55
* finished to send data for stream with id 57
* finished to send data for stream with id 53
* finished to send data for stream with id 7
* finished to send data for stream with id 13
* finished to send data for stream with id 29
* finished to send data for stream with id 33
* finished to send data for stream with id 35
* finished to send data for stream with id 51
* Create new stream (id 61 weight 220 exclusive 1 depends from stream with id 0 connection ffff8be8e3869538) (javascript code, h_len 46 b_len 191669)
* Create new stream (id 63 weight 147 exclusive 1 depends from stream with id 61 connection ffff8be8e3869538 (picture *.png format, h_len 72 b_len 24475)
* finished to send data for stream with id 61
* finished to send data for stream with id 63
* Create new stream (id 65 weight 147 exclusive 1 depends from stream with id 0 connection ffff8be8e3869538) (picture *.png format, h_len 40 b_len 2683)
* Create new stream (id 67 weight 220 exclusive 1 depends from stream with id 65 connection ffff8be8e3869538) (picture *.png format, h_len 69 b_len 384)
* finished to send data for stream with id 67
* finished to send data for stream with id 65
* Create new stream (id 3 weight 0 exclusive 0 depends from stream with id 0 connection ffff893da78054e0)
* Create new stream (id 5 weight 0 exclusive 0 depends from stream with id 0 connection ffff893da78054e0)
* Create new stream (id 7 weight 0 exclusive 0 depends from stream with id 0 connection ffff893da78054e0)
* Create new stream (id 9 weight 0 exclusive 0 depends from stream with id 0 connection ffff893da78054e0)
* Create new stream (id 11 weight 0 exclusive 0 depends from stream with id 0 connection ffff893da78054e0)
* Create new stream (id 13 weight 0 exclusive 0 depends from stream with id 0 connection ffff893da78054e0)
* Create new stream (id 15 weight 42 exclusive 0 depends from stream with id 13 connection ffff893da78054e0) (The HTML file for the page itself, h_len 244 b_len 25865)
* finished to send data for stream with id 15
* Create new stream (id 17 weight 42 exclusive 0 depends from stream with id 3 connection ffff893da78054e0) (css, h_len 33 b_len 8550)
* Create new stream (id 19 weight 22 exclusive 0 depends from stream with id 3 connection ffff893da78054e0) (css, h_len 79 b_len 2436)
* Create new stream (id 21 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 33 b_len 6040)
* Create new stream (id 23 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 75 b_len 31288)
* Create new stream (id 25 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 70 b_len 3547)
* Create new stream (id 27 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 32 b_len 2505)
* Create new stream (id 29 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 70 b_len 1776)
* Create new stream (id 31 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 80 b_len 2023)
* Create new stream (id 33 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 32 b_len 2892)
* Create new stream (id 35 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 40 b_len 4451)
* Create new stream (id 37 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 70 b_len 3469)
* Create new stream (id 39 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 40 b_len 14710)
* Create new stream (id 41 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 70 b_len 3962)
* Create new stream (id 43 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 32 b_len 2682)
* Create new stream (id 45 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 39 b_len 12739)
* Create new stream (id 47 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 33 b_len 4462)
* Create new stream (id 49 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 33 b_len 7344)
* Create new stream (id 51 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.svg format, h_len 32 b_len 3861)
* Create new stream (id 53 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.svg format, h_len 46 b_len 1057)
* Create new stream (id 55 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.svg format, h_len 32 b_len 1087)
* Create new stream (id 57 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.svg format, h_len 32 b_len 2865)
* Create new stream (id 59 weight 12 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.svg format, h_len 70 b_len 2143)
* Create new stream (id 61 weight 22 exclusive 0 depends from stream with id 5 connection ffff893da78054e0) (javascript code, h_len 34 b_len 21634)
* Create new stream (id 63 weight 22 exclusive 0 depends from stream with id 5 connection ffff893da78054e0) (css, h_len 56 b_len 4354)
* finished to send data for stream with id 31
* finished to send data for stream with id 29
* finished to send data for stream with id 25
* finished to send data for stream with id 53
* finished to send data for stream with id 59
* finished to send data for stream with id 55
* finished to send data for stream with id 37
* finished to send data for stream with id 43
* finished to send data for stream with id 33
* finished to send data for stream with id 19
* finished to send data for stream with id 17
* finished to send data for stream with id 63
* finished to send data for stream with id 57
* finished to send data for stream with id 27
* finished to send data for stream with id 47
* finished to send data for stream with id 41
* finished to send data for stream with id 51
* finished to send data for stream with id 21
* finished to send data for stream with id 61
* finished to send data for stream with id 45
* finished to send data for stream with id 39
* finished to send data for stream with id 35
* finished to send data for stream with id 49
* finished to send data for stream with id 23
* Create new stream (id 65 weight 13 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 84 b_len 376863)
* Create new stream (id 67 weight 13 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.JFIF format, h_len 72 b_len 30950)
* Create new stream (id 69 weight 13 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 196 b_len 4604)
* Create new stream (id 71 weight 13 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.JFIF format, h_len 45 b_len 11986)
* Create new stream (id 73 weight 13 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 47 b_len 1574)
* finished to send data for stream with id 73
* Create new stream (id 75 weight 22 exclusive 0 depends from stream with id 5 connection ffff893da77f3fa8) (javascript code, h_len 74 b_len 191669)
* finished to send data for stream with id 69
* finished to send data for stream with id 71
* finished to send data for stream with id 67
* finished to send data for stream with id 75
* Create new stream (id 77 weight 23 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 70 b_len 2683)
* Create new stream (id 79 weight 23 exclusive 0 depends from stream with id 11 connection ffff893da78054e0) (picture *.png format, h_len 12 b_len 2683)
* finished to send data for stream with id 65
* finished to send data for stream with id 77
* finished to send data for stream with id 79
* Create new stream (id 81 weight 13 exclusive 0 depends from stream with id 11 connection ffff893da77f3fa8) (picture *.png format, h_len 69 b_len 384)
* finished to send data for stream with id 81
* Create new stream (id 1 weight 256 exclusive 1 depends from stream with id 0 connection ffff8be8eb760000) (The HTML file for the page itself, h_len 244 b_len 25865)
* finished to send data for stream with id 1
* Create new stream (id 3 weight 256 exclusive 1 depends from stream with id 0 connection ffff8be8eb760000) (css, h_len 33 b_len 8550)
* Create new stream (id 5 weight 256 exclusive 1 depends from stream with id 3 connection ffff8be8eb760000) (css, h_len 79 b_len 2436)
* finished to send data for stream with id 5
* finished to send data for stream with id 3
* Create new stream (id 7 weight 183 exclusive 1 depends from stream with id 0 connection ffff8be8eb760000) (picture *.JFIF format, h_len 40 b_len 11986)
* Create new stream (id 9 weight 183 exclusive 1 depends from stream with id 7 connection ffff8be8eb760000) (css, h_len 56 b_len 4354)
* Create new stream (id 11 weight 147 exclusive 1 depends from stream with id 9 connection ffff8be8eb760000) (picture *.png format, h_len 41 b_len 6040)
* Create new stream (id 13 weight 147 exclusive 1 depends from stream with id 11 connection ffff8be8eb760000) (picture *.png format, h_len 78 b_len 31288)
* Create new stream (id 15 weight 147 exclusive 1 depends from stream with id 13 connection ffff8be8eb760000) (picture *.png format, h_len 70 b_len 3547)
* Create new stream (id 17 weight 147 exclusive 1 depends from stream with id 15 connection ffff8be8eb760000) (picture *.png format, h_len 62 b_len 2505)
* Create new stream (id 19 weight 147 exclusive 1 depends from stream with id 17 connection ffff8be8eb760000) (picture *.png format, h_len 32 b_len 1776)
* Create new stream (id 21 weight 147 exclusive 1 depends from stream with id 19 connection ffff8be8eb760000) (picture *.png format, h_len 80 b_len 2023)
* Create new stream (id 23 weight 147 exclusive 1 depends from stream with id 21 connection ffff8be8eb760000) (picture *.png format, h_len 32 b_len 2892)
* Create new stream (id 25 weight 147 exclusive 1 depends from stream with id 23 connection ffff8be8eb760000) (picture *.png format, h_len 33 b_len 4451)
* Create new stream (id 27 weight 147 exclusive 1 depends from stream with id 25 connection ffff8be8eb760000) (picture *.png format, h_len 70 b_len 3469)
* Create new stream (id 29 weight 147 exclusive 1 depends from stream with id 27 connection ffff8be8eb760000) (picture *.png format, h_len 37 b_len 14710)
* Create new stream (id 31 weight 147 exclusive 1 depends from stream with id 29 connection ffff8be8eb760000) (picture *.png format, h_len 74 b_len 3962)
* Create new stream (id 33 weight 147 exclusive 1 depends from stream with id 31 connection ffff8be8eb760000) (picture *.png format, h_len 70 b_len 2682)
* Create new stream (id 35 weight 147 exclusive 1 depends from stream with id 33 connection ffff8be8eb760000) (picture *.png format, h_len 161 b_len 12739)
* Create new stream (id 37 weight 147 exclusive 1 depends from stream with id 35 connection ffff8be8eb760000) (picture *.png format, h_len 33 b_len 4462)
* Create new stream (id 39 weight 147 exclusive 1 depends from stream with id 37 connection ffff8be8eb760000) (picture *.png format, h_len 33 b_len 7344)
* Create new stream (id 41 weight 147 exclusive 1 depends from stream with id 39 connection ffff8be8eb760000) (picture *.svg format, h_len 46 b_len 3861)
* Create new stream (id 43 weight 147 exclusive 1 depends from stream with id 41 connection ffff8be8eb760000) (picture *.svg format, h_len 32 b_len 1057)
* Create new stream (id 45 weight 147 exclusive 1 depends from stream with id 43 connection ffff8be8eb760000) (picture *.svg format, h_len 32 b_len 1087)
* Create new stream (id 47 weight 147 exclusive 1 depends from stream with id 45 connection ffff8be8eb760000) (picture *.svg format, h_len 32 b_len 2865)
* Create new stream (id 49 weight 147 exclusive 1 depends from stream with id 47 connection ffff8be8eb760000) (picture *.svg format, h_len 32 b_len 2143)
* Create new stream (id 51 weight 147 exclusive 1 depends from stream with id 49 connection ffff8be8eb760000) (picture *.png format, h_len 75 b_len 376863)
* Create new stream (id 53 weight 147 exclusive 1 depends from stream with id 51 connection ffff8be8eb760000) (picture *.JFIF format, h_len 72 b_len 30950)
* Create new stream (id 55 weight 147 exclusive 1 depends from stream with id 53 connection ffff8be8eb760000) (picture *.png format, h_len 35 b_len 4604)
* Create new stream (id 57 weight 147 exclusive 1 depends from stream with id 55 connection ffff8be8eb760000) (picture *.JFIF format, h_len 49 b_len 11986)
* Create new stream (id 59 weight 147 exclusive 1 depends from stream with id 57 connection ffff8be8eb760000) (picture *.png format, h_len 32 b_len 1574)
* finished to send data for stream with id 21
* finished to send data for stream with id 27
* finished to send data for stream with id 33
* finished to send data for stream with id 15
* finished to send data for stream with id 17
* finished to send data for stream with id 19
* finished to send data for stream with id 11
* finished to send data for stream with id 9
* finished to send data for stream with id 23
* finished to send data for stream with id 25
* finished to send data for stream with id 31
* finished to send data for stream with id 37
* finished to send data for stream with id 39
* finished to send data for stream with id 41
* finished to send data for stream with id 43
* finished to send data for stream with id 45
* finished to send data for stream with id 49
* finished to send data for stream with id 59
* finished to send data for stream with id 55
* finished to send data for stream with id 57
* finished to send data for stream with id 7
* finished to send data for stream with id 13
* finished to send data for stream with id 29
* finished to send data for stream with id 35
* finished to send data for stream with id 47
* finished to send data for stream with id 51
* finished to send data for stream with id 53
* Create new stream (id 61 weight 220 exclusive 1 depends from stream with id 0 connection ffff8be8eb760000) (javascript code, h_len 44 b_len 191669)
* finished to send data for stream with id 61
* Create new stream (id 63 weight 220 exclusive 1 depends from stream with id 0 connection ffff8be8eb760000 (picture *.png format, h_len 51 b_len 2683)
* finished to send data for stream with id 63
* Create new stream (id 65 weight 220 exclusive 1 depends from stream with id 0 connection ffff8be8eb760000 (h_len 35 b_len 0, text)
* finished to send data for stream with id 65