Skip to content

Stream stacking occurs when H2 processes HTTP2 RST_STREAM frames. As a result, the memory and CPU usage are high. #2877

Closed
hyperium/h2
#668
@qinyushun

Description

@qinyushun

Version
hyper-0.13.7
h2-0.2.4

Platform
wsl

Description

Summary:

​ Stream stacking occurs when Hyper processes HTTP2 RST_STREAM frames. As a result, the memory and CPU usage are high.

Step:

  1. Send an HEADERS frame to open the stream, followed by an RST_STREAM frame to request cancellation of the stream

    def reset_flood(h2_conn):
        for i in range(1, 20000):
            if i % 2 == 0:
                continue
            headers(h2_conn, i)
            rst_stream(h2_conn, i)
    
  2. Create multiple threads for sending.

    if __name__ == '__main__':
        for i in range(0, 400):
            try:
                _thread.start_new_thread(send, ("Thread-" + str(i),))
            except:
                print("Error: Can not start thread")
        while 1:
            pass
    

Result:

The CPU usage of the Hyper server keeps increasing. As a result, the VM memory is used up.
image

Vulnerability analysis:

When receiving a HEADERS frame, the h2 stores the corresponding stream content in the slab and sets a frame index to the ids. When receiving an RST_STREAM frame, h2 sets the stream to Closed and resets the related statistics. However, the stream memory is released only when stream.is_released() is true . When the RST_STREAM frame is received, the release is not triggered immediately. As a result, the size of the slab increases continuously.

Test procedure:

Add the slab_len(),ids_len() method for Store to return the length of all flows and active flows.
image

Add the preceding two stream lengths to the recv_reset() method.
image

After the test, when the HEADERS frame is repeatedly sent to open a stream or the RST_STREAM frame is sent to cancel the stream, the length of the ids does not exceed the value of max_recv, but the SLAB increases .The stream in the Closed state in the SLAB is released only after all frames on the connection are sent.
image
image

The max_concurrent_streams configuration can limit max_recv_streams, but it appears that in this scenario, the size of Slab is much larger than the max_concurrent_streams value and stacks up.

I think it is necessary to limit the size of the Slab or ensure that streams in the Slab can be released quickly after the RST_STREAM frame is received to prevent such attacks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-http2Area: HTTP/2 specific.C-bugCategory: bug. Something is wrong. This is bad!

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions