Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(http): read more before triggering TooLargeError #401

Merged
merged 1 commit into from
Mar 28, 2015
Merged

Conversation

seanmonstar
Copy link
Member

This includes a custom BufReader, since the one in libstd doesn't allow
reading additional data into the buffer without consuming it. This is
required because some connections may send shorter packets, and so we
need to perform multiple reads. After each read, the contents of the
buffer are passed to httparse to see if have a valid message. If so, the
proper amount of bytes are consumed. The additional bytes are left in
the buffer since they are the beginning of the body.

The buffer in this BufReader also grows in size, compared to the libstd
which is sized once. This is because we start with a smaller buffer,
since the majority of messages will be able to include their head in a
packet or 2. Therefore, it's a wasteful performance hit to allocate the
maximum size for every message. However, some headers can be quite big,
and to allow for many of them to be set, we include a maximum size. Once
we've hit the maximum buffer size, and still haven't determined the end
of the headers, a HttpTooLargeError will be returned.

Closes #389

cc @reem @r-darwish

This includes a custom BufReader, since the one in libstd doesn't allow
reading additional data into the buffer without consuming it. This is
required because some connections may send shorter packets, and so we
need to perform multiple reads. After each read, the contents of the
buffer are passed to httparse to see if have a valid message. If so, the
proper amount of bytes are consumed. The additional bytes are left in
the buffer since they are the beginning of the body.

The buffer in this BufReader also grows in size, compared to the libstd
which is sized once. This is because we start with a smaller buffer,
since the majority of messages will be able to include their head in a
packet or 2. Therefore, it's a wasteful performance hit to allocate the
maximum size for every message. However, some headers can be quite big,
and to allow for many of them to be set, we include a maximum size. Once
we've hit the maximum buffer size, and still haven't determined the end
of the headers, a HttpTooLargeError will be returned.

Closes #389
seanmonstar added a commit that referenced this pull request Mar 28, 2015
fix(http): read more before triggering TooLargeError
@seanmonstar seanmonstar merged commit 6c31ea8 into master Mar 28, 2015
@seanmonstar seanmonstar deleted the packets branch March 28, 2015 18:37
inner: R
}

const INIT_BUFFER_SIZE: usize = 4096;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turning this down should be a nice perf win for short connections.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Hyper client fails if the header isn't fully received in the first read
2 participants