-
Notifications
You must be signed in to change notification settings - Fork 12.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle out of memory errors in io:Read::read_to_end() #117925
Conversation
(rustbot has picked a reviewer for you, use r? to override) |
☔ The latest upstream changes (presumably #118412) made this pull request unmergeable. Please resolve the merge conflicts. |
ad4f209
to
fdc51b6
Compare
This comment has been minimized.
This comment has been minimized.
fdc51b6
to
1e2d37b
Compare
@rustbot r? |
Perhaps |
@fintelia I've made a separate PR, because I think it may be bikesheddable which error kind it should return. |
We discussed this in the libs-api meeting and we are happy with this change. It doesn't need an FCP since it isn't an API breaking change. Some concerns on the implementation:
|
1e2d37b
to
6218f5e
Compare
@Amanieu I've added the impls for |
We could use Or we could document that the caller should presize the Vec if they want more control over the allocation. |
I also disagree about |
I was thinking of the specific case where the size of the reader is known. But you're right, for an empty vec it already does the right thing. The only case where it could cause an issue is when the Vec already has been allocated but the allocated space is close but insufficient, in that case it could double the capacity when we could have done an exact allocation instead. That's a fairly narrow scenario. |
c928767
to
1be2869
Compare
I've added There's possibility of optimizing growth based on the hint — when the hinted size is between |
This comment has been minimized.
This comment has been minimized.
1be2869
to
60f4628
Compare
@bors r+ |
☀️ Test successful - checks-actions |
Finished benchmarking commit (5c9c3c7): comparison URL. Overall result: no relevant changes - no action needed@rustbot label: -perf-regression Instruction countThis benchmark run did not return any relevant results for this metric. Max RSS (memory usage)ResultsThis is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
CyclesThis benchmark run did not return any relevant results for this metric. Binary sizeResultsThis is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.
Bootstrap: 661.382s -> 662.576s (0.18%) |
Reject infinitely-sized reads from io::Repeat These calls would always run out of memory. Related to rust-lang#117925
Rollup merge of rust-lang#119991 - kornelski:endless-read, r=the8472 Reject infinitely-sized reads from io::Repeat These calls would always run out of memory. Related to rust-lang#117925
Wait, was it intentional to merge this with some Prior to this PR they all used |
I've ended up making the change to The default behavior remains to use The
|
This case is irrelevant because
This adds a quirk to one specific implementation that I'm not sure is justified. As mentioned, In order words, the only case that changed from this PR is when the read size requires an allocation between |
Follow-up to rust-lang#117925
Make File::read_to_end less special Follow-up to rust-lang#117925
Make File::read_to_end less special Follow-up to rust-lang#117925
#116570 got stuck due to a procedural confusion. Retrying so that it can get FCP with the proper team now. cc @joshtriplett @BurntSushi
I'd like to propose handling of out-of-memory errors in the default implementation of
io::Read::read_to_end()
andfs::read()
. These methods create/grow aVec
with a size that is external to the program, and could be arbitrarily large.Due to being I/O methods, they can already fail in a variety of ways, in theory even including
ENOMEM
from the OS too, so another failure case should not surprise anyone.While this may not help much Linux with overcommit, it's useful for other platforms like WASM. Internals thread.
I've added documentation that makes it explicit that the OOM handling is a nice-to-have, and not a guarantee of the trait.
I haven't changed the implementation of
impl Read for &[u8]
andVecDeque
out of caution, because in these cases users could assumeread
can't fail.This code uses
try_reserve()
+extend_from_slice()
which is optimized since #117503.