Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VecDeque implementation creates a slice pointing to possibly uninitialized memory #74189

Closed
ecstatic-morse opened this issue Jul 9, 2020 · 7 comments
Assignees
Labels
A-collections Area: `std::collection` C-bug Category: This is a bug. E-help-wanted Call for participation: Help is requested to fix this issue. T-libs Relevant to the library team, which will review and decide on the PR/issue.

Comments

@ecstatic-morse
Copy link
Contributor

ecstatic-morse commented Jul 9, 2020

VecDeque has an internal method called buffer_as_slice, which returns an &[T] containing the entire capacity of the VecDeque. This is undefined behavior if the VecDeque is not full, since some elements of the backing RawVec may be uninitialized. However, this invariant is not documented on buffer_as_slice and is not respected in practice. For example, VecDeque::iter calls buffer_as_slice unconditionally:

#[stable(feature = "rust1", since = "1.0.0")]
pub fn iter(&self) -> Iter<'_, T> {
Iter { tail: self.tail, head: self.head, ring: unsafe { self.buffer_as_slice() } }
}

This one seems so obvious that I'm wondering if I've overlooked something. cc @rust-lang/wg-unsafe-code-guidelines

Found while doing #74172.

@ecstatic-morse ecstatic-morse added C-bug Category: This is a bug. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Jul 9, 2020
@RalfJung
Copy link
Member

RalfJung commented Jul 9, 2020

This one seems so obvious that I'm wondering if I've overlooked something.

This code was probably written before we had a clear understanding of our invariants, and before we had MaybeUninit to properly work with uninitialized data.

Cc rust-lang/unsafe-code-guidelines#77

@ecstatic-morse
Copy link
Contributor Author

ecstatic-morse commented Jul 9, 2020

This one seems so obvious that I'm wondering if I've overlooked something.

I need to stop hedging 😄.

I also had assumed that the question of whether &T required T to be initialized was more settled than the discussion in rust-lang/unsafe-code-guidelines#77 suggests. With that in mind, how do we want to prioritize this? I think there's precedent for the standard library to rely on behavior that is technically undefined when there is active discussion about the rules and we aren't relying on that property for optimizations. That seems to be the case here?

@RalfJung
Copy link
Member

I think there's precedent for the standard library to rely on behavior that is technically undefined when there is active discussion about the rules and we aren't relying on that property for optimizations. That seems to be the case here?

Yeah, I think we can treat it like that.

@yaahc yaahc added the E-help-wanted Call for participation: Help is requested to fix this issue. label Jan 26, 2022
@the8472 the8472 added the A-collections Area: `std::collection` label Feb 8, 2022
@JmPotato
Copy link
Contributor

@rustbot claim

@JmPotato
Copy link
Contributor

Hi, @ecstatic-morse @RalfJung. After reading the code, it seems that safety is guaranteed by the check of the head and tail index, so there is no implementation that will access these uninitialized memories currently. However, considering the possibility of these codes being modified in the future, we do need to do something to make it be aware. Should I change &[T] to something like &[MaybeUninit<T>]. or just add some document? Any guidance would be helpful and appreciated!

@RalfJung
Copy link
Member

RalfJung commented Feb 28, 2022

The main concern is that this code is violating the rules that are explicitly stated at https://doc.rust-lang.org/reference/behavior-considered-undefined.html. So we have two options, both of which seem reasonable to me:

  • Adjust the code to follow the rules.
  • Since this is the standard library, we could alternatively document that we rely on privileged compiler-internal knowledge saying that violating these particular rules in this particular way is actually fine. If the compiler changes in the future in a way that makes this no longer fine, we will have to adjust that code, but since the standard library and the compiler are developed together, that kind of coupling is acceptable. (This would not be an option if the exact same code was written in a regular crate.)

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Mar 9, 2022
…ue, r=m-ou-se

Use MaybeUninit in VecDeque to remove the undefined behavior of slice

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Ref rust-lang#74189. Adjust the code to follow the [doc.rust-lang.org/reference/behavior-considered-undefined.html](https://doc.rust-lang.org/reference/behavior-considered-undefined.html).

* Change the return type of `buffer_as_slice` from `&[T]` to `&[MaybeUninit<T>]`.
* Add some corresponding safety comments.

Benchmark results:

master 8d6f527

```rust
test collections::vec_deque::tests::bench_pop_back_100       ... bench:          47 ns/iter (+/- 1)
test collections::vec_deque::tests::bench_pop_front_100      ... bench:          50 ns/iter (+/- 4)
test collections::vec_deque::tests::bench_push_back_100      ... bench:          69 ns/iter (+/- 10)
test collections::vec_deque::tests::bench_push_front_100     ... bench:          72 ns/iter (+/- 6)
test collections::vec_deque::tests::bench_retain_half_10000  ... bench:     145,891 ns/iter (+/- 7,975)
test collections::vec_deque::tests::bench_retain_odd_10000   ... bench:     141,647 ns/iter (+/- 3,711)
test collections::vec_deque::tests::bench_retain_whole_10000 ... bench:     120,132 ns/iter (+/- 4,078)
```

This PR

```rust
test collections::vec_deque::tests::bench_pop_back_100       ... bench:          48 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_pop_front_100      ... bench:          51 ns/iter (+/- 3)
test collections::vec_deque::tests::bench_push_back_100      ... bench:          73 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_push_front_100     ... bench:          73 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_retain_half_10000  ... bench:     131,796 ns/iter (+/- 5,440)
test collections::vec_deque::tests::bench_retain_odd_10000   ... bench:     137,563 ns/iter (+/- 3,349)
test collections::vec_deque::tests::bench_retain_whole_10000 ... bench:     128,815 ns/iter (+/- 3,289)
```
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Mar 9, 2022
…ue, r=m-ou-se

Use MaybeUninit in VecDeque to remove the undefined behavior of slice

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Ref rust-lang#74189. Adjust the code to follow the [doc.rust-lang.org/reference/behavior-considered-undefined.html](https://doc.rust-lang.org/reference/behavior-considered-undefined.html).

* Change the return type of `buffer_as_slice` from `&[T]` to `&[MaybeUninit<T>]`.
* Add some corresponding safety comments.

Benchmark results:

master 8d6f527

```rust
test collections::vec_deque::tests::bench_pop_back_100       ... bench:          47 ns/iter (+/- 1)
test collections::vec_deque::tests::bench_pop_front_100      ... bench:          50 ns/iter (+/- 4)
test collections::vec_deque::tests::bench_push_back_100      ... bench:          69 ns/iter (+/- 10)
test collections::vec_deque::tests::bench_push_front_100     ... bench:          72 ns/iter (+/- 6)
test collections::vec_deque::tests::bench_retain_half_10000  ... bench:     145,891 ns/iter (+/- 7,975)
test collections::vec_deque::tests::bench_retain_odd_10000   ... bench:     141,647 ns/iter (+/- 3,711)
test collections::vec_deque::tests::bench_retain_whole_10000 ... bench:     120,132 ns/iter (+/- 4,078)
```

This PR

```rust
test collections::vec_deque::tests::bench_pop_back_100       ... bench:          48 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_pop_front_100      ... bench:          51 ns/iter (+/- 3)
test collections::vec_deque::tests::bench_push_back_100      ... bench:          73 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_push_front_100     ... bench:          73 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_retain_half_10000  ... bench:     131,796 ns/iter (+/- 5,440)
test collections::vec_deque::tests::bench_retain_odd_10000   ... bench:     137,563 ns/iter (+/- 3,349)
test collections::vec_deque::tests::bench_retain_whole_10000 ... bench:     128,815 ns/iter (+/- 3,289)
```
Dylan-DPC added a commit to Dylan-DPC/rust that referenced this issue Mar 9, 2022
…ue, r=m-ou-se

Use MaybeUninit in VecDeque to remove the undefined behavior of slice

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Ref rust-lang#74189. Adjust the code to follow the [doc.rust-lang.org/reference/behavior-considered-undefined.html](https://doc.rust-lang.org/reference/behavior-considered-undefined.html).

* Change the return type of `buffer_as_slice` from `&[T]` to `&[MaybeUninit<T>]`.
* Add some corresponding safety comments.

Benchmark results:

master 8d6f527

```rust
test collections::vec_deque::tests::bench_pop_back_100       ... bench:          47 ns/iter (+/- 1)
test collections::vec_deque::tests::bench_pop_front_100      ... bench:          50 ns/iter (+/- 4)
test collections::vec_deque::tests::bench_push_back_100      ... bench:          69 ns/iter (+/- 10)
test collections::vec_deque::tests::bench_push_front_100     ... bench:          72 ns/iter (+/- 6)
test collections::vec_deque::tests::bench_retain_half_10000  ... bench:     145,891 ns/iter (+/- 7,975)
test collections::vec_deque::tests::bench_retain_odd_10000   ... bench:     141,647 ns/iter (+/- 3,711)
test collections::vec_deque::tests::bench_retain_whole_10000 ... bench:     120,132 ns/iter (+/- 4,078)
```

This PR

```rust
test collections::vec_deque::tests::bench_pop_back_100       ... bench:          48 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_pop_front_100      ... bench:          51 ns/iter (+/- 3)
test collections::vec_deque::tests::bench_push_back_100      ... bench:          73 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_push_front_100     ... bench:          73 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_retain_half_10000  ... bench:     131,796 ns/iter (+/- 5,440)
test collections::vec_deque::tests::bench_retain_odd_10000   ... bench:     137,563 ns/iter (+/- 3,349)
test collections::vec_deque::tests::bench_retain_whole_10000 ... bench:     128,815 ns/iter (+/- 3,289)
```
bors added a commit to rust-lang-ci/rust that referenced this issue Mar 11, 2022
…, r=m-ou-se

Use MaybeUninit in VecDeque to remove the undefined behavior of slice

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Ref rust-lang#74189. Adjust the code to follow the [doc.rust-lang.org/reference/behavior-considered-undefined.html](https://doc.rust-lang.org/reference/behavior-considered-undefined.html).

* Change the return type of `buffer_as_slice` from `&[T]` to `&[MaybeUninit<T>]`.
* Add some corresponding safety comments.

Benchmark results:

master 8d6f527

```rust
test collections::vec_deque::tests::bench_pop_back_100       ... bench:          47 ns/iter (+/- 1)
test collections::vec_deque::tests::bench_pop_front_100      ... bench:          50 ns/iter (+/- 4)
test collections::vec_deque::tests::bench_push_back_100      ... bench:          69 ns/iter (+/- 10)
test collections::vec_deque::tests::bench_push_front_100     ... bench:          72 ns/iter (+/- 6)
test collections::vec_deque::tests::bench_retain_half_10000  ... bench:     145,891 ns/iter (+/- 7,975)
test collections::vec_deque::tests::bench_retain_odd_10000   ... bench:     141,647 ns/iter (+/- 3,711)
test collections::vec_deque::tests::bench_retain_whole_10000 ... bench:     120,132 ns/iter (+/- 4,078)
```

This PR

```rust
test collections::vec_deque::tests::bench_pop_back_100       ... bench:          48 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_pop_front_100      ... bench:          51 ns/iter (+/- 3)
test collections::vec_deque::tests::bench_push_back_100      ... bench:          73 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_push_front_100     ... bench:          73 ns/iter (+/- 2)
test collections::vec_deque::tests::bench_retain_half_10000  ... bench:     131,796 ns/iter (+/- 5,440)
test collections::vec_deque::tests::bench_retain_odd_10000   ... bench:     137,563 ns/iter (+/- 3,349)
test collections::vec_deque::tests::bench_retain_whole_10000 ... bench:     128,815 ns/iter (+/- 3,289)
```
@JmPotato
Copy link
Contributor

@ecstatic-morse Hi, I think we can close this issue now since it has been fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-collections Area: `std::collection` C-bug Category: This is a bug. E-help-wanted Call for participation: Help is requested to fix this issue. T-libs Relevant to the library team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

5 participants