Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix FuturesOrdered #2664

Merged
merged 1 commit into from
Jan 15, 2023
Merged

fix FuturesOrdered #2664

merged 1 commit into from
Jan 15, 2023

Conversation

conradludgate
Copy link
Contributor

@conradludgate conradludgate commented Nov 12, 2022

fixes #2663

@conradludgate conradludgate changed the title fix FuturesOrdered #2663 fix FuturesOrdered Nov 12, 2022
@taiki-e
Copy link
Member

taiki-e commented Nov 20, 2022

Thanks for the PR! Considering the overflow of isize (which can actually happen on 32-bit targets), this still seems broken, but this seems more correct than the current one.

@conradludgate
Copy link
Contributor Author

conradludgate commented Nov 20, 2022

Yeah. I have some ideas how to fix the overflow too, but it's not as trivial a fix

https://github.com/conradludgate/futures-buffered/blob/7db2b4ca822ee95bb53871fee7867b420a04a7fa/src/futures_ordered_bounded.rs#L216-L231

Essentially, if the next_outgoing_index has the MSB set, then it flips the MSB of every task/output. This preserves the order and ensures that the outgoing is <= isize::MAX. The length of the queue would have to be another isize::MAX+1 in order to overflow, but you'd OOM before that.

Reason this is correct Let's say our system is 16-bit. So we can have a theoretical maximum memory of 65536 bytes. Our futures are `[(usize, F)]` which is at least 2 bytes assuming ZST future. Our ready queue similarly stores `[(usize, F::Output)]` which is also at least 2 bytes with a ZST output.

In total, we can only fit 65536/2 entries into memory, ignoring any extra necessary overhead. Therefore, we only need to be able to store at maximum (usize::MAX+1)/2 entries.

In our FuturesOrdered, we have next_outgoing_index as the head, and next_incoming_index as the tail.

[_, _, .., _, _, _, _, _, .., _, _]
           ^ head.     ^ tail

If our head gets over half way through our usize, then to store our theoretical maximum, tail can be at most usize::MAX+1, which is 0. We can't insert any more without OOM as shown, so this index is never read or compared against.

Since all other indices must be head <= x < tail, then we know that they all have the MSB set, so flipping the MSB will be the same as x - (usize::MAX+1)/2 which must preserve the order of the queues.

It should be possible to implement here too, but it might not be super efficient with the task linked list

@taiki-e taiki-e merged commit 54e07b8 into rust-lang:master Jan 15, 2023
@taiki-e taiki-e added A-stream Area: futures::stream 0.3-backport: pending The maintainer accepted to backport this to the 0.3 branch, but backport has not been done yet. labels Jan 15, 2023
@taiki-e
Copy link
Member

taiki-e commented Jan 15, 2023

Thanks! Opened #2689 to track overflow issue.

@taiki-e taiki-e added 0.3-backport: completed and removed 0.3-backport: pending The maintainer accepted to backport this to the 0.3 branch, but backport has not been done yet. labels Jan 30, 2023
taiki-e pushed a commit that referenced this pull request Jan 30, 2023
@taiki-e taiki-e mentioned this pull request Jan 30, 2023
taiki-e pushed a commit that referenced this pull request Jan 30, 2023
crapStone pushed a commit to Calciumdibromid/CaBr2 that referenced this pull request Feb 2, 2023
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [futures](https://rust-lang.github.io/futures-rs) ([source](https://github.com/rust-lang/futures-rs)) | dependencies | patch | `0.3.25` -> `0.3.26` |

---

### Release Notes

<details>
<summary>rust-lang/futures-rs</summary>

### [`v0.3.26`](https://github.com/rust-lang/futures-rs/blob/HEAD/CHANGELOG.md#&#8203;0326---2023-01-30)

[Compare Source](rust-lang/futures-rs@0.3.25...0.3.26)

-   Add `Either::as_pin_mut` and `Either::as_pin_ref` ([#&#8203;2691](rust-lang/futures-rs#2691))
-   Add `Shared::ptr_eq` and `Shared::ptr_hash` ([#&#8203;2691](rust-lang/futures-rs#2691))
-   Implement `FusedStream` for `Buffered` ([#&#8203;2676](rust-lang/futures-rs#2676))
-   Implement `FusedStream` for all streams in `ReadyChunks` ([#&#8203;2693](rust-lang/futures-rs#2693))
-   Fix bug in `FuturesOrdered::push_front` ([#&#8203;2664](rust-lang/futures-rs#2664))
-   Remove `Fut::Output: Clone` bounds from some `Shared` methods ([#&#8203;2662](rust-lang/futures-rs#2662))
-   Remove `T: Debug` bounds from `Debug` implementations of `mpsc` and `oneshot` types ([#&#8203;2666](rust-lang/futures-rs#2666), [#&#8203;2667](rust-lang/futures-rs#2667))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xMTYuMSIsInVwZGF0ZWRJblZlciI6IjM0LjExOS4yIn0=-->

Co-authored-by: cabr2-bot <cabr2.help@gmail.com>
Reviewed-on: https://codeberg.org/Calciumdibromid/CaBr2/pulls/1762
Reviewed-by: crapStone <crapstone@noreply.codeberg.org>
Co-authored-by: Calciumdibromid Bot <cabr2_bot@noreply.codeberg.org>
Co-committed-by: Calciumdibromid Bot <cabr2_bot@noreply.codeberg.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

FuturesOrdered push_front does not work at the start
2 participants