Reduce the amount of generic code for ParallelExtend #887
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For unindexed parallel itererators, we've implemented `ParallelExtend`
for most collections using an intermediate `LinkedList<Vec<T>>` like:
```rust
par_iter
.into_par_iter()
.fold(Vec::new, vec_push)
.map(as_list)
.reduce(LinkedList::new, list_append)
```
However, this introduces `Fold`, `Map`, and `Reduce` types that are all
dependent on the input iterator type. When it comes to very complicated
cases like nested tuple unzips, this can add up quickly. For example, in
rust-lang/rust#68926 an 8-way unzip leads to 3.7GB of LLVM IR, with
lines up to 67K characters in long generic types.
Now we add a new `ListVecConsumer` that is not generic at all itself,
and implements `Consumer<T>` etc. generic only on the item type. So each
collection now gets the same `LinkedList<Vec<T>>` as before with:
```rust
par_iter.into_par_iter().drive_unindexed(ListVecConsumer);
```
Each implementation now also separates the code that doesn't need to be
iterator-specific to a separate function, for their `reserve` and final
`extend` from the list data.
That 8-way unzip is now _only_ 1.5GB with lines up to 17K characters.
Compile time drops from 12.8s to 7.7s debug, 32.1s to 26.9s release.
|
bors r+ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For unindexed parallel itererators, we've implemented
ParallelExtendfor most collections using an intermediate
LinkedList<Vec<T>>like:par_iter .into_par_iter() .fold(Vec::new, vec_push) .map(as_list) .reduce(LinkedList::new, list_append)However, this introduces
Fold,Map, andReducetypes that are alldependent on the input iterator type. When it comes to very complicated
cases like nested tuple unzips, this can add up quickly. For example, in
rust-lang/rust#68926 an 8-way unzip leads to 3.7GB of LLVM IR, with
lines up to 67K characters in long generic types.
Now we add a new
ListVecConsumerthat is not generic at all itself,and implements
Consumer<T>etc. generic only on the item type. So eachcollection now gets the same
LinkedList<Vec<T>>as before with:Each implementation now also separates the code that doesn't need to be
iterator-specific to a separate function, for their
reserveand finalextendfrom the list data.That 8-way unzip is now only 1.5GB with lines up to 17K characters.
Compile time drops from 12.8s to 7.7s debug, 32.1s to 26.9s release.