-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow concurrency in concatMap
#5519
Comments
Hi @JosephSilber (and sorry if I misunderstood your problem), but would the following code solve your issue? from(orderNumbers.map((order, idx) => [order, idx]))
.pipe(mergeMap(([order, idx]) => shipOrder(order).then(x => [x, idx]), 5))
.pipe(toArray())
.pipe(
mergeMap(labelsByIdx =>
labelsByIdx.sort((a, b) => a[1] - b[1]).map(([label]) => label)
)
)
.pipe(printLabel); |
@josepot that would wait for all orders to have shipped, before printing the very first label. That could take quite a while. I want the labels to be printed as soon as possible, while maintaining their original sort order. Here's a simulation of what I'm after: |
@JosephSilber I've been thinking about this today and these are my thoughts: (FTR: I'm not a maintainer of this library and I don't pretend to be an authority on these topics. I'm just a RxJS aficionado sharing my thoughts 🙂 ) If the After thinking quite a bit about this I've gone from: "nah, this is too confusing" to "this actually makes sense". What made me change my mind was the fact that I realized that understood However, if we consider the "concatenation" behavior, without thinking about how it's implemented, then I think that you are suggesting makes a lot of sense. Perhaps the best way to convince the maintainers of RxJS that these changes are worth it, would be to first implement these operators in user-land, show that there is a need for them (ppl are using them) and after we've got a good implementation and a good set of tests in user-land, then open a PR against RxJS. In that sense, perhaps it would make sense to send a PR to rxjs-etc (a library maintained by @cartant) with these operators and try to prove with examples and tests that it's worth it to bring those changes to core. I just quickly implemented these operators in a personal library of mine. Whenever I have some time I will add some tests and I will send a PR to rxjs-etc. By all means, feel free to do that yourself, because I don't know when I will have the time to add the tests and send the PR. |
I see that as a |
Hi. I'm not sure that I have understood everything (I wasn't reading carefully though), but, if I'm right, you may want to have something like RxJava's concatEager. If this is something you need, and if you implement it, please post your solution here as that's something I may need. If I'm wrong, I'm sorry, just ignore this. |
💯
🤔 I guess... I tried, and it seems to work, but I think that I would rather have a custom operator using This is what I came up with using @cartant 's suggestion (please don't judge me, I did that while being in a rush 🙈 ): const DONE = Symbol("DONE");
const DONE$ = of(DONE);
const sortedMergeMap = <I, O>(
mapper: (i: I) => ObservableInput<O>,
concurrent = 1
) => (source$: Observable<I>) =>
source$.pipe(
mergeMap(
(value, idx) =>
concat(mapper(value), DONE$).pipe(map(x => [x, idx] as const)),
concurrent
),
scan(
(acc, [value, idx]) => {
if (idx === acc.currentIdx) {
if (value === DONE) {
let currentIdx = idx;
const valuesToEmit = [];
do {
currentIdx++;
const nextValues = acc.buffer.get(currentIdx);
if (!nextValues) {
break;
}
valuesToEmit.push(...nextValues);
acc.buffer.delete(currentIdx);
} while (valuesToEmit[valuesToEmit.length - 1] === DONE);
return {
...acc,
currentIdx,
valuesToEmit: valuesToEmit.filter(x => x !== DONE) as O[]
};
} else {
return {
...acc,
valuesToEmit: [value]
};
}
} else {
if (!acc.buffer.has(idx)) {
acc.buffer.set(idx, []);
}
acc.buffer.get(idx)!.push(value);
if (acc.valuesToEmit.length > 0) {
acc.valuesToEmit = [];
}
return acc;
}
},
{
currentIdx: 0,
valuesToEmit: [] as O[],
buffer: new Map<number, (O | typeof DONE)[]>([[0, []]])
}
),
mergeMap(scannedValues => scannedValues.valuesToEmit)
); |
@jakovljevic-mladen yes, that's almost exactly what we are talking about. The only difference is that Yesterday night I took some time to carefully implement a version of this operator for JS. Today I've added a few tests and I've opened this PR against |
Closing this because there is now a user-land equivalent of RxJava's ATM, inclusion of an operator like |
Core Team Meeting: This has been asked for quite a few times. Although |
Should this be a new operator though? or a variant of I guess there might be some confusion as the I cannot find it unfortunately, but there was discussion last year about restructuring the implementation of these operators to a single operator with concurrency, switching and ordering capabilities, and have the |
Yeah, I've changed my mind on this. IMO, the OP's initial suggestion is the best solution. |
Yea agreed whatever it ends up as, it should have restrictive concurrency capabilities - my actual use case is with webworkers, I want the inner subscriptions to all be run in parallel up to a maximum number (thread count), but have the combination of the results of those workers to be in the original order that the data was passed to them. With the currently implementation of concatAll, only one worker is used at a time, which doesn't help with performance :) |
@benlesh with I'd be happy to have a crack at it and raise a PR if there is appetite to extend the API. |
Feature Request
Is your feature request related to a problem? Please describe.
We have
mergeMap
which can map values concurrently, but the new values it produces are not in the original order. If you want to maintain order, you must useconcatMap
, butconcatMap
doesn't run concurrently (it's equivalent to callingmergeMap
withconcurrent
set to1
).Describe the solution you'd like
It would be great if
concatMap
could take aconcurrent
parameter, just likemergeMap
already does:Describe alternatives you've considered
I don't really have a full-on alternative. In my particular use-case the inner observables only produced a single value (they're
Promise
s), so I was able to build a convoluted workaround. See the next section.Additional context
Here's my use-case:
I'm shipping a bunch of orders, and then printing their shipping label.
Here are those functions:
Now, I want to ship multiple orders at once:
...but I want the shipping labels to be printed in their original sort order.
This is where
concatMap
with concurrency would help:The orders would still be shipped in parallel, but the labels will all print in the correct order.
Here's a StackOverflow question and answer with way more details:
https://stackoverflow.com/questions/57045892/rxjs-mergemap-with-original-order
The text was updated successfully, but these errors were encountered: