Workers spawns jobs without any limit

[mikonieminen]

Workers seem to spawn jobs without any limit of how many should be run concurrently. This is pretty bad if you have hundreds of thousands of jobs and worker will pull new jobs from active list as fast as backend is configured to check for new jobs.

The problematic code is in https://github.com/geofmureithi/apalis/blob/master/packages/apalis-core/src/worker/mod.rs#L455

                                worker.spawn(fut.map(move |res| {
                                    if let Err(e) = res {
                                        if let Some(ctx) = state.context.as_ref() {
                                            ctx.notify(Worker {
                                                state: Event::Error(e.into()),
                                                id: WorkerId::new_with_instance(
                                                    worker_id.name(),
                                                    instance,
                                                ),
                                            });
                                        };
                                    }
                                }));

Much healthier approach is to limit workers to run just a single job at a time and then expect the user of apalis to configure suitable number of workers, or alternatively have some kind of configuration for maximum concurrency like older version used to have.

In the current shape, worker might end up running hundreds or even thousands of jobs concurrently that can have very nasty side-effects.

[geofmureithi]

The key thing to note is that the part you are highlighting is conditional to the service being ready.
This means you can manage concurrency by adding something like ConcurrencyLayer.
See this comment and thread for a similar conversation.