Skip to content
This repository has been archived by the owner on Jul 16, 2021. It is now read-only.

Allow pausing batches #2664

Open
MatanYadaev opened this issue Jul 12, 2021 · 0 comments
Open

Allow pausing batches #2664

MatanYadaev opened this issue Jul 12, 2021 · 0 comments

Comments

@MatanYadaev
Copy link

What

An ability to store a batch in the database and control when it should be executed.
It can be implemented by pause and continue methods.

Why

I, for example, want to dispatch 5,000 jobs in a batch. Every job contains a User model with many relationships. Fetching these models consumes much memory, so I must split the fetching into a few chunks, and push every chunk to the existing batch.
But the problem is that the batch is already running. There's a race condition problem here: the workers might finish the pending jobs before the next chunk is fetched and pushed to the batch - and this will result in completing the batch before it really gets finished.
If there was an option to pause the existing batch, and start it just after the whole chunks got stored - it will fix the race condition problem.

Example - Current state

$batch = Bus::batch([])->allowFailures()->dispatch();

User::query()
    ->addManyRelations()
    ->chunkById(1000, static function (Collection $usersChunk) use ($batch): void {
        $jobs = $usersChunk->map(static function (User $user) {
            return new SendNotificationToUserJob($user);
        });

        // The `$batch` might be finished before the second chunk is fetched.
        $batch->add($jobs);
    });

Example - After implementing pausing batches

$batch = Bus::batch([])->allowFailures()->dispatchPaused(); // or `->dispatch()->pause()`

User::query()
    ->addManyRelations()
    ->chunkById(1000, static function (Collection $usersChunk) use ($batch): void {
        $jobs = $usersChunk->map(static function (User $user) {
            return new SendNotificationToUserJob($user);
        });

        $batch->add($jobs);
    });

$batch->continue();
  • I'm not sure "pause" is the best term for this feature. But the ability to decide when a batch should be executed is what I meant.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant