[8.x] [RFC] PoC for making chained jobs batchable

[ksassnowski]

This is a proof-of-concept to allow chained jobs in a batch. I didn't write any new tests for now, but made sure the old tests still ran.

What's the intention?

I want to be able to define complex processes that consist of multiple jobs. Some of them can happen in parallel but some of them have dependencies on each other. However, I still want to be able to treat the whole batch as one "process". This is similar to the Process Manager (or Saga) pattern.

Example

After adding a new employee to the system, we want to start an onboarding process that will set up all required accounts automatically. None of these steps really have anything to do with each other per se, but together they make up the onboarding process. The onboarding isn't completed unless all of these steps have finished (hence why I want to refer to them as one unit – the batch).

Some of these steps can run in parallel but some can't run unless other steps have finished. You would be able to model this process like this:

Bus::batch([
    new AddUserToGsuite($newEmployee),
    new InstallProfiles($newEmployee),
    Bus::chain([
        new CreateActiveDirectoryUser($newEmployee),
        new SyncLocalActiveDirectoryWithAzure($newEmployee),
        new AssignOffice365LicenseToEmployee($newEmployee),
    ])
]);

Changes

To allow for this, I added a new method dispatchInBatch to PendingChain that only returns the first job of the chain instead of directly dispatching it. This will essentially turn this

Bus::batch([
    new Job1(),
    Bus::chain([
        new Job2(),
        new Job3(),
    ])->dispatchInBatch(),
]);

into this

Bus::batch([
    new Job1(),
    // This job works like a regular chained job. Its $chained property is correctly 
    // filled with the jobs to run after it succeeded.
    new Job2(),
]);

In order for the batch to correctly calculate the number of jobs it needs to run, we can no longer simply count how many entries the $jobs array of the batch has. Instead, for each job we have to determine if it has chained jobs and add them to the total. Otherwise the batch would end too soon (or rather, the then callback would fire too soon. The remaining jobs in the chain would still be executed afterwards).

I want to ensure that the jobs in the chain are also aware that they're being run inside a batch. For this reason we need to ensure that every job in the chain has the reference to the batch set correctly. The first job in the chain gets handled automatically by the batch itself. For each subsequent job we have to manually set the $batchId to the previous job's batch id (if it exists).

---

This code probably has a bunch of issues and edge cases I might have overlooked. I'm also not sure how nesting this even deeper would behave. Treat this as a proof-of-concept so we can discuss the idea.

[taylorotwell]

The general idea seems sound if you want to explore edge cases and nesting behavior.

[taylorotwell]

Feel free to open this back up when you have more time to explore edge cases and nesting.