Refactor Data Processor to Use a Shared Queue Across Workers

[bhimrazy]

Title: Refactor Data Processor to Use a Shared Queue Across Workers

Description:

Currently, each worker is assigned its own queue and specific data to process. This leads to inefficiencies—particularly when some workers (stragglers) finish much later than others, delaying overall processing time.

We should refactor the data processor to use a single shared queue that all workers on the machine can pull from. This would enable better load distribution and help ensure that all workers finish at roughly the same time.

Additionally, in cloud environments, we could also move toward using a single queue architecture to enable fault tolerance and autoscaling.

Suggested by: @tchaton