feat: use a shared queue across workers for data processing

[deependujha]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Fixes #556

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

Attention: Patch coverage is 49.36709% with 40 lines in your changes missing coverage. Please review.

Project coverage is 71%. Comparing base (9fb8019) to head (b26c596).
Report is 9 commits behind head on main.

❌ Your patch status has failed because the patch coverage (49%) is below the target coverage (50%). You can increase the patch coverage or adjust the target coverage.

❗ There is a different number of reports uploaded between BASE (9fb8019) and HEAD (b26c596). Click for more details.

HEAD has 12 uploads less than BASE

Flag	BASE (9fb8019)	HEAD (b26c596)
unittests	6	2
3.10	3	1
3.9	3	1
ubuntu-22.04	2	0
macos-14	2	0

Additional details and impacted files

@@         Coverage Diff          @@
##           main   #559    +/-   ##
====================================
- Coverage    79%    71%    -8%     
====================================
  Files        40     40            
  Lines      6098   6136    +38     
====================================
- Hits       4822   4384   -438     
- Misses     1276   1752   +476     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tchaton]

@deependujha Let's keep 2 base class. One ordered and one not with a single queue

[deependujha]

@benchmark

[github-actions]

Hi @deependujha!

❌ Benchmark failed

Traceback (most recent call last):
  File "/home/runner/work/litData/litData/.github/benchmark/benchmark.py", line 156, in <module>
    main()
  File "/home/runner/work/litData/litData/.github/benchmark/benchmark.py", line 150, in main
    benchmark.run()
  File "/home/runner/work/litData/litData/.github/benchmark/benchmark.py", line 83, in run
    self.setup_studio()
  File "/home/runner/work/litData/litData/.github/benchmark/benchmark.py", line 92, in setup_studio
    self.studio = Studio(
                  ^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/lightning_sdk/studio.py", line 57, in __init__
    self._studio_api = StudioApi()
                       ^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/lightning_sdk/api/studio_api.py", line 55, in __init__
    self._client = LightningClient(max_tries=7)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/lightning_sdk/lightning_cloud/rest_client.py", line 184, in __init__
    super().__init__(api_client=create_swagger_client())
                                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/lightning_sdk/lightning_cloud/rest_client.py", line 68, in create_swagger_client
    api_client.default_headers["Authorization"] = Auth().authenticate()
                                                  ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.12/site-packages/lightning_sdk/lightning_cloud/login.py", line 131, in authenticate
    self._run_server()
  File "/home/runner/.local/lib/python3.12/site-packages/lightning_sdk/lightning_cloud/login.py", line 115, in _run_server
    AuthServer().login_with_browser(self)
  File "/home/runner/.local/lib/python3.12/site-packages/lightning_sdk/lightning_cloud/login.py", line 174, in login_with_browser
    raise RuntimeError("Failed to authenticate to Lightning. When running without access to a browser, 'LIGHTNING_USER_ID' and 'LIGHTNING_API_KEY' should be exported.")
RuntimeError: Failed to authenticate to Lightning. When running without access to a browser, 'LIGHTNING_USER_ID' and 'LIGHTNING_API_KEY' should be exported.

cc: @tchaton @deependujha @bhimrazy

[deependujha]

closing in favour of #592