Update scan queue

[JonoYang]

The scan queue has several issues:

• process_scan crashes when it loses connection with scancode.io. We should ensure that we handle this exception correctly and stop the queue if we can't communicate with scancode.io
• The code that deals with calling scancode.io should also be updated along the lines of https://github.com/nexB/scancode.io/blob/main/scanpipe/pipes/purldb.py .
• process_scan should rotate between the three in-progress scans to see which is available first to be indexed, rather than waiting on one.
• Submitted scans would not always update their status properly and would be stuck in the scan submitted stage.

[404-geek]

@JonoYang can you assign this to me ?

[mohammedabualsoud]

@JonoYang I’m a computer scientist applying through G-SOC24 with a passion for backend technologies.

My focus lies in cloud computing and system design, areas where I’ve honed my expertise through hands-on experience.
My background includes designing microservices, leveraging event-driven architectures, and crafting RESTful APIs. I’m enthusiastic about exploring the codebase.

Looking forward to connecting and discuss the project milestones.

[404-geek]

@JonoYang @keshav-space @pombredanne @tdruez

I have made a Draft GSOC Proposal for the issue linked with below project idea for GSOC

https://github.com/nexB/aboutcode/wiki/GSOC-2024-Project-Ideas/#purldb-add-improved-scan-queue-with-multiple-scio-instances

Proposal Link

Can anyone please have a look into the proposal and vet it to see if the approaches are justified?