[dev] Optimize batch fetch method to boost throughput

[NiuBlibing]

Description

The previous start url fetching method only working when spider is idle, which is not full concurrency.This patch optimizes it by using request_left_downloader signal.

There maybe need a lock for calculating the need_size.

Fixes #119

How Has This Been Tested?

• tox -e py38 -- tests/

Test Configuration:

• OS version: debian 11 and Windows10
• Necessary Libraries (optional):

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[LuckyPigeon]

@NiuBlibing Please resolve the assertion error. And add unit test for fill_requests_queue, thanks!

[LuckyPigeon]

@rmax How do you think about this implementation, it disabled spider_idel usage. I wonder if we need a switch between spider_idle and fill_requests_queue.

[rmax]

@rmax How do you think about this implementation, it disabled spider_idel usage. I wonder if we need a switch between spider_idle and fill_requests_queue.

Interesting the use of the other signal. What scrapy version is required for the new signal?

What happens with existing users that override the spider_idle method?

Does it make sense to bump the major version? Or somewhat related, shall we migrate to calendar versioning?