Poller should fill all available slots

[Tarliton]

Today we can choose "poll_interval", "max_proc" and "max_proc_per_cpu". "max_proc" and "max_proc_per_cpu" are for limiting a maximum of jobs that can be running.

If "poll_interval" is high that maximum would never be reached. That happens because after each poll interval we start only one job.

E.g.: poll_interval = 30 and some spider takes about 2 minutes to finish. The maxium jobs that would run in this case is 4~5, no matter what "max_proc" and "max_proc_per_cpu" values are.

Maybe it should have an option "jobs_to_poll". With that we could choose how many jobs to poll each time, instead of only one.

What you guys think about that?

Thank you

[Digenis]

I'd expect developers in this case to write spiders
that don't close but stay open to crawl on demand
but I do think that your concern is a valid.
However I'm not the appropriate person to look into it.
Someone more familiar with twisted can have a look.

[Digenis]

Actually it makes sense for the poller
to poll for exactly as many spiders as the amount of free slots.
This should be easier if the project queues are unified in a single db(and/or table)
as I suggest in #187

[Digenis]

By the way, you can set the poller to poll on a sub-second interval.
I add it in 10dfee0

[jpmckinney]

Closing as duplicate of #187 and/or #197

Edit: Also, in this scenario, why not just poll more frequently? The problem seems to be the unusually high poll interval.

[jpmckinney]

Re-opening as this is actually separatable from the other issues.

Poller.poll knows when all slots are full by checking dq.waiting, and it knows when it has no pending jobs to return by checking queue.count. So, we should be able to just add these checks into loops.