scrapyd not fully utilizing cpu

[KarlMaresch]

We have a lot of perodic crawler jobs to do and therefore bought a server with a strong 24 core - 48 threads amd epyc cpu. Currently there are always between 7000 & 9000 crawler jobs on pending, so there is enough work in the pipeline. However, scrapyd is only running between ~8 and ~max. 20 jobs simultaneously.

Already tried many changes on the config file, but none fixed the problem.

Current config:

[scrapyd]
eggs_dir    = eggs
logs_dir    = logs
items_dir   = items        
jobs_to_keep = 2000000     
dbs_dir     = dbs
max_proc    = 0
max_proc_per_cpu = 48      
finished_to_keep = 2000000 
poll_interval = 0.1        
bind_address = 0.0.0.0
http_port   = 6800
debug       = on
runner      = scrapyd.runner
application = scrapyd.app.application
launcher    = scrapyd.launcher.Launcher
webroot     = scrapyd.website.Root
```

Anybody experienced similar behaviour or found a way how to fix this?

[my8100]

max_proc

The maximum number of concurrent Scrapy process that will be started. If unset or 0 it will use the number of cpus available in the system multiplied by the value in max_proc_per_cpu option. Defaults to 0.

max_proc    = 2000000

[KarlMaresch]

max_proc

The maximum number of concurrent Scrapy process that will be started. If unset or 0 it will use the number of cpus available in the system multiplied by the value in max_proc_per_cpu option. Defaults to 0.

max_proc    = 2000000

I thought from the description it means that if we leave max_proc to zero it will anyway use the maxmimum available number. But I will try as soon as the pipeline is cleared your proposal with max_proc = 2000000

[Digenis]

@KarlMaresch,
any chance that the rate of scheduling is greater that the rate of running?
In other words:
If the poll_interval is 5 seconds,
on average a new crawl is scheduled every 4 seconds
and an average run lasts 5 seconds
the slots will never fill.

#173 can be a solution.

Also, if you have more than one project, you are affected by #187.

[jpmckinney]

Closing as no follow-up after maintainers' advice.