computeGCBias is really slow in deepTools2 compared to deepTools 1.5

[asrichter]

computeGCBias is VERY SLOW in the current deepTools 2 develop version. Even on small input BAM files with only thousands of alignments, it runs hours. When using 16 or more CPUs for example, each thread is at about 5-10% CPU usage only, also at directly attached storage without NFS bottleneck. The problem occurs in both python 2 and 3.
In deepTools 1.5.9.1, it used to be much faster.

Any ideas which change could have caused the performance loss?

[dpryan79]

Is this with python2, python3, or both?

[asrichter]

The problem occurs in both python 2 and 3.

[dpryan79]

Eeek

[fidelram]

I suspect that the 'chunksize' that is given to map_reduce is too small.

[dpryan79]

That could be, the other big change is the switch from bx-python to twobitreader for the 2bit stuff. They're both python, but perhaps the latter isn't as performant.

[fidelram]

I ran a test with 11M reads and default parameters that took only 3 mins.

@asrichter are you using a genome assembly with lot of contigs? I wonder if this is related to #385

[asrichter]

Yes, I did. Please have a look at human genome assembly hs37d5.

[vivekbhr]

I had the same issue with many contigs before with plotEnrichment. So the fix that @dpryan79 used for plotEnrichment in my case earlier, should also work for computeGCbias i think..

But then I think we should have a generic solution to handle multiprocessing with many contigs.

[vivekbhr]

@dpryan79 does shuffling tasks for multiprocessing ( #386 ) solve this issue for now?

[dpryan79]

@vivekbhr Unlikely, though perhaps. I think the GC bias stuff is the only code not using the unified mapReduce module, so they won't then benefit from that. I need to run some tests with this and also python3 versions of tools to see where slow downs are happening (in the case of python3 there may be nothing we can do, since the language itself is simply slower than python2).

[dpryan79]

I've started profiling 1.5.11 and 2.3.2 under python2 with the Input.bam file in the test dataset. I'll post more when that's done (it takes a bit of time when run single threaded...).