Address Pandas Dataframe Merge and Other Performance Issues

[d33bs]

Pandas Dataframe merge (and likely other) performance during various runtime operations may hinder or completely stall progress. This issue is dedicated to addressing Pandas merge and other performance challenges, including solutions which may not involve or migrate from Pandas itself.

Issues which may be related or tied to this:

• Potential memory leak in SingleCell's .merge_single_cells() method #195
• Address SQLite Read Performance Issues #198

[d33bs]

Hi @gwaygenomics, just wanted to provide a quick update on this issue based on some findings yesterday. I ran some tests using some of the work from #198 to see just how much we could reduce the memory consumption. In addition to the dataset being modified to include NULL's in the float columns and using ConnectorX for the SQLite read, there are a few performance tweaks that I found decreased the memory consumption:

• Reduce to only one pd.DataFrame.rename() for columns after the final merge in merge_single_cells (acknowledging both dictionaries).
• Switch the final merge to be how="left" (also swapping left and right merge members).
• Retain no-copy, no-inplace rename work from Potential memory leak in SingleCell's .merge_single_cells() method #195

1. As-is performance with SQLite 'nan's:
   • Duration: ~28 minutes
   • Peak Memory Consumption: 64.7 GB
   • Link to flamegraph
2. ConnectorX reads for load_compartment and with SQLite 'nan's replaced by NULL:
   • Duration: ~6 minutes (down ~22 minutes from as-is)
   • Peak Memory Consumption: 42.4 GB (down 22.3 GB from as-is)
   • Link to flamegraph
3. Same as "2." and with the above "performance tweaks" added.
   • Duration: ~5 minutes (down ~23 minutes from as-is)
   • Peak Memory Consumption: 32.7 GB (down 32 GB from as-is)
   • Link to flamegraph

I'm still working on the merges themselves as these seem to still be a large bottleneck of memory consumption. In exploring performance with another resource profiling tool, Scalene, it predicts (using some presumptions built into that library) that the Pandas merges are a possible source of memory leakage, but only does so when using large datasets. I've stored some early results from this tool here. Please note there are some differences in how this tool vs Memray measure performance, so they do not show the exact same results.

[gwaybio]

@d33bs - does #219 solve this issue? Can we close?

[d33bs]

Hi @gwaybio - I feel good about closing this issue given #219's solution. If there are other known areas where Pandas DataFrame merges are causing performance issues we could address those more specifically as they come up.