Skip to content

ENH: Performance enhancements to nanops using numba #62449

@eoincondron

Description

@eoincondron

The performance of array reductions in nanops/bottleneck can be significantly improved upon for large data using numba. The improvements are due to two factors:

  • single-pass algorithms when null values are present and avoiding any copies.
  • multi-threading over chunked of array or over an axis in a single axis reduction.

This screenshot demonstrates a potential 4x improvement on a DataFrame of 10-million rows and 5 columns of various types.

Image

I am running the code on a features branch, and all unit tests for the feature branch are passing locally.
https://github.com/eoincondron/pandas/tree/nanops-numba-implementation

The hardware is a new MacBook Pro with 8 cores.

The performance is still slightly better at 1-million rows and is even greater at larger magnitudes (8x at 100 million rows).
The caveat is that all JIT-compilation is already completed.
I have carried out a more comprehensive performance comparison and these results hold up.

Similarly to bottleneck, these codepaths can be toggled on and off.

Metadata

Metadata

Assignees

No one assigned

    Labels

    DependenciesRequired and optional dependenciesNeeds DiscussionRequires discussion from core team before further actionPerformanceMemory or execution speed performanceReduction Operationssum, mean, min, max, etc.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions