Improve product export performance

[convenient]

The Bug

I discovered this bug in EE 1.14 while investigating it for a client. The same problem exists in Magento 2.

It boils down to the fact that as the number of categories and attributes increases the vanilla Magento product export gets significantly slower. All due to the bottleneck of the PHP array_intersect function.

The array_intersect function has remained largely unchanged from 5.3 to 5.5, so this problem will likely affect most versions of magento.

Based on my reading on stackoverflow and elsewhere I tried replacing array_intersect with a combination of array_combine and array_intersect_key, and saw that it greatly improved performance. The full product export for the client dropped from ~10 minutes to ~2 minutes.

Replication

My (not very scientific) method of testing and replicating this locally was to install magento/module-sample-data at 1.0.0-beta.

As this sample data was not enough to trigger the array_intersect bottleneck, I programatically added another 2500 categories. Giving me a grand total of 2540 categories.

Click 99b944e to see the script that generated them.

Profiling

I used xhprof to profile the export method, see the commits c6342a8 and cb4a42b.

In the following xhprof screenshots you can easily see that array_intersect takes first place as the bottleneck, once the code change is applied the biggest bottleneck sensibly goes back to PDOStatement::execute

Using array_intersect

Using array_combine and array_intersect_key

[kokoc]

@convenient thank you for contribution! Great investigation! Internal reference: MAGETWO-41083

[convenient]

@kokoc Thanks! You guys may want to back port this as a patch for 1.14 and 1.9 etc. 👍

[kokoc]

@convenient your contribution is merged into develop branch. Thanks again!

[convenient]

@kokoc Woo! Glad to see it got merged straight in. Thanks.