Speedup data reader

[mr-1993]

Issue #2195, if available:

Description of changes: @emptymalei @julian-sieber and I have implemented a significant speedup of the data reader. @jaheba Please have a look. We significantly simplified the from_schema method in the decoder by using dictionaries directly instead of the need to infer the idx of the ndarray columns (maybe you have further ideas to simplify the code? we did not do this yet in order not to break anything). For the moment, the implementation is only for data frames the rows of which contain 1D- or 2D-arrays.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Please tag this pr with at least one of these labels to make our release process faster: other change

[lostella]

@mr-1993 looks like style checks are failing; we use black to enforce coding styles, you can fix the issue by installing it and running

black src test

and committing the changes it applies.

[mr-1993]

Thanks @lostella! Should be fixed by now

[mr-1993]

Some tests fail atm, but not related to our implementation (numerical instabilities)

[jaheba]

I don't think we want to do this check for every entry.

Rather, I would keep storing ndarray_columns and then just itereate over them.

[mr-1993]

We would have to run some additional experiments to see whether using ndarray_columns really speeds things up significantly. The codebase without it seems to be more elegant. We will provide you with benchmarks as soon as we have them.

[jaheba]

Thanks for the PR.

Do you have any comparison numbers on how much this improves decoding?

[jaheba]

Why stack the value here?

[mr-1993]

Thanks for the comment, does not seem to be necessary, will delete it

[jaheba]

Couldn't we just check the shape of the array?

[emptymalei]

Hi @jaheba ,

The problem is inline with what we discussed last time. What comes out of the pandas conversion for 2d array is something like

np.array([
    np.array([1,2,3]),
    np.array([4,5,6])
])

then if we check the shape, we would only get 1d array. This is why we have

and len(value.shape) == 1

here.

But would be glad to adapt if you have a better idea.

[mr-1993]

To the best of our knowledge this does not work. The problem lies in the pandas conversion of multi-dimensional arrays. One obtains arrays of arrays (instead of multi dim arrays)

[jaheba]

Oh, I see. Thanks for the clarification.

I think neither arrow nor pandas support arrays with 2 or more dimensions, thus resulting in these nested arrays, with dtype=object.

That's why the re-shaping approach makes things so much easier.

Playing around it looks like np.stack is indeed able to handle nested arrays just fine, even with dimensions higher than 2.

However, I think we can enable stacking only on arrays which dtype == object.

[mr-1993]

Thanks for the PR.

Do you have any comparison numbers on how much this improves decoding?

Atm, we have just tested it on our internal data which we are not allowed to share (but which contains a lot of 2D arrays). We will run some consecutive experiments on open source data, but for benchmarking, we already provide this screenshot with the results:

[jaheba]

Thanks! These are massive improvements. Can you share a bit what shape the data has? I assume the larger each row is the bigger the performance difference gets. If I remember correctly, I saw a 2x improvement on some simpler case I tested, but this looks much more impressive!

[mr-1993]

Thanks! These are massive improvements. Can you share a bit what shape the data has? I assume the larger each row is the bigger the performance difference gets. If I remember correctly, I saw a 2x improvement on some simpler case I tested, but this looks much more impressive!

Our dataset contains 500 rows with: 12 columns that contain strings or integers, and 11 columns that contain arrays. 6 of these arrays are one-dimensional and 5 are two-dimensional. They have varying sizes, the largest of the one-dim arrays contain around 270 elements, the largest two-dim arrays contain around 15x270 entries. The file itself is 2.5MB large.

[jaheba]

@mr-1993 I went ahead and implemented the changes myself. Hope you don't feel overlooked!

Thanks a lot for the PR 🎉

[lostella]

Thanks @mr-1993 @emptymalei !

[mr-1993]

@jaheba @lostella Thank you very much for the small collaboration! Hope to contribute other things as well! Also, thank you for implementing the last changes and sorry for our delay in that regard...we were blocked last week.