DataTable.add_rows() takes weirdly long time

[leemthai]

I'm trying to figure out why add_rows() in DataTable appears to take much longer than code around it.

The headline timing is that for 40 rows (10 columns a row), it is taking 80ms just to perform add_rows() i.e. 2ms per row. Even without testing code around it for speed, that seems pretty slow. It's almost 1/10s just to add_rows() for 40 rows. This "2ms per row" number scales up, as well. It takes 2ms per row even with many more rows.

Just to give some context about my local peformance, before calling add_rows() I do the following preparatory work:
a) sort 3500 rows of data (using my own sort, not datatable stuff)
b) generate the actual data to put into the rows (this involves all kind of formatting, creation of Text objects etc for each cell)

(a) and (b) together take just 4ms. Before doing any performance testing, I was guessing the preparatory work would have been the bottleneck. It just doesn't "feel" right.

Can someone offer any insight into what code in add_rows() might be taking a long time, or how I can profile it myself? add_rows() performance is kind of crucial to my app :)

Thanks all!

[leemthai]

I have done some profiling and managed to hopefully discover something of interest. In add_row() there are the following two lines:

self._require_update_dimensions = True
self.cursor_coordinate = self.cursor_coordinate

If you remove them, add_row() starts taking 0.01ms instead of 2.5ms.

On searching for 'self._require_update_dimensions' I found this function:
async def _on_idle(self, _: events.Idle) -> None:

which has this code:

if self._require_update_dimensions:
    self._require_update_dimensions = False
    new_rows = self._new_rows.copy()
    self._new_rows.clear()
    self._update_dimensions(new_rows)

So is the issue that if my code is just doing add_row in a loop (or add_rows() because add_rows() calls add_row() in a loop), we end up calling on_idle repeatedly with self._require_update_dimensions=True, and hence run the copy and update_dimensions code for every row?

Thoughts appreciated :)
Cheers,

[willmcgugan]

The on idle should only get called when there are no more messages.

Can we see the code you are using to populate the table?

[leemthai]

It's just this at the moment:
cointable_obj.add_rows(self.master_table)
i.e. I have all the data pre-prepared in a big table.

I have a 2nd version which does rows individually:

for idx, row in enumerate(self.master_table):
   cointable_obj.add_row(*row, label=Text(str(idx), style = "italic", justify="right"))

But issues seem just the same because it looks like add_rows() just calls add_row in a loop anyway

[leemthai]

Yeah. I am not sure at all on idle is being called. Probably a red herring. The only 'fact' I have rn is removing:

self._require_update_dimensions = True
self.cursor_coordinate = self.cursor_coordinate

speeds everything right back up so we can start from those two lines I guess.

It's actually 0.42ms to run add_rows() without these lines, 80ms with them in :)