Optimizing my Textual project

[grvstick]

App Summary

• The app reads chunks of data every 100ms and optionally 200ms from serial communication
• The serial comm library is battle-tested and ran against a lot of GUI options. Including web like using websockets, tkinter, etc. So It is pretty sure it is not blocking. It uses pyserial-asyncio
• 100ms data is updated into 6 Label widgets, each using query_one method with id selector and expected_types.
• 200ms data has variable length, but with fixed columns. This is stored in Datatable
• There are over 200 parameters that can be updated on-the-fly with this HW. This parameter is shown in separate pane in TabbedContent, with 3 Label widgets and a Input widget to change the value for the parameter

Issue

The app was not really responsive while and after creating around 800 widgets for the parameters.

Profiling

I ran pyInstruments as Will suggested and it turns out query_one method was the bottleneck.

Troubleshotting

• It is documented that updating widgets within workers is not optimal. I switched to sending Message from workers and handled updates within the message receiver. (Small improvement)
• Using query_one from the direct parent of the widget, instead of App, improved performance(60% of total app runtime -> 40%)
• Ditching query_one and caching label widgets as variables for direct access greatly improved performance(60% -> 5%). While this is not recommended, it may be worth using in hot loops with widgets that are not being removed and mounted again.
• When mounting over 800 widgets dynamically, mount_all instead of mount with for loop greatly improved performance. Also, removing widgets using remove_children("*") greatly improved performance as well.

Remaining Issue

• 200 parameters are being shown on VertialScroll container. This lags once in a while during scrolling.

[willmcgugan]

Thanks for the detailed analysis. Interesting that query_one is a bottleneck. I think we can do something to improve that.

Can you share your project? We might be able to offer more suggestions.

[grvstick]

Could you give me a brief breakdown on how query_one works? or if there is some material already, a pointer would be helpful

[willmcgugan]

It recursively checks each widget in the DOM until it finds a match.

There are some recent optimizations in main that will speed that up.

You may also be able to get some wins by reducing the scope of the query. if you do app.query_one it will traverse the entire app. If you query from a widget, it will only look at the children, which is likely faster.

[grvstick]

Any plans for applying algos in these queries? I'm not an expert in this topic but hashmap or tree search algos are known to be good in searching.

I'm sure you have spent some efforts on optimizing this but just giving out random ideas.

[willmcgugan]

It is a tree search, and we cache with a hash map. Try v0.69.0 (just released) which is faster.

[grvstick]

I've just tried it out, and pyinstruments does not even show query_one in expanded view. This would improve things a lot!
Just to be sure, it is v0.79.0, not v0.69.0 right?

[willmcgugan]

#4950