optional memory usage optimization and show mem usage #437
Conversation
There was a problem hiding this comment.
So I think this is great idea. As a way to make the Instances popup not take longer to load. I'm going to adjust this code after I merge it so that I load the "Memory" column separately and you'll see a "spinner" in that column value until its populated.
I also think this presents an interesting option to build out some UI for allowing users to "Optimize" data they have already loaded and do a memory comparison before accepting the optimization. Nice work!
| num_unique_values = len(df[col].unique()) | ||
| num_total_values = len(df[col]) | ||
| if num_unique_values / num_total_values < 0.5: | ||
| df[col] = df[col].astype("category") |
There was a problem hiding this comment.
I was always told to use .iloc, but I heard differing things. Do you know which is more correct?
| df[col] = df[col].astype("category") | |
| df.loc[:, col] = df[col].astype("category") |
There was a problem hiding this comment.
Thanks @Centropy-io 🙏 I remember pandas throwing warnings at one point in time if you didnt use iloc because of setting values on a copy, but in this instance it doesnt apply.
Allow memory usage optimizaton of large dataframes by converting object columns to Category type (https://www.dataquest.io/blog/pandas-big-data/)
show instance memory usage in Instances pop up (A bit slower to open due to calculating deep mem usage)