[SimEngine][Streamlit] - Tech Spike - Update SimEngine Streamlit plots to support key UX Use Cases

[idiom-bytes]

Background / motivation

As laid out in pdr-private #87 we are going to tech spike a JS app that provides the same/similar plots as the current SimEngine Plots for Streamlit.

However, in order to compare apples to apples, we need to test against the same "UX Use Cases".
These "UX Use Cases" will be replicated across Streamlit, JS, and PyQT implementations.

In order to validate these, we may need to update the current plots to support this functionality.

TODOs / DoD

• Utilize the DuckDB node.js lib to connect to DuckDB and access the SimEngine data that should be getting generated via a continuous-running background process
• Update plots and web components, such that when the user interacts with a plot (1), other plots (2)(3) or web components (4) like a text field/table, can respond to the event and update as needed.

UX Use Cases for Tech Spike Comparison

• UX Use Case 1 - Select a data point in a scatter plot, and have other plots/ui-components update in an immediate, responsive manner
• UX Use Case 2 - Drag + Select an area from a scatter plot, and have other plots/ui-components update in an immediate, responsive manner

Related github issues

• pdr-private #87

[trentmc]

@calina-c what parts of this are done, versus pushed back for later? Will be good to update the issue description with the actual status. Thanks.

[calina-c]

@trentmc

Utilize the DuckDB node.js lib to connect to DuckDB and access the SimEngine data that should be getting generated via a continuous-running background process
this was discarded due to the concurrency issues with DuckDB

the rest of the use cases described here are related to inter-chart reactivity. During one of our calls, we discussed various use cases which I think are relevant, i.e. variable selection from a list, selecting a time period instead of a specific point in time etc. [input-output separation] and inter-chart reactivity [input-output reactivity]. We didn't decide which would be the order of implementation of these categories and specific implementations. Do you think we can go over the options and prioritise them as needed?

I would suggest starting with variable selection, which is easy, has I/O separation and doesn't affect the real-time updates of charts, i.e. you can do this WHILE running the simulation.

[trentmc]

Do you think we can go over the options and prioritise them as needed?

Here's an approximate priority:

1. for predictoor bots: build all the key predictoor analytics plots. Including predictoor accuracy; predictoor f1/precision/recall; distribution of predictoor profits
2. in sim: var importance plot: click on a bar (for a var), and show response vs. var.
3. in predictoor bots: (ideally) plot sim var importance plot, response surface contours & 1d. This won't be easy, because that data isn't available on chain, so we need to expose it somehow for the analytics tool to see. (With proper privacy: I can only see that info for my predictoor bot, and any other bot that has shared with me.)
4. in predictoor analytics var. importance plot: click on a bar (for a var), and show response vs. var.

[trentmc]

The main part of this ticket is done.

Follow-up issue is #906