-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Query dependency graph #510
Labels
projects
Project ideas for Mosaic
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This builds on #388. The idea here is that Mosaic sends queries to a backend for execution but can only run so many queries simultaneously. With fine grained dependencies, we could increase parallelism and potentially eliminate superfluous queries.
We have to be careful about dependencies so that visualizations don't show stale data. As an example, imagine a dashboard with three charts A, B, and C. The user brushes in A which updates B and C. The brush changes create three queries b_1, b_2, b_3 to update B and c_1, c_2, c_3 to update C. Imagine these come in as b_1, c_1, b_2, c_2, b_3, c_3 and the backend can only run two queries at a time. Right now, Mosaic only resolves queries in the same order they were queued even though b_1 and c_1 could resolve in any order without causing issues. b_2 and c_2 do not really need to be sent to the backend if we already know their results will be superseded by b_3 and c_3 so if we can keep track of that as well, we can basically drop the queries from the queue. There are many more dependencies such as queries that create new tables (such as indexes) which need to run in the correct order. For now (even with #475) exec queries never run in parallel since we don't know whether there is a dependency between them.
This could be related to #461 since we can queue up and drop superseded queries when the chart comes into view again.
The text was updated successfully, but these errors were encountered: