Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Windowing] Rewrite window function implementation to use real SQLite windows #20

Merged
merged 6 commits into from
Feb 20, 2024

Conversation

ohaibbq
Copy link

@ohaibbq ohaibbq commented Feb 15, 2024

This drastically improves the performance of complex queries that use window functions by no longer requiring duplicate subqueries and delegating sorting / partitioning to SQLite directly.

For one of our ingest views, the compiled query goes from ~217,000 characters with 272 SELECT statements down to ~117,000 characters with 78 SELECT statements.

The numbering functions such as RANK(), DENSE_RANK(), CUME_DIST() implementations are nearly identical to both SQLite's and Postgres' internal implementation.

SQLite window functions are implemented using the following interface:

Callback Description
xStep It is invoked to add a row to the current window. The function arguments, if any, corresponding to the row being added are passed to the implementation of xStep.
xFinal It is invoked to return the current value of the aggregate (determined by the contents of the current window), and to free any resources allocated by earlier calls to xStep.
xValue This method is only required for window aggregate functions. The presence of this method is what distinguishes a window aggregate function from a legacy aggregate function. This method is invoked to return the current value of the aggregate. Unlike xFinal, the implementation should not delete any context.
xInverse This method is only required for window aggregate functions, not legacy aggregate function implementations. It is invoked to remove the oldest presently aggregated result of xStep from the current window. The function arguments, if any, are those passed to xStep for the row being removed.

Closes Recidiviz/recidiviz-data#20742
Closes Recidiviz/recidiviz-data#20739
Closes Recidiviz/recidiviz-data#20740
Closes Recidiviz/recidiviz-data#20751
Closes Recidiviz/recidiviz-data#20773
Closes Recidiviz/recidiviz-data#20754
Closes Recidiviz/recidiviz-data#27338

Copy link

@ageiduschek ageiduschek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✨ 🐎 ✨

@ohaibbq ohaibbq merged commit ab50fe1 into candidate/rb20240201 Feb 20, 2024
7 checks passed
@ohaibbq ohaibbq deleted the dan/new-window-impl branch March 28, 2024 18:12
ohaibbq added a commit that referenced this pull request Jun 29, 2024
… windows (#20)

* [Windowing] Rewrite window function implementation to use real SQLite windows

* remove wip

* update fork

* lint

* more lint

* use recidiviz fork

fix window err rebase
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants