Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transfers optimize #1503

Merged
merged 2 commits into from
Sep 5, 2022
Merged

Transfers optimize #1503

merged 2 commits into from
Sep 5, 2022

Conversation

aalan3
Copy link
Contributor

@aalan3 aalan3 commented Sep 5, 2022

This PR should bring transfers time filtering to the same speed as the rest of the tables.

The reason why it doesn't work atm is because there's a date_trunc in the where. We don't need to truncate the date there, we can just pass the value.

Thanks to @jkylling for pointing this out and solving this!

@aalan3 aalan3 requested a review from soispoke September 5, 2022 14:45
@jkylling
Copy link
Contributor

jkylling commented Sep 5, 2022

Just a note about why the date_trunc breaks the query: Spark does not know much about the date_trunc('hour', -) function, and for the planner it is likely equivalent to writing random(). Hence, the query planner needs to evaluate random() > now() - interval 2 days for each row in the table, while when the query is rewritten to block_time > now() - interval 2 days, it knows that most files which make up the table does not contain any row which could match this predicate.

Copy link
Contributor

@soispoke soispoke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh interesting, thanks @aalan3 and @jkylling !

@soispoke
Copy link
Contributor

soispoke commented Sep 5, 2022

Can this be applied to erc721 and erc1155 transfers as well ?

@aalan3
Copy link
Contributor Author

aalan3 commented Sep 5, 2022

Can this be applied to erc721 and erc1155 transfers as well ?

It can but those tables are not yet z-ordered but we can still change the queries. I can follow up with PR after this one.

@aalan3 aalan3 merged commit aaf3caa into main Sep 5, 2022
@aalan3 aalan3 deleted the transfers-optimize branch September 5, 2022 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants