Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement]: Add cache for loading table when get table meta in unkeyed table #2296

Closed
3 tasks done
Tracked by #2176
HuangFru opened this issue Nov 13, 2023 · 3 comments · Fixed by #2298
Closed
3 tasks done
Tracked by #2176

[Improvement]: Add cache for loading table when get table meta in unkeyed table #2296

HuangFru opened this issue Nov 13, 2023 · 3 comments · Fixed by #2298

Comments

@HuangFru
Copy link
Contributor

Search before asking

  • I have searched in the issues and found no similar issues.

What would you like to be improved?

With a query in Trino, operations such as fetching metadata information require multiple loading tables. This can be affected by objective factors such as network latency, resulting in slower queries.

In the Iceberg format, this caching is already implemented (native Iceberg implementation). We have this cache for our Mix-format's keyed tables, and we should add this cache for our unkeyed tables as well. This cache is also applicable to queries on the base store and change store.

How should we improve?

Add a cache for loading tables.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

@HuangFru HuangFru changed the title [Improvement][Trino]: Add cache for loading table when get table meta in unkeyed table [Improvement]: Add cache for loading table when get table meta in unkeyed table Nov 13, 2023
@baiyangtx
Copy link
Contributor

The Mixed-Iceberg format has been refactored, and the refactored Mixed-Iceberg will directly use the Iceberg Catalog, making it possible to reuse the caching capability of the Iceberg Catalog. You can refer to this issue, #1336, to learn more about the refactored Mixed-Iceberg Catalog.

@baiyangtx
Copy link
Contributor

The core implementation of the refactored MixedIceberg Catalog will look like this: link to the code.

@HuangFru
Copy link
Contributor Author

The core implementation of the refactored MixedIceberg Catalog will look like this: link to the code.

Native Iceberg's Caching Catalog is now only implemented on Engines(Spark, Flink), but we can reuse it. Looks like implementing a caching catalog on the refactored Mixed-Iceberg Catalog is no harm, but I think the trino module's cache also should be implemented because loading the mix table not only loads the iceberg table but also has some other operations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants