Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create Intermediate Representation Between arroyo-sql::SqlOperator and arroyo-datastream::Program #66

Closed
jacksonrnewhouse opened this issue Apr 19, 2023 · 1 comment
Assignees
Labels
sql Related to the DataFusion SQL integration

Comments

@jacksonrnewhouse
Copy link
Contributor

Currently this is translated in a single call to get_program_from_operator().

However, this code path is overloaded, responsible for code generation, creation of the operator graph, edge construction and several in-lined optimizations. This results in unclear, complex code.

For this issue, the task is to define another representation, SqlPlanGraph which can be more directly translated to the Program.

Before

[SqlOperator] - (graph construction, code gen, optimization) -> [Program].

After

[SqlOperator] -(graph construction) -> [SqlPlanGraph] - (optimizations) -> [SqlPlanGraph] - (code gen) -> [Program]

@jacksonrnewhouse jacksonrnewhouse self-assigned this Apr 19, 2023
@jacksonrnewhouse jacksonrnewhouse added the sql Related to the DataFusion SQL integration label Apr 19, 2023
@jacksonrnewhouse
Copy link
Contributor Author

Finished with #80.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sql Related to the DataFusion SQL integration
Projects
None yet
Development

No branches or pull requests

1 participant