Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tracking: stream values executor #7857

Closed
5 of 7 tasks
Tracked by #7854
st1page opened this issue Feb 13, 2023 · 1 comment
Closed
5 of 7 tasks
Tracked by #7854

tracking: stream values executor #7857

st1page opened this issue Feb 13, 2023 · 1 comment
Assignees
Milestone

Comments

@st1page
Copy link
Contributor

st1page commented Feb 13, 2023

the old design does not consider the failover very well, waiting for a new design

We used to think users do not need stream values in streaming queries. But now we found it: #7854
Now we can not support the constant Rows in our streaming execution because we have not supported StreamValuesExecutor.

@BugenZhao
Copy link
Member

BugenZhao commented Mar 16, 2023

  • Rewrite LogicalValues to StreamBarrierRecv + StreamValues + StreamRowIdGen in the optimizer.

Note: this part might be tricky, as after we attach the RowIdGen node to the plan, the schema of the Values is changed. So this should be done in logical_rewrite_for_stream and maybe we also need to introduce the logical one of RowIdGen.

  • Streaming Values executor.

This can simply behave like a trivial Chain executor which embeds the expression evaluation, that is, we evaluate the expression in a blocking manner and output all of the results during the first epoch. It's worth noting that...

  • like Chain, if we find that we're on fail-over, we should not output again
  • like Chain, this should also be treated as a part of the executors that are tracked for the creating materialized view progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants