Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable substream slicers and Cartesian product slicers in connector builder #20835

Closed
sherifnada opened this issue Dec 22, 2022 · 3 comments · Fixed by #20861
Closed

Enable substream slicers and Cartesian product slicers in connector builder #20835

sherifnada opened this issue Dec 22, 2022 · 3 comments · Fixed by #20861
Assignees

Comments

@sherifnada
Copy link
Contributor

sherifnada commented Dec 22, 2022

Both substream slicers and cartesian product slicers should be supported in the UI.

Acceptance criteria

  • Substream slicer should be selectable from the list of slicing options
  • Parent stream should select reference to another stream in the UI
  • Circular dependencies should be caught
  • Cartesian product slicers should be selectable from the list of slicing options
  • Any slicers should be able to be added to the list of sub-slicers for cartesian product slicers
@sherifnada
Copy link
Contributor Author

@flash1293 to fill in the details of this ticket

@flash1293 flash1293 changed the title Enable substream slicers in connector builder Enable substream slicers and Cartesian product slicers in connector builder Dec 22, 2022
@flash1293
Copy link
Contributor

Implementation notes:

  • add stable id to each stream in the builder values
  • Use it to reference the parent stream
  • before mapping to the manifest, check for cycles in the references - if there are some, set an additional error state and hide the stream test panel like if the listing call fails
  • On building manifest, recursively inline stream definitions into each other

@flash1293
Copy link
Contributor

In review, however it's unclear how to handle some weird part about how substream slicers properties are managed on the CDK level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants