Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Output transformer #82

Closed
goodwanghan opened this issue Oct 24, 2020 · 0 comments · Fixed by #97
Closed

[FEATURE] Output transformer #82

goodwanghan opened this issue Oct 24, 2020 · 0 comments · Fixed by #97

Comments

@goodwanghan
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
NA

Describe the solution you'd like
Currently Fugue transformers must return something. Sometimes we just want to distributedly process something and then stop. For example, for a big dataframe we can partition by certain keys, then we can use a transformer to serialize and save each partition (in its own way).

For the current design, we need to transform and persist to trigger this execution. It works fine but concept wise, it's a hack. We need to have a clear definition of this as a new type of extension, maybe OutputTransformer?

And we need to add programming and sql interface, and interfaceless for it. Also update the doc.

Describe alternatives you've considered
NA

Additional context
NA

@goodwanghan goodwanghan added this to the 0.4.5 milestone Oct 24, 2020
@goodwanghan goodwanghan linked a pull request Nov 4, 2020 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant