You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
NA
Describe the solution you'd like
Currently Fugue transformers must return something. Sometimes we just want to distributedly process something and then stop. For example, for a big dataframe we can partition by certain keys, then we can use a transformer to serialize and save each partition (in its own way).
For the current design, we need to transform and persist to trigger this execution. It works fine but concept wise, it's a hack. We need to have a clear definition of this as a new type of extension, maybe OutputTransformer?
And we need to add programming and sql interface, and interfaceless for it. Also update the doc.
Describe alternatives you've considered
NA
Additional context
NA
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
NA
Describe the solution you'd like
Currently Fugue transformers must return something. Sometimes we just want to distributedly process something and then stop. For example, for a big dataframe we can partition by certain keys, then we can use a transformer to serialize and save each partition (in its own way).
For the current design, we need to transform and persist to trigger this execution. It works fine but concept wise, it's a hack. We need to have a clear definition of this as a new type of extension, maybe
OutputTransformer
?And we need to add programming and sql interface, and interfaceless for it. Also update the doc.
Describe alternatives you've considered
NA
Additional context
NA
The text was updated successfully, but these errors were encountered: