-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Split general component behavior from user implementation #257
Labels
Core
Core framework
Comments
RobbeSneyders
moved this from Ready for development
to In Progress
in Fondant development
Jul 4, 2023
2 tasks
github-project-automation
bot
moved this from Validation
to Done
in Fondant development
Jul 18, 2023
satishjasthi
pushed a commit
to satishjasthi/fondant
that referenced
this issue
Jul 21, 2023
This PR follows up on the PoC presented in ml6team#268 --- Fixes ml6team#257 It splits the implementation and execution of components, this has some advantages: - Pandas components can use `__init__` instead of setup, which is probably more familiar to users - Other components can use `__init__` as well instead of receiving all arguments to their transform or equivalent method, aligning implementation of different component types - Component implementation and execution should be easier to test separately I borrowed the executor terminology from KfP. --- Fixes ml6team#203 Since I had to update all the components, I also switched some of them to subclass `PandasTransformComponent` instead of `DaskTransformComponent`. --- These changes open some opportunities for additional improvements, but I propose to tackle those as separate PRs as this PR is already quite huge due to all the changes to the components. - [ ] ml6team#300 - [ ] ml6team#301
Hakimovich99
pushed a commit
that referenced
this issue
Oct 16, 2023
This PR follows up on the PoC presented in #268 --- Fixes #257 It splits the implementation and execution of components, this has some advantages: - Pandas components can use `__init__` instead of setup, which is probably more familiar to users - Other components can use `__init__` as well instead of receiving all arguments to their transform or equivalent method, aligning implementation of different component types - Component implementation and execution should be easier to test separately I borrowed the executor terminology from KfP. --- Fixes #203 Since I had to update all the components, I also switched some of them to subclass `PandasTransformComponent` instead of `DaskTransformComponent`. --- These changes open some opportunities for additional improvements, but I propose to tackle those as separate PRs as this PR is already quite huge due to all the changes to the components. - [ ] #300 - [ ] #301
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently users implement components by subclassing one of the provided component classes, which themselves provide a lot of general component behavior, such as data loading, validating, and writing.
Instead, we could split this by letting the user implement their components by implementing a class with a specific interface which is passed to a general component class.
Current:
Future:
We can still provide an interface which the user can subclass, for instance using a Protocol.
This has some advantages:
__init__
instead ofsetup
which will be more familiar to usersThe text was updated successfully, but these errors were encountered: