Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fondant base image #801

Merged
merged 31 commits into from
Jan 25, 2024
Merged

Add fondant base image #801

merged 31 commits into from
Jan 25, 2024

Conversation

mrchtr
Copy link
Contributor

@mrchtr mrchtr commented Jan 19, 2024

The PR contains the docker file for the base image and adjusts the code to use the base image for the python components.

Up for a better naming of the base image. Currently I've used fndnt/fondant-base:fondant_version-python_version,
e.g. fndnt/fondant-base:0.9.0-python3.10

Fix #782

docker/Dockerfile Outdated Show resolved Hide resolved
Copy link
Contributor

@PhilippeMoussalli PhilippeMoussalli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Matthias!

Do we also plan on providing base fondant pytorch images for CUDA components? I guess there we'll have to make more limited choices (which pytorch version, python versions are limited)

https://github.com/cnstark/pytorch-docker

docker/Dockerfile Outdated Show resolved Hide resolved
src/fondant/pipeline/lightweight_component.py Outdated Show resolved Hide resolved
@mrchtr
Copy link
Contributor Author

mrchtr commented Jan 22, 2024

Do we also plan on providing base fondant pytorch images for CUDA components? I guess there we'll have to make more limited choices (which pytorch version, python versions are limited)

I thought about this. I guess we don't need them right now. It can be quite complex and I think when people start using the GPU we should recommend building a custom component the old way. This components won't be lightweight neither.

@PhilippeMoussalli
Copy link
Contributor

Do we also plan on providing base fondant pytorch images for CUDA components? I guess there we'll have to make more limited choices (which pytorch version, python versions are limited)

I thought about this. I guess we don't need them right now. It can be quite complex and I think when people start using the GPU we should recommend building a custom component the old way. This components won't be lightweight neither.

alright makes sense! Some tests still to be failing though

Copy link
Contributor

@PhilippeMoussalli PhilippeMoussalli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks :)

@mrchtr
Copy link
Contributor Author

mrchtr commented Jan 23, 2024

Tested the build process in one of the latest commits.
Successfully executed run here: https://github.com/ml6team/fondant/actions/runs/7626293892/job/20772373591?pr=801

Copy link
Collaborator

@GeorgesLorre GeorgesLorre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Looking very good!

images/Dockerfile Outdated Show resolved Hide resolved
src/fondant/pipeline/lightweight_component.py Outdated Show resolved Hide resolved
tests/pipeline/test_python_component.py Show resolved Hide resolved
@GeorgesLorre
Copy link
Collaborator

Btw I tested the base image in a local pipeline and it worked fine!

@lightweight_component(
    base_image="fndnt/fondant-base:dc8d9b208802ee85796238ff9a903c8bf66906e8-python3.8"
)
class FeatOne(PandasTransformComponent):
    def transform(self, dataframe: pd.DataFrame) -> pd.DataFrame:
        dataframe["one"] = dataframe["x"].map(lambda x: x + 1)
        return dataframe

Copy link
Collaborator

@GeorgesLorre GeorgesLorre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! thx

How come I only see 1 tag on docker hub ?

Copy link
Member

@RobbeSneyders RobbeSneyders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mrchtr!

Left some small comments, but looks good!

For the name of the images, I would propose just fondant, and I would abbreviate the python versions to eg. py3.11.

src/fondant/pipeline/lightweight_component.py Outdated Show resolved Hide resolved
src/fondant/pipeline/lightweight_component.py Outdated Show resolved Hide resolved
@mrchtr mrchtr merged commit 7356b6a into main Jan 25, 2024
11 checks passed
@mrchtr mrchtr deleted the feature/build-fondant-base-image branch January 25, 2024 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add CI/CD to build default Fondant base image
4 participants