Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add typing check #50

Merged
merged 8 commits into from
Apr 27, 2023
Merged

Add typing check #50

merged 8 commits into from
Apr 27, 2023

Conversation

PhilippeMoussalli
Copy link
Contributor

@PhilippeMoussalli PhilippeMoussalli commented Apr 26, 2023

This PR adds type checking into the CICD pipelines of Fondant. We're leveraging the latest version of mypy.
#46

@PhilippeMoussalli PhilippeMoussalli linked an issue Apr 26, 2023 that may be closed by this pull request
@PhilippeMoussalli PhilippeMoussalli added the CI/CD Related to CI/CD of the repository label Apr 26, 2023
@PhilippeMoussalli PhilippeMoussalli self-assigned this Apr 26, 2023
@@ -239,7 +244,7 @@ def from_fondant_component_spec(
@staticmethod
def _dump_args(args: t.List[Argument]) -> t.List[t.Union[str, t.Dict[str, str]]]:
"""Dump Fondant specification arguments to kfp command arguments."""
dumped_args = []
dumped_args: t.List[t.Union[str, t.Dict[str, str]]] = []
Copy link
Contributor

@NielsRogge NielsRogge Apr 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tbh I find these typing hints pretty hard to read and I'm not sure they're helpful for us. They seem useful for computers but not for humans

t.List[t.Union[str, t.Dict[str, str]]] => I mean just look at this

Just my personal opinion :D

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I twas getting errors from mypy as follows
Incompatible return value type (got "List[str]", expected "List[Union[str, Dict[str, str]]]") which went away when I explicitly defined the type.
However, the args of kubeflow does required to have a list that can contains either a string or a dictionary so in that sense I think the type hint in the function's return is valid.

I'll just add a mypy exception for that one

Copy link
Member

@RobbeSneyders RobbeSneyders Apr 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The typehints are meant to be useful for computers though, that's why we are adding mypy :)

If we have complex types that are reused across Fondant, we can create aliases for them. Eg:
https://github.com/encode/starlette/blob/master/starlette/types.py

I would rather have a correct complex type than ignore the issue. Defining the types helps catch bugs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alright I think that's a better approach indeed :) thanks for the suggestion. Added


spec_data = pkgutil.get_data("fondant", "schemas/manifest.json")

if spec_data is not None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, but the following is a bit easier to follow:

spec_data = pkgutil.get_data("fondant", "schemas/manifest.json")

if spec_data is None:
    raise FileNotFoundError("component_spec.json not found in fondant schema")

spec_str = spec_data.decode("utf-8")
spec_schema = json.loads(spec_str)

Copy link
Member

@RobbeSneyders RobbeSneyders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @PhilippeMoussalli!

I would revert ignoring the args type, but otherwise LGTM.

@PhilippeMoussalli PhilippeMoussalli merged commit a8258af into main Apr 27, 2023
@RobbeSneyders RobbeSneyders deleted the add-typing-check branch May 4, 2023 07:34
Hakimovich99 pushed a commit that referenced this pull request Oct 16, 2023
This PR adds type checking into the CICD pipelines of Fondant. We're
leveraging the latest version of mypy.
#46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI/CD Related to CI/CD of the repository
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add typing check in Github actions
3 participants