Add getting started documentation #250

GeorgesLorre · 2023-06-29T13:05:59Z

No description provided.

GeorgesLorre · 2023-06-29T13:07:35Z

docs/getting_started.md

+)
+
+my_pipeline.add_op(load_from_hf_hub, dependencies=[])
+```


Not sure if it is the best example since it seems quite complex

We might just have to explain it a bit better, as it is the most useful component to easily load some data to get started. Not sure if we want to describe this as a third kind of component, which is a "generic" component.

docs/getting_started.md

RobbeSneyders

Thanks @GeorgesLorre! don't forget to add the file to the documentation index.

docs/getting_started.md

RobbeSneyders · 2023-06-29T14:09:04Z

docs/getting_started.md

+)
+
+my_pipeline.add_op(load_from_hf_hub, dependencies=[])
+```


We might just have to explain it a bit better, as it is the most useful component to easily load some data to get started. Not sure if we want to describe this as a third kind of component, which is a "generic" component.

RobbeSneyders · 2023-06-29T14:11:17Z

docs/getting_started.md

+
+## Running your pipeline
+
+A Fondant pipeline needs to be compiled before it can be ran. This means translating the user friendly Fondant pipeline definition into something that can be executed by a runner.


I would leave out the compilation step here and only explain the combined functionality which you get when using fondant run.

RobbeSneyders · 2023-06-29T14:12:40Z

pyproject.toml

+kfp = { version = ">= 1.8.19"}

-kfp = { version = ">= 1.8.19", optional = true }
 kubernetes = { version = ">= 18.20.0", optional = true }
 pandas = { version = ">= 1.3.5", optional = true }

 [tool.poetry.extras]
-pipelines = ["kfp", "kubernetes"]
+pipelines = ["kubernetes"]


What's the reason for these changes? kfp is quite a big dependency which you don't need when installing fondant in a component, which is why we added it as an optional dependency to the pipeline extra.

kfp is used by the pipeline.py so it is needed for fondant compile and fondant run

Yes, so then the user needs to install fondant[pipelines] (fondant[kfp] might be better). But we don't want to install this in every component.

mrchtr · 2023-06-29T14:38:58Z

docs/getting_started.md

+
+Note that if you use a local `base_path` in your pipeline declaration that this path will be mounted in the docker containers. This means that the data will be stored locally on your machine. If you use a cloud storage path, the data will be stored in the cloud.
+
+Now that we have compiled our pipeline, we can run it:


Maybe we should mentioned that the docker images are stored in the Github container registry. I didn't pulled images from there before which leads me into an additional docker login step.
Basically following the steps here: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry

Do we need to login when the images are public ? Or only for the private ones ?

RobbeSneyders

Thanks @GeorgesLorre, some minor comments, but otherwise looks good!

RobbeSneyders · 2023-07-03T12:00:24Z

README.md

@@ -85,6 +85,11 @@ Eg. generating logos:

 <p align="right">(<a href="#chocolate_bar-fondant">back to top</a>)</p>

+## 💨 Getting Started


I would move this before the example pipelines

RobbeSneyders · 2023-07-03T12:02:10Z

components/image_resolution_extraction/src/main.py

-            dataframe[[("images", "data")]].map(extract_dimensions)
+        dataframe[[("images", "width"), ("images", "height")]] = dataframe[
+            [("images", "data")]
+        ].apply(lambda x: extract_dimensions(x.iloc[0]), axis=1)


Can we use the column name here instead of the index? That would make it easier to understand.

RobbeSneyders · 2023-07-03T12:03:22Z

docs/getting_started.md

+
+my_pipeline = Pipeline(
+    pipeline_name='my_pipeline',
+    base_path='/home/username/my_pipeline', <--- Make sure to update this


Suggested change

base_path='/home/username/my_pipeline', <--- Make sure to update this

base_path='/home/username/my_pipeline', # TODO: update this

RobbeSneyders · 2023-07-03T12:04:13Z

docs/getting_started.md

+
+Now that we have a pipeline, we can add components to it. Components are the building blocks of your pipeline. They are the individual steps that will be executed in your pipeline. There are 2 main types of components:
+
+- reusable components: These are components that are already created by the community and can be easily used in your pipeline. You can find a list of reusable components [here](https://github.com/ml6team/fondant/tree/main/components). They often have arguments that you can set to configure them for your use case.


Suggested change

- reusable components: These are components that are already created by the community and can be easily used in your pipeline. You can find a list of reusable components [here](https://github.com/ml6team/fondant/tree/main/components). They often have arguments that you can set to configure them for your use case.

- **reusable components**: These are components that are already created by the community and can be easily used in your pipeline. You can find a list of reusable components [here](https://github.com/ml6team/fondant/tree/main/components). They often have arguments that you can set to configure them for your use case.

RobbeSneyders · 2023-07-03T12:04:27Z

docs/getting_started.md

+
+- reusable components: These are components that are already created by the community and can be easily used in your pipeline. You can find a list of reusable components [here](https://github.com/ml6team/fondant/tree/main/components). They often have arguments that you can set to configure them for your use case.
+
+- custom components: These are the components you create to solve your use case. A custom component can be easily created by adding a `fondant_component.yaml`, `dockerfile` and `main.py` file to your component subdirectory. The `fondant_component.yaml` file contains the specification of your component. You can find more information about it [here](https://github.com/ml6team/fondant/blob/main/docs/component_spec.md)


Suggested change

- custom components: These are the components you create to solve your use case. A custom component can be easily created by adding a `fondant_component.yaml`, `dockerfile` and `main.py` file to your component subdirectory. The `fondant_component.yaml` file contains the specification of your component. You can find more information about it [here](https://github.com/ml6team/fondant/blob/main/docs/component_spec.md)

- **custom components**: These are the components you create to solve your use case. A custom component can be easily created by adding a `fondant_component.yaml`, `dockerfile` and `main.py` file to your component subdirectory. The `fondant_component.yaml` file contains the specification of your component. You can find more information about it [here](https://github.com/ml6team/fondant/blob/main/docs/component_spec.md)

RobbeSneyders · 2023-07-03T12:08:10Z

docs/getting_started.md

+        logger.info("Filtering dataset...")
+
+        dataframe[[("images", "width"), ("images", "height")]] = \
+            dataframe[[("images", "data")]].apply(lambda x:extract_dimensions(x.iloc[0]), axis=1)


Same comment as above.

RobbeSneyders

LGTM!

Add first part of getting started documentation

cbc2b1f

GeorgesLorre requested review from mrchtr, RobbeSneyders and NielsRogge June 29, 2023 13:05

GeorgesLorre commented Jun 29, 2023

View reviewed changes

GeorgesLorre requested a review from shayorshay June 29, 2023 13:26

shayorshay reviewed Jun 29, 2023

View reviewed changes

docs/getting_started.md Outdated Show resolved Hide resolved

RobbeSneyders reviewed Jun 29, 2023

View reviewed changes

mrchtr reviewed Jun 29, 2023

View reviewed changes

GeorgesLorre added 3 commits June 29, 2023 17:09

Add more documentation on expanding your pipeline

75e6f4e

Add documentation for exploring

885304e

Add documentation for exploring

224e562

GeorgesLorre changed the title ~~Add first part of getting started documentation~~ Add getting started documentation Jul 3, 2023

RobbeSneyders reviewed Jul 3, 2023

View reviewed changes

Do some reordering

f5a3519

RobbeSneyders approved these changes Jul 3, 2023

View reviewed changes

GeorgesLorre merged commit 2009870 into main Jul 3, 2023

GeorgesLorre deleted the feature/getting-started-docs branch July 3, 2023 19:32

RobbeSneyders linked an issue Jul 4, 2023 that may be closed by this pull request

Create 'Getting Started' documentation #61

Closed

Hakimovich99 pushed a commit that referenced this pull request Oct 16, 2023

Add getting started documentation (#250)

11c09ab

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add getting started documentation #250

Add getting started documentation #250

GeorgesLorre commented Jun 29, 2023

GeorgesLorre Jun 29, 2023

RobbeSneyders Jun 29, 2023

RobbeSneyders left a comment

RobbeSneyders Jun 29, 2023

RobbeSneyders Jun 29, 2023

RobbeSneyders Jun 29, 2023

GeorgesLorre Jun 29, 2023

RobbeSneyders Jun 29, 2023

mrchtr Jun 29, 2023

GeorgesLorre Jun 29, 2023

RobbeSneyders left a comment

RobbeSneyders Jul 3, 2023

RobbeSneyders Jul 3, 2023

RobbeSneyders Jul 3, 2023

RobbeSneyders Jul 3, 2023

RobbeSneyders Jul 3, 2023

RobbeSneyders Jul 3, 2023

RobbeSneyders left a comment


		## Running your pipeline

		A Fondant pipeline needs to be compiled before it can be ran. This means translating the user friendly Fondant pipeline definition into something that can be executed by a runner.


		Note that if you use a local `base_path` in your pipeline declaration that this path will be mounted in the docker containers. This means that the data will be stored locally on your machine. If you use a cloud storage path, the data will be stored in the cloud.

		Now that we have compiled our pipeline, we can run it:

		@@ -85,6 +85,11 @@ Eg. generating logos:

		<p align="right">(<a href="#chocolate_bar-fondant">back to top</a>)</p>

		## 💨 Getting Started

	base_path='/home/username/my_pipeline', <--- Make sure to update this
	base_path='/home/username/my_pipeline', # TODO: update this


		Now that we have a pipeline, we can add components to it. Components are the building blocks of your pipeline. They are the individual steps that will be executed in your pipeline. There are 2 main types of components:

		- reusable components: These are components that are already created by the community and can be easily used in your pipeline. You can find a list of reusable components [here](https://github.com/ml6team/fondant/tree/main/components). They often have arguments that you can set to configure them for your use case.

	- reusable components: These are components that are already created by the community and can be easily used in your pipeline. You can find a list of reusable components [here](https://github.com/ml6team/fondant/tree/main/components). They often have arguments that you can set to configure them for your use case.
	- reusable components: These are components that are already created by the community and can be easily used in your pipeline. You can find a list of reusable components [here](https://github.com/ml6team/fondant/tree/main/components). They often have arguments that you can set to configure them for your use case.


		- reusable components: These are components that are already created by the community and can be easily used in your pipeline. You can find a list of reusable components [here](https://github.com/ml6team/fondant/tree/main/components). They often have arguments that you can set to configure them for your use case.

		- custom components: These are the components you create to solve your use case. A custom component can be easily created by adding a `fondant_component.yaml`, `dockerfile` and `main.py` file to your component subdirectory. The `fondant_component.yaml` file contains the specification of your component. You can find more information about it [here](https://github.com/ml6team/fondant/blob/main/docs/component_spec.md)

Add getting started documentation #250

Add getting started documentation #250

Conversation

GeorgesLorre commented Jun 29, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RobbeSneyders left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RobbeSneyders left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RobbeSneyders left a comment

Choose a reason for hiding this comment