Reduce the data used in the spaceflights tutorial and starters

[merelcht]

Description

The spaceflights starter gets used in demos and testing, it takes a considerate of time to run the pipeline. For example in the Kedro bootcamp demoing catalog.load("shuttles") takes like 15-20 seconds and is a bit awkward for demo purpose.

Context

See more details on the spaceflights project + tutorial in the docs: https://docs.kedro.org/en/stable/tutorial/spaceflights_tutorial.html

We're now creating all new starters based on spaceflights so all of these examples could benefit from a smaller dataset.

Related to: #2008

[astrojuanlu]

Now that we're reviewing the spaceflights data, I transferred #3110 to this repo

[laizaparizotto]

@merelcht You mean literally making the dataset smaller?

[merelcht]

@merelcht You mean literally making the dataset smaller?

Yes 🙂 It's just an example project to demonstrate how Kedro works, so we don't care much about model accuracy. This ticket is to try and reduce the size of all input datasets: companies, reviews and shuttles.

[laizaparizotto]

This should also be adressed at kedro-starters?

I saw the shuttles.xlsx file is located in 4 different paths:

• https://github.com/kedro-org/kedro-starters/blob/main/spaceflights/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/shuttles.xlsx
• https://github.com/kedro-org/kedro-starters/blob/main/spaceflights-pyspark/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/shuttles.xlsx
• https://github.com/kedro-org/kedro-starters/blob/main/spaceflights-pandas-viz/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/shuttles.xlsx
• https://github.com/kedro-org/kedro-starters/blob/main/spaceflights-pyspark-viz/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/shuttles.xlsx

To check if my understanding is correct, should changes be made to these .xlsx files in all these locations?

And the problem is related only toshuttles.xlsx or also companies.csv and reviews.csv?

[merelcht]

This should also be adressed at kedro-starters?

Yes this should indeed be done in kedro-starters.

I saw the shuttles.xlsx file is located in 4 different paths:

• https://github.com/kedro-org/kedro-starters/blob/main/spaceflights/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/shuttles.xlsx
• https://github.com/kedro-org/kedro-starters/blob/main/spaceflights-pyspark/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/shuttles.xlsx
• https://github.com/kedro-org/kedro-starters/blob/main/spaceflights-pandas-viz/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/shuttles.xlsx
• https://github.com/kedro-org/kedro-starters/blob/main/spaceflights-pyspark-viz/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/shuttles.xlsx

To check if my understanding is correct, should changes be made to these .xlsx files in all these locations?

I'd suggest doing the change first for just spaceflights and then when it's reviewed they can be replicated for the other spaceflights-... repositories. Is that okay? It will be faster to get it reviewed that way.

And the problem is related only toshuttles.xlsx or also companies.csv and reviews.csv?

Ideally all the datasets would be reduced in size if that's possible 🙂

[stichbury]

Hi @laizaparizotto Is this a ticket you are actively working on? Shall I assign it to you and mark as "in progress"?

[laizaparizotto]

Hi @stichbury, I will be able to work on it this weekend. If not a problem, yes, you can assign that to me :D.

[laizaparizotto]

I left a comment in the PR asking if I should also open a PR in this repo?