Add kedro package as a CI check

[ElenaKhaustova]

Motivation and Context

Solves kedro-org/kedro#3209

How has this been tested?

1. make install-test-requirements
2. behave - to run all tests or behave -n <Scenario> to run specific test

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Assigned myself to the PR
• Added tests to cover my changes

[ankatiyar]

Do you mind also adding this in the github actions workflow - https://github.com/kedro-org/kedro-starters/blob/main/.github/workflows/all-checks.yml so it runs in the CI

[ElenaKhaustova]

Do you mind also adding this in the github actions workflow - https://github.com/kedro-org/kedro-starters/blob/main/.github/workflows/all-checks.yml so it runs in the CI

ah, yes, sure

[astrojuanlu]

Wondering if there's an easier way to setup Hadoop on Windows, maybe through conda (or better, micromamba)? According to conda-forge/pyspark-feedstock#34 (comment), -c conda-forge pyspark does not depend on Hadoop, and when I did tests with @ravi-kumar-pilla , an environment.yml like this was enough:

name: kedro-pyspark
channels:
- conda-forge
dependencies:
- findspark
- openjdk
- pyspark
- python=3.11

The fact that you managed to get Hadoop.exe installed through PowerShell is truly impressive, but I doubt Windows users would go that far, and therefore it would maybe be more useful if the way we do things in our CI is representative of how users get to install Kedro.

(Not to mention that I highly doubt anybody is trying to use Kedro + PySpark on Windows natively, but that's another story)

[astrojuanlu]

(I don't intend to block the PR over this though, it was more of an informative comment - "if it works, don't touch it")

[ElenaKhaustova]

(I don't intend to block the PR over this though, it was more of an informative comment - "if it works, don't touch it")

Since we use that approach to run other e2e tests as well, maybe we should address this in a separate PR?

kedro-starters/.github/workflows/all-checks.yml

Line 27 in dc60fc3

run-tests:

[ankatiyar]

LGTM!
I think starters CI needs an update anyway (also need to update actions versions) we can do it separately.

[ElenaKhaustova]

@astrojuanlu, @ankatiyar - separate issue opened for updating the whole CI for starters: #237

[astrojuanlu]

Dropped the Windows + Hadoop comment because I was passing by but please always take my comments on CI and infra work as suggestions and with a full bucket of salt 😊 Leaving the spot for others to do the final approval 👍🏼

[merelcht]

LGTM! 👍