Kedro does not play well with Poetry

[fkromer]

Description

I have created a virtual environment with Poetry into <kedro-project-root>/.venv/ and added Kedro as development dependency. If I try to run the Kedro CLI an exception is raised.

Context

This issue implies that you cannot use Kedro with Python 3.8 if you don't want to install Kedro and Kedro project dependencies into the system or user installation of Python. Kedro-docker does not support Python 3.8 yet.

Steps to Reproduce

1. Create venv: poetry config --local virtualenvs.in-project true, poetry init
2. If not done automatically (e.g. in case VSCode is used), activate venv: poetry shell
3. Add kedro as deve dependency: poetry add -D kedro==0.16.1
4. Check if kedro is installed successfully: poetry show | grep kedro
5. Run kedro CLI: kedro -V

Expected Result

Tell us what should happen.

Actual Result

$ kedro -V
/home/florian/.local/lib/python3.8/site-packages/kedro/context/__init__.py:42: DeprecationWarning: All the modules in `kedro.context` have been moved to `kedro.framework.context`, and `kedro.context` will be removed in Kedro 0.17.0. Please update import paths from `kedro.context` to `kedro.framework.context` in your Kedro project.
  warnings.warn(
/home/florian/.local/lib/python3.8/site-packages/kedro/cli/__init__.py:40: DeprecationWarning: All the modules in `kedro.cli` have been moved to `kedro.framework.cli`, and `kedro.cli` will be removed in Kedro 0.17.0. Please update import paths from `kedro.cli` to `kedro.framework.cli` in your Kedro project.
  warnings.warn(
Traceback (most recent call last):
  File "/home/florian/.local/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 619, in main
    kedro_cli = importlib.import_module("kedro_cli")
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/florian/gitlab/<kedro-project-root/kedro_cli.py", line 54, in <module>
    from kedro.context import KEDRO_ENV_VAR, load_context
ImportError: cannot import name 'KEDRO_ENV_VAR' from 'kedro.context' (/home/florian/.local/lib/python3.8/site-packages/kedro/context/__init__.py)
/usr/lib/python3/dist-packages/apport/report.py:13: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import fnmatch, glob, traceback, errno, sys, atexit, locale, imp
Traceback (most recent call last):
  File "/home/florian/.local/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 619, in main
    kedro_cli = importlib.import_module("kedro_cli")
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/florian/gitlab/kedro-project-root/kedro_cli.py", line 54, in <module>
    from kedro.context import KEDRO_ENV_VAR, load_context
ImportError: cannot import name 'KEDRO_ENV_VAR' from 'kedro.context' (/home/florian/.local/lib/python3.8/site-packages/kedro/context/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/florian/.local/bin/kedro", line 8, in <module>
    sys.exit(main())
  File "/home/florian/.local/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 623, in main
    _handle_exception(f"Cannot load commands from {kedro_cli_path}")
  File "/home/florian/.local/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 643, in _handle_exception
    raise KedroCliError(msg)
kedro.framework.cli.utils.KedroCliError: Cannot load commands from /home/florian/gitlab/<kedro-project-root/kedro_cli.py

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used (pip show kedro or kedro -V): kedro==0.16.1
• Python version used (python -V): Python 3.8.2
• Operating system and version: Ubuntu 20.04 LTS

[mzjp2]

Is this a fresh project? Looking at the error message, I can't see what the poetry specific thing here is and it looks like your project template contains references to things within kedro that no longer exist. Have you followed the migration guide in RELEASE.md? Specifically this section:

https://github.com/quantumblacklabs/kedro/blob/develop/RELEASE.md#migration-for-cli-and-kedro_env-environment-variable

[mzjp2]

With regards to plugin support of Python 3.8, that's following soon. Having kedro which is a dependency of a kedro plugin support Python 3.8 is a pre-requisite and that's just come out. The plugins are following very soon :)

[fkromer]

Is this a fresh project? Looking at the error message, I can't see what the poetry specific thing here is and it looks like your project template contains references to things within kedro that no longer exist. Have you followed the migration guide in RELEASE.md? Specifically this section:

https://github.com/quantumblacklabs/kedro/blob/develop/RELEASE.md#migration-for-cli-and-kedro_env-environment-variable

Thx a lot. I did not know about that. Looks like this should fix the issue.

[mzjp2]

Is this a fresh project? Looking at the error message, I can't see what the poetry specific thing here is and it looks like your project template contains references to things within kedro that no longer exist. Have you followed the migration guide in RELEASE.md? Specifically this section:
https://github.com/quantumblacklabs/kedro/blob/develop/RELEASE.md#migration-for-cli-and-kedro_env-environment-variable

Thx a lot. I did not know about that. Looks like this should fix the issue.

Awesome, glad to hear. I'll go ahead and close this :)

[fkromer]

@mzjp2 After migrating properly Kedro plays very well with Poetry :)

[mzjp2]

I'm very glad to hear that! :)