[KED-2110] kedro viz command fails when kedro plugin with automatic hook discovery is installed

[turn1a]

Description

As per title, kedro viz command fails when kedro plugin with automatic hook discovery (kedro.hooks entry point) is installed (see traceback below).

Context

During the kedro viz the kedro_viz/server.py in _call_viz is getting the project context (context = get_project_context("context", env=env), line 511). This function in core Kedro is returning a deep copy of the context. Deep copy fails when calling __reduce_ex__ that's being getattr-ed from Pluggy's DistFacade class. DistFacade is overwriting the __getattr__ and __dir__ dunders to include _dist in it's dictionary and I think this is where the problem lies, but I don't know how or where to fix this or even if I'm submitting an issue in the right place. kedro viz is failing due to this issue hence I'm submitting a bug report here, but it might be as well a bug in Kedro Core or Pluggy.

I created a minimal kedro plugin and a minimal kedro project which you can use to investigate.

Steps to Reproduce

1. git clone https://github.com/kaemo/kedro-minimal-plugin
2. git clone https://github.com/kaemo/kedro-minimal-project
3. cd kedro-minimal-plugin && make setup && make build
4. cd ../kedro-minimal-project && python -m venv .venv && source .venv/bin/activate && python -m pip install kedro && kedro install && python -m pip install ../kedro-minimal-plugin/dist/kedro_minimal_plugin-0.0.1-py3-none-any.whl
5. kedro viz

Expected Result

A web app should start and Kedro Viz app should be open in a web browser.

Actual Result

❯ kedro viz
2020-09-03 13:20:15,688 - root - INFO - Registered hooks from 1 installed plugin(s): kedro-minimal-plugin-0.0.1
Traceback (most recent call last):
  File "/Users/olszewk2/dev/kedro-minimal-project/.venv/lib/python3.8/site-packages/kedro_viz/server.py", line 468, in viz
    _call_viz(host, port, browser, load_file, save_file, pipeline, env)
  File "/Users/olszewk2/dev/kedro-minimal-project/.venv/lib/python3.8/site-packages/kedro_viz/server.py", line 511, in _call_viz
    context = get_project_context("context", env=env)
  File "/Users/olszewk2/dev/kedro-minimal-project/.venv/lib/python3.8/site-packages/kedro/framework/cli/cli.py", line 663, in get_project_context
    return deepcopy(value)
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 205, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 210, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 210, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/Users/olszewk2/miniconda3/lib/python3.8/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: 'NoneType' object is not callable
Error: 'NoneType' object is not callable

Your Environment

• macOS Mojave 10.14.6
• zsh 5.8
• Python 3.8.5
• kedro 0.16.4
• kedro-viz 3.4.0
• pluggy 0.13.1

[turn1a]

@richardwestenra could you have a look at this one? This bug will break kedro-viz whenever there's another plugin installed that's using automatic hook discovery functionality.

[richardwestenra]

@kaemo Hey sorry for the delay in responding, looks like this fell through the cracks! I've created a JIRA ticket to ensure that it's escalated, and I'm going to see if we can assign this to one of our python devs to take a look at asap. Thanks for reporting!

[turn1a]

@richardwestenra Amazing, thank you!

[921kiyo]

@kaemo I've looked into this issue and was able to reproduce the error with 0.16.4. This was fixed in 0.16.5 so could you please upgrade your kedro version to 0.16.5? I'm closing this issue, but please let us know if you have any follow-up questions.

[rodra-go]

@921kiyo , I could reproduce the same error using kedro version 0.16.5. I don't understand it, because kedro viz was working fine with this version, but suddenly stoped working.

[921kiyo]

Which version of kedro and kedro-viz are you using? And which kedro version did you use to generate your project template? And can you paste the full error stack here?

Note if your kedro project was generated with 0.16.5, and try to run with kedro 0.17.*, there's a breaking change (see https://github.com/quantumblacklabs/kedro/blob/master/RELEASE.md#other-breaking-changes-to-the-api), so it might be causing the issue.

If your kedro project was created with 0.16.5 and continue using kedro 0.16.* and the latest kedro-viz (or your kedro project was created with 0.17., and try to run 0.17. and latest kedro-viz), that should work without any issue.

Hope this helps :)

[rodra-go]

@921kiyo , thank you for your reply! The problem was being caused by my Dockerfile, so I could fix it and now everything works fine. Cheers!