Rich output is too long

[deepyaman]

Description

I'm always frustrated when my traceback is 838 lines long for a trivial issue.

Context

How has this bug affected you? What were you trying to accomplish?

Possible Implementation

Make Rich an optional dependency (not just on certain platforms like Databricks).

Steps to Reproduce

1. pip install kedro==0.18.2
2. kedro new --starter spaceflights and accept defaults
3. Modify src/spaceflights/pipelines/data_processing/nodes.py, changing companies["iata_approved"] = _is_true(companies["iata_approved"]) to companies["iata_approved"] = _is_true(companies["iata_approve"]) (removed a "d").
4. kedro run

Expected Result

Tell us what should happen.

0.18.1 (not Rich) traceback is 57 lines long:

Traceback (most recent call last):
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 163, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'iata_approve'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/bin/kedro", line 8, in <module>
    sys.exit(main())
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/framework/cli/cli.py", line 217, in main
    cli_collection()
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/framework/cli/cli.py", line 145, in main
    super().main(
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/framework/cli/project.py", line 352, in run
    session.run(
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/framework/session/session.py", line 391, in run
    run_result = runner.run(
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/runner/runner.py", line 87, in run
    self._run(pipeline, catalog, hook_manager, session_id)
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/runner/sequential_runner.py", line 69, in _run
    run_node(node, catalog, hook_manager, self._is_async, session_id)
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/runner/runner.py", line 211, in run_node
    node = _run_node_sequential(node, catalog, hook_manager, session_id)
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/runner/runner.py", line 305, in _run_node_sequential
    outputs = _call_node_run(
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/runner/runner.py", line 273, in _call_node_run
    raise exc
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/runner/runner.py", line 263, in _call_node_run
    outputs = node.run(inputs)
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/pipeline/node.py", line 353, in run
    raise exc
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/pipeline/node.py", line 342, in run
    outputs = self._run_with_one_input(inputs, self._inputs)
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/kedro/pipeline/node.py", line 373, in _run_with_one_input
    return self._func(inputs[node_input])
  File "/Users/deepyaman/spaceflights/src/spaceflights/pipelines/data_processing/nodes.py", line 29, in preprocess_companies
    companies["iata_approved"] = _is_true(companies["iata_approve"])
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/pandas/core/frame.py", line 3505, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/Users/deepyaman/miniconda3/envs/kedro-argo/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3623, in get_loc
    raise KeyError(key) from err
KeyError: 'iata_approve'

Actual Result

Tell us what happens instead.

0.18.2 (Rich) traceback is 838 lines long:

https://pastebin.com/raw/efg3bhD1 (I'd paste it here, but GitHub errors out saying body is too long/exceeds 65536 characters 🤦).

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used (pip show kedro or kedro -V): 0.18.2 (vs 0.18.1)
• Python version used (python -V): 3.10.5
• Operating system and version: M1 Mac

[antonymilne]

Thanks for raising this @deepyaman. I have also found the rich tracebacks too verbose and was kind of waiting to see if anyone had the same opinion independently. @SajidAlamQB

I think the right solution here would be:

• set show_locals to False, which is already done in Use show_locals False default for rich logging #1726. This reduces the size of the traceback a lot
• if you think that is still too verbose, maybe we should reduce the extra_lines (currently at default value of 3). Or feel free to change any other settings for rich.traceback.install which you think would help - we're not wedded to the current options at all, so I'm happy to accept a PR which changes these if you think there's more suitable settings
• make rich traceback options configurable for users, e.g. if they want to switch show_locals back on again or turn rich exception handling off altogether. As per Make rich traceback and REPL handling more configurable #1728, this is actually already possible (if a bit hacky), so if you have a view on how to do it better please do add it there 🙏

We could make rich an optional dependency, but I would prefer that we keep it, change any settings so that the default behaviour is better for kedro users and make it more easily configurable for users who want to change those settings or disable it altogether.

[antonymilne]

For reference, here's what the same traceback would look like now we've turned show_locals off: https://pastebin.com/raw/irPFPMzJ (232 lines)

And with extra_lines = 0 also: https://pastebin.com/raw/635rUPmv (128 lines).

Personally I think a non-zero extra_lines is helpful but don't mind if it's less than the default of 3. Turning off show_locals is the thing that makes the biggest difference to clarity of the tracebacks here I think.

[antonymilne]

From @deepyaman:

I think it's fine to go with show_locals=False and leave the extra_lines change for if and when get a critical mass of people saying it's still too long. i think it's very subjective at some point, but the 4x reduction so far is appreciated 😄

I'm going to close the issue for now, but others should feel free to re-open it if and when they think we should tinker with these settings further.