-
Notifications
You must be signed in to change notification settings - Fork 910
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add the Kedro RichHandler #2592
Add the Kedro RichHandler #2592
Conversation
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
…RichHandler Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
class: rich.logging.RichHandler | ||
class: kedro.extras.logging.RichHandler | ||
rich_tracebacks: True | ||
# tracebacks_show_locals: False |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Like settings.py
, we could have some commented default here to help user discover possible options. I put show_locals
because initially when we introduce rich
, we want to enable it as True
. It turns out to be a bit noisy for some users, and there was no good way to turn it off by the user.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this idea!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also like the idea, but remember this file won't exist in the default template in 0.19 and it will move to docs instead - see #2426 (this was a relatively recent decision in tech design: #2281 (comment)).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also note you changed the test_starter
here, not the actual kedro template starter.
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
kedro/extras/logging/rich_logger.py
Outdated
# Mapping Arguments from RichHandler's Constructor to rich.traceback.install | ||
for key, value in kwargs.items(): | ||
prefix = "tracebacks_" | ||
if key.startswith(prefix): | ||
key_prefix_removed = key[len(prefix) :] | ||
if key_prefix_removed == "suppress": | ||
mapped_kwargs[key_prefix_removed] += value | ||
else: | ||
mapped_kwargs[key_prefix_removed] = value | ||
elif key in ("locals_max_length", "locals_max_string"): | ||
mapped_kwargs[key] = value | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The original issue propose to support self.rich_tracebacks
only, I take it further because it is possible to support most of the arguments as long as they both exists in RichHandler
and rich.traceback.install
, the signature are slightly different so we need to map it.
kedro/extras/logging/rich_logger.py
Outdated
if self.rich_tracebacks: | ||
if "DATABRICKS_RUNTIME_VERSION" not in os.environ: | ||
# https://rich.readthedocs.io/en/stable/reference/logging.html?highlight=rich%20handler#rich.logging.RichHandler | ||
# Support compatible arguments between RichHandler and rich.traceback.install | ||
rich.traceback.install(**mapped_kwargs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alternatively, we can also remove the special logic about Databricks but just change the default to False in the starter because it is now customisable.
I didn't do it because it's still a breaking change, and there's no point to use Rich's traceback on Databricks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would rather keep it as you have it, so the default is True
and we include the special logic that disables it on Databricks.
logging.captureWarnings(True) | ||
|
||
# We suppress click here to hide tracebacks related to it conversely, | ||
# kedro is not suppressed to show its tracebacks for easier debugging. | ||
# sys.executable is used to get the kedro executable path to hide the | ||
# top level traceback. | ||
# Rich traceback handling does not work on databricks. Hopefully this will be | ||
# fixed on their side at some point, but until then we disable it. | ||
# See https://github.com/Textualize/rich/issues/2455 | ||
if "DATABRICKS_RUNTIME_VERSION" not in os.environ: | ||
rich.traceback.install(suppress=[click, str(Path(sys.executable).parent)]) | ||
rich.pretty.install() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The rich's traceback won't be installed when kedro.framework.project
is imported, it's now just a logging handler. This is an improvement because importing a module no longer has side-effect.
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To answer your questions:
Where should we put the logger? We have kedro.extras.logging but we plan to remove this namespace, it is already removed in dev. Do we need to put it in kedro.extras.logging and move it to kedro.logging in 0.19.0 or just do it now since it's not breaking?
I would just put it in kedro.logging
straightaway, so we don't need to move it again for the 0.19.0
release.
If some error happen before the log was set up it won't be rich-formatted properly, i.e. invalid RichHandler. I think it's fine but worth to mention.
That makes sense to me and I don't think there's a way around that.
I did some extra work to map arguments between RichHandler to rich.traceback.install to maximize the flexibility (Tests are not written yet, but if we agree this is useful then I will go on to write them). For example tracebacks_show_locals is mapped as show_locals for rich.pretty.install
I don't know that much about Rich so I'm not entirely sure yet what this means, but it sounds good 😄 But it would be good to get @antonymilne 's view on this.
class: rich.logging.RichHandler | ||
class: kedro.extras.logging.RichHandler | ||
rich_tracebacks: True | ||
# tracebacks_show_locals: False |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this idea!
Let me start by answering the questions and will then review the code.
Agree with @merelcht that we should just put it in
This is unavoidable really and should be fine. Users will be able to fix it using the
I think this is a good idea 👍
Probably you don't need to test all the known available arguments - just one of them should be fine. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good in general! I will try it out but basically I think you're on the right track here and should go ahead and write tests. The only big thing is to put the code in kedro/logging.py
instead of in extras.
Edit: tried it out and all seems to work as expected 👍 Note you need to set KEDRO_LOGGING_CONFIG=conf/base/logging.yml
to really test this properly at the moment. Again, things will be waaaaay simpler soon when we've done #2426.
class: rich.logging.RichHandler | ||
class: kedro.extras.logging.RichHandler | ||
rich_tracebacks: True | ||
# tracebacks_show_locals: False |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also like the idea, but remember this file won't exist in the default template in 0.19 and it will move to docs instead - see #2426 (this was a relatively recent decision in tech design: #2281 (comment)).
class: rich.logging.RichHandler | ||
class: kedro.extras.logging.RichHandler | ||
rich_tracebacks: True | ||
# tracebacks_show_locals: False |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also note you changed the test_starter
here, not the actual kedro template starter.
kedro/extras/logging/rich_logger.py
Outdated
"""Identical to rich's logging handler but with a few extra behaviours: | ||
* warnings issued by the `warnings` module are redirected to logging | ||
* pretty printing is enabled on the Python REPL (including IPython and Jupyter) | ||
* all tracebacks are handled by rich when rich_tracebacks=True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should have anoter point here to describe the extra rich.install
behaviour you added.
kedro/extras/logging/rich_logger.py
Outdated
if self.rich_tracebacks: | ||
if "DATABRICKS_RUNTIME_VERSION" not in os.environ: | ||
# https://rich.readthedocs.io/en/stable/reference/logging.html?highlight=rich%20handler#rich.logging.RichHandler | ||
# Support compatible arguments between RichHandler and rich.traceback.install | ||
rich.traceback.install(**mapped_kwargs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would rather keep it as you have it, so the default is True
and we include the special logic that disables it on Databricks.
kedro/extras/logging/rich_logger.py
Outdated
# Rich traceback handling does not work on databricks. Hopefully this will be | ||
# fixed on their side at some point, but until then we disable it. | ||
# See https://github.com/Textualize/rich/issues/2455 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment should be moved down to where it belongs.
kedro/extras/logging/rich_logger.py
Outdated
mapped_kwargs[key_prefix_removed] += value | ||
else: | ||
mapped_kwargs[key_prefix_removed] = value | ||
elif key in ("locals_max_length", "locals_max_string"): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the set of arguments that are the same for rich.logging.RichHandler
and rich.tracebacks.install
? Might be worth a comment explaining why they're here,
docs/source/logging/logging.md
Outdated
|
||
Kedro's `kedro.extras.logging.RichHandler` is a subclass of [`rich.logging.RichHandler`](https://rich.readthedocs.io/en/stable/reference/logging.html#rich.logging.RichHandler) and supports the same set of arguments. By default, `rich_tracebacks` is set to `True` to use `rich` to render exceptions. However, you can disable it by setting `rich_tracebacks: False`. | ||
|
||
When `rich_tracebacks` is set to `True`, the configuration is propagated to [`rich.traceback.install`](https://rich.readthedocs.io/en/stable/reference/traceback.html#rich.traceback.install). If an argument is compatible with `rich.pretty.install`, it will be passed to the traceback's settings. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When `rich_tracebacks` is set to `True`, the configuration is propagated to [`rich.traceback.install`](https://rich.readthedocs.io/en/stable/reference/traceback.html#rich.traceback.install). If an argument is compatible with `rich.pretty.install`, it will be passed to the traceback's settings. | |
When `rich_tracebacks` is set to `True`, the configuration is propagated to [`rich.traceback.install`](https://rich.readthedocs.io/en/stable/reference/traceback.html#rich.traceback.install). If an argument is compatible with `rich.traceback.install`, it will be passed to the traceback's settings. |
Unless I misunderstood something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My bad! I notice this yesterday but the changes are still in my local branch, good catch!
Co-authored-by: Antony Milne <49395058+antonymilne@users.noreply.github.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
.. autoclass:: RichHandler | ||
:members: | ||
:undoc-members: | ||
:inherited-members: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we'll have to remove the :undoc-members:
and :inherited-members:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! This seems to work. Sphinx is a puzzle, I cannot reproduce the same error in my local environment, but I am glad it's fixed now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't believe it worked 😂 I guess I'm a Sphinx black belt now 🥷
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com>
* Remove unnecessary files and subheaders in documentation (#2563) Tidying up. * Leverage PEP-585 (#2540) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> * [READY] Bring deployment docs up-to-date and add new pages for additional targets (#2557) Ready at last. Still some conversation to be had about how we reflect "confidence" in the plugins we describe (what version they were last tested against) but the text is ready and better than what's there currently, so let's go! * Remove mentions to `.egg` and stop producing them from `kedro package` (#2568) * Stop producing `.egg` files from `kedro package` Fix gh-2273. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Upgrade import-linter Fix gh-2570. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Partially remove reliance on setup.py for micropackaging Incomplete effort, see gh-2414. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Revert "Partially remove reliance on setup.py for micropackaging" This reverts commit fd3efa8. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * OmegaConfigLoader returns `Dict` instead of `DictConfig`, resolves runtime_params properly (#2467) * Fix typehint Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * test push Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * POC of fix to solve the runtime param resolution problem Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Fix OmegaConfigLoadaer - resolve runtime_params early Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Delegate the intialization of runtime_params to AbstractConfigLoader Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Add test for interpolated value and globals Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * add more test and linting Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * refactor Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * update release note Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Apply comments and refactor the test Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Update RELEASE.md Co-authored-by: Antony Milne <49395058+antonymilne@users.noreply.github.com> --------- Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Co-authored-by: Antony Milne <49395058+antonymilne@users.noreply.github.com> * Revert special logic handling DictConfig (#2582) * Partially revert #2378 as OCL no longer returning DictConfig Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Revert #2176 with special logging handling Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * remove unused import Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Re-title and modify introduction to databricks testing guide in contribution section (#2579) * modify databricks testing guide for contributors Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/contribution/development_for_databricks.md Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> * Update development_for_databricks.md --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> * Quick fix .gitpod.yml (#2585) * `OmegaConfigLoader` should not show `MissingConfigException` when the file is empty (#2584) * Allow empty config files Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Remove condition for empty config Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Revert "Remove condition for empty config" This reverts commit 7961155. * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add unit test Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Proposal of updated settings.py (#2587) * Proposal of updated settings.py Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Update settings after comment Signed-off-by: Nok <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Collaborative Experiment Tracking Docs (#2589) * Add metadata attribute to `kedro.io` datasets (#2537) * Add metadata attribute to kedro.io datasets Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Update specs in pipeline hooks docstrings (#2598) * Update pipeline hooks docstrings Signed-off-by: Tomas Van Pottelbergh <github@tomasvanpottelbergh.com> * Acknowledge community contribution --------- Signed-off-by: Tomas Van Pottelbergh <github@tomasvanpottelbergh.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Miscellaneous hooks improvements (#2596) * Add debug level logging for hook execution order Fix gh-1942. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Clarify role of setuptools in hooks Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Remove mentions to outdated and unused `pkg_resources` Fix gh-2391. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Adapt example hook to spec signature Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Grammar and style fixes in docs Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Revert "Adapt example hook to spec signature" This reverts commit ac1b495. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Minor modification to titles of deployment docs to replace "deprecated" with "legacy" (#2606) * FIX: Typo in markdown for kedro notebooks docs (#2610) * FIX: Typo in markdown for kedro notebooks docs Signed-off-by: debugger24 <rahulcomp24@gmail.com> * Acknowledge community contribution --------- Signed-off-by: debugger24 <rahulcomp24@gmail.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add the Kedro RichHandler (#2592) * Add the Kedro RichHandler Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Clean up imports and comments Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Map arguments to add flexibility to customize arguments available in RichHandler Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Clean up rich_logger.py and unused imports Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Add docs - first iteration Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Fix imports and small refactoring Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Improve the doc Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Update the project's template Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * format docstring Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Apply Jo's comment on the doc Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Fix the link with RichHandler Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Apply suggestions from code review Co-authored-by: Antony Milne <49395058+antonymilne@users.noreply.github.com> * Fix broken doc Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * update template defaults Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * refactoring RichHandler and improve doc Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Fix broken test due to new default RichHandler Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Fix doc Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Move kedro.extras.logging.RichHandler -> kedro.logging.RichHandler Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * add kedro/logging.py and remove the unnecessary import that cause namespace collision Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Refactor test Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Add more tests Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Fix lots of import, docstring Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Reorder docstring Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Fix tests Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Link options to doc Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Clean up doc and unused logger Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Fix spelling `customise` Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Refactor test with fixture Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Remove unnecessary copy() because it is now a fixture Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Split up test_rich_traceback_configuration() to smaller tests Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Add RELEASE.md Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Update doc Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> * Fix template logging * Mention awesome-kedro in docs (#2620) * Mention awesome-kedro in docs See kedro-org/kedro-devrel#40 * Update CONTRIBUTING.md * Spelling and style fixes Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update SnowparkTableDataSet docs (#2485) * Update SnowparkTableDataSet docs * Add missing snowpark dependencies for docs building Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Release 0.18.9 (#2619) * Update __init__.py * Update release note Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Update Version Numbers across docs Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Update RELEASE.md * Update Release Note Signed-off-by: Nok <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add documentation for deploying packaged Kedro projects on Databricks (#2595) * Add deployment workflow page Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add table of contents, entry point guide, data and conf upload guide Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add detailed instructions for creating a job on Databricks Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add images and automated deployment resources Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Remove use of 'allows', add summary Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Remove link to missing image Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add deployment workflow to toctree Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Lint and fix missing link Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Minor style, syntax and grammar improvements Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Fixes for correctness during validation Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add instructions for creating log output location Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Lint Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Lint databricks_run Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Minor wording change in reference to logs Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Modify reference to Pyspark-Iris Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Fix linter errors to enable docs build for inspection Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update build-docs.sh * Fix broken link Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Remove spurious word Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add advantages subheading Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update docs/source/integrations/databricks_deployment_workflow.md Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add alternative ways to upload data to DBFS Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Move note on unpackaged config and data Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Fix broken links Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Move databricks back into deployment section Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Remove references to PySpark Iris (pyspark-iris) starter Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Graphics links fixes, revise titles Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Fix broken internal link Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Fix links broken by new folder Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Remove logs directory Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Update image of final job configuration Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Add full stops in list. Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Fix conda environment name. Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Modify wording and image for creating a new job cluster Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Modify wording in guide to create new job cluster Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Remove --upgrade option Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add both ways of creating a new job Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> --------- Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Update docs font family to Inter and swap the logo (#2611) * Update docs font family to Inter Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Match weights to previous Google font Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Update logo images; revert css formatting for easier PR review Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Update text styles Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Add trailing whitespace (?) Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Update Kedro image path Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Update font size Signed-off-by: Tynan DeBold <thdebold@gmail.com> * update font to font-family: 'Inter', sans-serif; Signed-off-by: huongg <huongg1409@gmail.com> * testing with important for font-family Signed-off-by: huongg <huongg1409@gmail.com> * Make project name lowercase in docs Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * revert back to kedro/develop Signed-off-by: huongg <huongg1409@gmail.com> --------- Signed-off-by: Tynan DeBold <thdebold@gmail.com> Signed-off-by: huongg <huongg1409@gmail.com> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> Co-authored-by: huongg <huongg1409@gmail.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Enable variable interpolation for the catalog with `OmegaConfigLoader` (#2621) * Enable variable interpolation in catalog by escaping _ Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * Add archive link on docs home page (#2636) Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * add ravi-kumar-pilla to tsc (#2629) Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Add file for search console verification (#2631) * add verification file and change to conf Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Fix conf.py so google file is moved to root of docs build Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Chore/update assets for docs (#2625) * Update docs font family to Inter Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Match weights to previous Google font Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Update logo images; revert css formatting for easier PR review Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Update text styles Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Add trailing whitespace (?) Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Update Kedro image path Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Update font size Signed-off-by: Tynan DeBold <thdebold@gmail.com> * update font to font-family: 'Inter', sans-serif; Signed-off-by: huongg <huongg1409@gmail.com> * testing with important for font-family Signed-off-by: huongg <huongg1409@gmail.com> * update assets for experiment tracking, docs, pipeline, focus mode, matplotlib and plotly * update modular pipelines section Signed-off-by: huongg <huongg1409@gmail.com> * autoreload Signed-off-by: huongg <huongg1409@gmail.com> * reupload autoreload gif Signed-off-by: huongg <huongg1409@gmail.com> * revert back to develop branch Signed-off-by: huongg <huongg1409@gmail.com> * gif for ET comparison Signed-off-by: huongg <huongg1409@gmail.com> * remove jupyter notebook screenshot since its not used anywhere Signed-off-by: huongg <huongg1409@gmail.com> * Update databricks viz demo Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * pipeline_show_metrics gif Signed-off-by: huongg <huongg1409@gmail.com> * update different gif format Signed-off-by: huongg <huongg1409@gmail.com> * point to main instead of develop for kedro-banner Signed-off-by: huongg <huongg1409@gmail.com> --------- Signed-off-by: Tynan DeBold <thdebold@gmail.com> Signed-off-by: huongg <huongg1409@gmail.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> Co-authored-by: Tynan DeBold <thdebold@gmail.com> Co-authored-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Fix the level heading for collab exp tracking so it shows in sidebar (#2647) Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update Kedro icons in docs with for rebrand (#2652) Signed-off-by: Tynan DeBold <thdebold@gmail.com> * Modify logos (#2651) Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> * Fix lint Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> * remove the subsection of kedro.extras.extension Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Fix lint Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * remove the subsection of kedro.extras.extension Signed-off-by: Nok <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Nok Chan <nok.lam.chan@quantumblack.com> Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> Signed-off-by: Tomas Van Pottelbergh <github@tomasvanpottelbergh.com> Signed-off-by: debugger24 <rahulcomp24@gmail.com> Signed-off-by: Jannic Holzer <jannic.holzer@quantumblack.com> Signed-off-by: Tynan DeBold <thdebold@gmail.com> Signed-off-by: huongg <huongg1409@gmail.com> Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Co-authored-by: Jannic <37243923+jmholzer@users.noreply.github.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Co-authored-by: Antony Milne <49395058+antonymilne@users.noreply.github.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: rashidakanchwala <37628668+rashidakanchwala@users.noreply.github.com> Co-authored-by: Ahdra Merali <90615669+AhdraMeraliQB@users.noreply.github.com> Co-authored-by: Tomas Van Pottelbergh <85937396+tomasvanpottelbergh@users.noreply.github.com> Co-authored-by: Rahul Kumar <rahulcomp24@gmail.com> Co-authored-by: Tynan DeBold <thdebold@gmail.com> Co-authored-by: Huong Nguyen <32060364+Huongg@users.noreply.github.com> Co-authored-by: huongg <huongg1409@gmail.com> Co-authored-by: Ravi Kumar Pilla <ravi_kumar_pilla@mckinsey.com> Co-authored-by: Jannic Holzer <jannic.holzer@quantumblack.com> Co-authored-by: Merel Theisen <merel.theisen@quantumblack.com>
Description
Fix #2277
It should also open up the solutions for these logging issues
This will enable users to customise
rich
logging and optionally disable the rich's traceback if needed (Some platform doesn't play very well with it, e.g. Databricks and we have some logic to disable it by default)Bonus: It now also supports more arguments that is compatible with RichHandler and
rich.traceback.install
Questions for Reviewer
kedro.extras.logging
but we plan to remove this namespace, it is already removed indev
. Do we need to put it inkedro.extras.logging
and move it tokedro.logging
in 0.19.0 or just do it now since it's not breaking?rich.traceback.install
to maximize the flexibility (Tests are not written yet, but if we agree this is useful then I will go on to write them). For exampletracebacks_show_locals
is mapped asshow_locals
forrich.pretty.install
Note: I haven't created unit tests since there are many decisions about what should be enable/disable, I will add the test after the 1st round of review.
Development notes
sys.modules
for testing_ProjectLogging
, it may be easier now because this is configurable we don't have to reload module.Checklist
Manual Testing
kedro new -s pandas-iris
conf/base/logging.yml
RELEASE.md
fileNote:
Currently user need to set
KEDRO_LOGGING_CONFIG
to disable rich's traceback, because the default logging installed it and there is no way to uninstall it.We need to solve this by either implementing it ourselves or raise a PR to
rich