Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep exceptions logs #898

Merged
merged 20 commits into from
Oct 29, 2023
Merged

Keep exceptions logs #898

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
612dea7
Add a function to create a logger.error for an exception
ekatef Oct 14, 2023
33787cc
Stream exception info into a log file
ekatef Oct 14, 2023
efeb56b
Move logger definitions inside an exceptions hook
ekatef Oct 15, 2023
01f9c41
Improving an error message
ekatef Oct 15, 2023
9e9eab8
Merge branch 'main' into keep_exceptions_logs
ekatef Oct 16, 2023
2b1f73d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 16, 2023
1fc05f6
Add keyboard interruptions to logs
ekatef Oct 23, 2023
337fd78
Add a dedicated function to create a logger
ekatef Oct 23, 2023
5309f26
Apply log functions inside a module
ekatef Oct 23, 2023
785e72b
Apply log functions inside a module
ekatef Oct 23, 2023
893c386
Merge remote-tracking branch 'origin/keep_exceptions_logs' into keep_…
ekatef Oct 28, 2023
0dc22d7
Fix a logger name
ekatef Oct 28, 2023
8931d5a
Add docstrings
ekatef Oct 28, 2023
8e4fc80
Update logging setup across the scripts
ekatef Oct 28, 2023
2d3907a
Add a missed import
ekatef Oct 28, 2023
e0d89fa
Add logging updates
ekatef Oct 28, 2023
bad5bb9
Update a release note
ekatef Oct 28, 2023
0d349f9
Amend a release note
ekatef Oct 28, 2023
b75b1eb
Remove empty lines
ekatef Oct 28, 2023
64388a3
Fix logger calls and imports
ekatef Oct 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions scripts/_helpers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import logging
import os
import subprocess
import sys
from pathlib import Path

import country_converter as coco
Expand All @@ -22,6 +23,33 @@

REGION_COLS = ["geometry", "name", "x", "y", "country"]

logger = logging.getLogger(__name__)


def handle_exception(exc_type, exc_value, exc_traceback):
davide-f marked this conversation as resolved.
Show resolved Hide resolved
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
handler = logging.StreamHandler(stream=sys.stdout)
logger.addHandler(handler)

if issubclass(exc_type, KeyboardInterrupt):
sys.__excepthook__(exc_type, exc_value, exc_traceback)
return

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, drop the empty line for PEP8 standard

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

tb = exc_traceback
while tb.tb_next:
tb = tb.tb_next
flname = tb.tb_frame.f_globals.get("__file__")
funcname = tb.tb_frame.f_code.co_name

logger.error(
"An error happened in module %r, function %r: %s",
flname,
funcname,
exc_value,
exc_info=(exc_type, exc_value, exc_traceback),
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be simplified with something like

    def handle_exception(exc_type, exc_value, tb):
      ...
      [Exclude Interrupt]
      ...
      logger.exception(
          "Exception {0}:{1} has occurred in the workflow.".format(exc_type, exc_value),
          exc_info=True, stack_info=True
      )

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmm on second though, I'd log also the keyboardinterrupt so to keep track that it has been a manual action

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely agree on keeping info on KeyBoardInterrupt. Added a functionality for that.



def read_osm_config(*args):
"""
Expand Down
4 changes: 2 additions & 2 deletions scripts/build_shapes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import xarray as xr
from _helpers import (
configure_logging,
handle_exception,
sets_path_to_root,
three_2_two_digits_country,
two_2_three_digits_country,
Expand All @@ -36,8 +37,7 @@
from shapely.validation import make_valid
from tqdm import tqdm

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
sys.excepthook = handle_exception
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shall be placed into the create_logger fuction but here we shall create the local logger like:
logger = create_logger(__name__, logging.INFO)


sets_path_to_root("pypsa-earth")

Expand Down