log2d is an incredibly helpful wrapper around Python's logging module in the standard library and can be installed via PIP in the normal way:
python -m pip install log2d --upgrade
It aims to provide the best parts of logging (like automatic, rotating backup files) to users who don't want or need to learn every nuance of the module itself and perhaps simply want to wean themselves off print() statements and organise their output in a better "2 Dimensional" way (hence the name - log2d).
If you've dipped into the standard logging documentation you'd be forgiven for thinking you can only log output according to the prescribed log levels: DEBUG, INFO, WARNING, ERROR, or CRITICAL/FATAL. Such an approach is linear or "1 Dimensional" since it's based solely on the importance of a message.
A very common use case however is the need to capture different types of message. Hence TWO dimensions.
In web-scraping apps for example, it's useful to collect the HTTP requests which succeeded or failed or needed a few retries, quite apart from any general Exceptions arising from your actual code. You might also want a nice (separate) log of overall progress and timings i.e. how long particular scrapers take to complete.
log2d makes it simple to create, customise, and use a new logger for each of these types of output, for example sending progress messages just to the console, and creating separate .log files for successes, failures, retries, exceptions, and timings. It also lets you search these logs for specific message text, level or over specific time periods.
It does so in a concise, readable, and (dare I claim?) "Pythonic" way, that doesn't require mastery of the logging module itself. It allows you to create a sophisticated logger with powerful default features enabled in just one line of code, then send output to that logger whenever and from wherever you like - also in just one line.
At the end of this README there are some simple Cookbook recipes for dynamically creating a log for any given Module, Class, or Instance.
Simply import the Log class into your Python script:
from log2d import Log
Create a named logger that only outputs to the console ("stdout") using default message formatting and date format:
Log("main")
Log.main.warning("Danger, Will Robinson!")
Output:
main|WARNING |2022-10-25T19:34:30+0100|Danger, Will Robinson!
In place of
.warningyou can use any of the standard log levels, either upper or lower case: DEBUG, INFO, WARNING, ERROR, and CRITICAL/FATAL
log_success = Log("success", to_file=True)
log_success("log2d for the win!")
Log.success.critical("Alert! Alert!")
(Creates and updates ./success.log)
NB The 'sensible' default logic of
log2dassumes that if you ONLY specifyto_fileas a parameter, you ONLY want output to go to file, and output to the console (to_stdout) is automatically set toFalseThe default values for
to_stdoutandto_fileareTrueandFalserespectively. In other words if you provide neither parameter, logging is to the console only.The default file logging
modeisa- append log messages to the existing log mode indefiintely. See below for file logging modewwhich overwrites the log file each time the script is run.
Log("my_title", path="./output")
NB
to_fileis automatically set to True if apathis supplied.The default path is the current working directory
""or"."If a non-existent folder/directory is specified,
FileNotFoundErrorwill be raised.
Log("my_title", level="WARNING")
For simple scripts with no or few imports you might like to create your own shortcut functions like this:
log_failure = Log("failures")
log_failure("Insert your failure message here")
Output:
failures|DEBUG |2022-10-25T19:35:06+0100|Insert your failure message here
Normal considerations regarding namespaces apply however, and for longer/more complex scripts it might be wiser to stick with the explicit naming convention
Log.logger_name().You could use this shortcut feature to overwrite the
print()line into a logging command._print = print; print = Log("print")The default log level used by
log2dis actually DEBUG, whereas theloggingdefault is WARNING. This change is intended to make things safer and more predictable for new users who might otherwise be sending DEBUG and INFO level messages and wondering why they're not being logged.
-
You can create a logger name with spaces and other characters rather than underscores, but you wouldn't then be able to use Python's nice
.attributenotation. If your log name was "my main log" you'd need to usegetattr(Log, "my main log").warning("...")instead, which is a bit messy. Best to just use underscores if you can. -
Just as in the standard
loggingmodule, the name "root" is reserved for a special type of logger which actually inherits from other loggers. This can be very helpful if you want a single "master" logger that records absolutely everything, but also a bit annoying if you weren't aware of it and have already explicitly disabled output at a particular level, only to see it appear in your "root" logger. Here's a quick example to demonstrate how this works:
Log("main")
Log.main.info("This is the MAIN logger")
Output:
main|INFO |2022-10-08T23:47:08+0100|This is the MAIN logger
Log("root", fmt=Log.presets['timestamp_only'])
Log.root.info("This message is for ROOT only")
Output:
2022-10-08T23:52:09+0100|This message is for ROOT only
Log.main.info("This message will be echoed by BOTH loggers")
Output:
main|INFO |2022-10-08T23:49:23+0100|This message will be echoed by BOTH loggers
2022-10-08T23:49:23+0100|This message will be echoed by BOTH loggers
If you don't want to use this 'inheritance' feature, just avoid using the name "root" for any of your log2d loggers.
(See also USING LOGGING AND LOG2D AT THE SAME TIME below)
from log2d import Log, logging
mylog = Log("mylog")
mylog.add_level("NewError", below="ERROR")
mylog.add_level("NewInfo", above="INFO")
Log.mylog.newerror(f"New log level {logging.NEWERROR} below ERROR")
Log.mylog.newinfo(f"New log level {logging.NEWINFO} above INFO ")
Output:
"New log level 'newerror' added with value: 39"
"New log level 'newinfo' added with value: 21"
mylog|NEWERROR|2022-11-09T06:17:00+0000|New log level 39 below ERROR
mylog|NEWINFO |2022-11-09T06:17:17+0000|New log level 21 above INFO
This method returns a confirmation message rather than logging its own output and potentially messing up your pristine logging schema. You can suppress it by assigning a dummy variable e.g.
_ = mylog.add_level("NewError", below="ERROR")
.add_level() will also overwrite previous log levels at a given value if they already exist, and create the new log level immediately above or below the reference log level i.e. without leaving any gaps. For explicit control over the postion of log levels, you can also specify the log level value numerically:
mylog.add_level("TRACE", 15)
Log.mylog.trace("Trace message...")
The .find() method helps you easily search for text in messages above a particular level and/or within a particular time period:
mylog = Log("testlog", to_file=True)
Log.testlog.info("Test info message")
Log.testlog.error("Test error message")
Log.testlog.info("Test info message")
Log.testlog.info("Three line message\n\twith more data on this line\n\t\tand also on this line too!")
Log.testlog.info("Test info message")
mylog.find()
# Returns all entries in the named logfile in the last 7 days (default)
Output:
['testlog|INFO |2022-12-10T10:07:07+0000|Test info message',
'testlog|ERROR |2022-12-10T10:07:07+0000|Test error message',
'testlog|INFO |2022-12-10T10:07:08+0000|Test info message',
'testlog|INFO |2022-12-10T10:07:08+0000|Three line message',
'with more data on this line',
'and also on this line too!',
'testlog|INFO |2022-12-10T10:07:08+0000|Test info message']
You can specify a minimum logging level as follows:
mylog.find(level="error")
Output:
['testlog|ERROR |2022-12-10T10:07:07+0000|Test error message']
And you can search for particular text and set case sensisitive like this:
results = mylog.find(text="data", ignorecase=True, deltadays=-31)
# Case insensitive search for all messages containing 'data' within the 31 days
If you want to search a log file by path e.g. one not created by log2d you can use the Class method:
results = Log.find(path="some_other_log.log")
results = Log("session_results", to_file=True, mode="w")
results = Log("session_results", to_file=True, mode="w", backup_count=10)
The current log file will always be
session_results.logbut for subsequent sessions this will be copied tosession_results.log.1thensession_results.log.2etc until the backup count is reached, then start again on a rotating basis.If
backup_countis not specified, the default number of backups is 5.
Preview a particular message format and/or date format - either one of the supplied presets, or one of your own design:
Log.preview(fmt=Log.presets["timestamp_only"], datefmt=Log.date_formats["time"])
Output:
13:10:06|This is a preview log entry.
Log.preview(datefmt="%m-%d::%H:%M")
Output:
temp_preview|09-25::15:36|This is a preview log entry.
Log.preview_all()
Log.index
logger = Log("main").logger
type(logger)
<class 'logging.Logger'>
dir(logger)
[...
'addFilter', 'addHandler', 'callHandlers', 'critical', 'debug', 'disabled', 'error', 'exception', 'fatal', 'filter', 'filters', 'findCaller', 'getChild', 'getEffectiveLevel', 'handle', 'handlers', 'hasHandlers', 'info', 'isEnabledFor', 'level', 'log', 'makeRecord', 'manager', 'name', 'parent', 'propagate', 'removeFilter', 'removeHandler', 'root', 'setLevel', 'warn', 'warning']
from log2d import Log, Path
if __name__ == '__main__':
log = Log(Path(__file__).stem, to_file=True).logger
# Then just reuse the log object elsewhere in your script e.g.:
file_name = Path(__file__).name
log.critical(f'critical message from: {file_name}')
log.error(f'error message from: {file_name}')
log.warning(f'new warning message from: {file_name}')
log.info(f'info message from: {file_name}')
log.debug(f'debug message from: {file_name}')
"""
OUTPUT:
my_file|CRITICAL|2022-10-25T16:32:50+0100|critical message from: my_file.py
my_file|ERROR |2022-10-25T16:32:50+0100|error message from: my_file.py
my_file|WARNING |2022-10-25T16:32:50+0100|new warning message from: my_file.py
my_file|INFO |2022-10-25T16:32:50+0100|info message from: my_file.py
my_file|DEBUG |2022-10-25T16:32:50+0100|debug message from: my_file.py
"""
from log2d import Log
class MyClass:
def __init__(self, name):
params = {"fmt": Log.presets["name_and_time"]}
self.log = Log.index.get(name) or Log(name, **params)
def method_1(self):
# Do something
self.log("method_1 did something!")
x = MyClass("Instance 1")
x.method_1()
y = MyClass("Instance 2")
y.method_1()
x.log("This message was logged directly")
y.log("Likewise, but different instance")
"""
OUTPUT:
Instance 1|2022-10-16T08:50:29+0100|method_1 did something!
Instance 2|2022-10-16T08:29:05+0100|method_1 did something!
Instance 1|2022-10-16T08:53:17+0100|This message was logged directly
Instance 2|2022-10-16T08:54:06+0100|Likewise, but different instance
"""
from log2d import Log
class MyAbstractClass:
def __init__(self, name, *args, **kwargs):
params = {"fmt": Log.presets["name_and_time"]}
self.log = Log.index.get(name) or Log(name, **params)
class MyClass(MyAbstractClass):
def __init__(self, name, *args, **kwargs):
super().__init__(self.__class__.__name__, *args, **kwargs)
self.name = name
def method_1(self):
# Do something
self.log(f"method_1 of {self.name} did something!")
x = MyClass("Instance X")
x.method_1()
y = MyClass("Instance Y")
y.method_1()
x.log(f"This message was logged by {x.name}")
y.log(f"And this one by {y.name}")
"""
OUTPUT:
MyClass|2022-10-16T08:43:31+0100|method_1 of Instance X did something!
MyClass|2022-10-16T08:43:45+0100|method_1 of Instance Y did something!
MyClass|2022-10-16T08:57:52+0100|This message was logged by Instance X
MyClass|2022-10-16T08:58:18+0100|And this one by Instance Y
"""
fmt = Log.presets["file_func_name"]
datefmt = Log.date_formats["date_and_time"]
Log("main", fmt=fmt, datefmt=datefmt)
fmt = '%(asctime)s:%(levelname)s:%(name)s:%(message)s'
datefmt = '%d/%m/%Y %I:%M:%S %p'
Log("main", fmt=fmt, datefmt=datefmt)
For more information on composing your own formats see:
https://docs.python.org/3/library/logging.html#formatter-objects
https://docs.python.org/3/library/logging.html#logging.LogRecord
Recipe 5: Add a new date format or message format preset at the Class level, so that future instances can use them:
fmt = "%(asctime)s (%(name)s): %(message)s"
datefmt = "%m-%d %H:%M"
Log.presets["my_message_format"] = fmt
Log.date_formats["my_date_format"] = datefmt
As shown earlier, values for level, fmt, datefmt, to_file, to_stdout, path, mode, and backup_count can be set for a specific logger by supplying them as keyword arguments on initialisation.
Where no argument is supplied for a new logger, the Class level defaults will be used. Default attributes can also be set at a class level so that all subsequent loggers have the same or similar settings:
from log2d import Log
ROOT_DIR = "./my_app_path"
Log.path = f"{ROOT_DIR}/logs/"
Log.fmt = Log.presets["name_and_time"]
Log.datefmt = Log.date_formats["time"]
Log.to_file = True
Log.to_stdout = False
Log.mode = "w"
Log.backup_count = 10
Log("main", to_stdout=True)
Log("selenium")
Log("requests")
Log("timings")
Log("results")
Log("retries")
Log("errors")
You may find yourself using code that already has logging enabled. This won't interfere with any loggers you subsequently create with log2d but you might find that some log2d messages are repeated (inhrerited) by the existing RootLogger logging.Logger.root, for example echoing WARNING level messages to stdout even though you've explicitly disabled this in your log2d logger.
In case you were wondering, a log2d logger called "root" will inherit from other log2d loggers in a similar way to RootLogger but it is not the same entity as logging.Logger.root created by downstream code you've imported from. Both can exist at the same time and be configured to behave differently.
So... if you're able to amend the original code where logging is enabled, the best way to ensure consistent behaviour is to replace logging with log2d entirely:
- Replace
import loggingwithfrom log2d import Log - Create a new "root" logger with
log2dand specify how you want it to work in the normal way e.g.Log("root", level="INFO") - Replace
logger.info(),logger.warning()etc withLog.root.info(),Log.root.warning()etc.
Another option is to disable the RootLogger using the log2d convenience function:
Log.disable_rootlogger()
This is a bit of a blunt instrument and basically finds all the Handlers used by RootLogger (logging.Logger.root.handlers) and runs the .removeHandler() method on them.
If neither of these approaches give you what you need, you can dive a bit deeper into the Log.get_handlers method of log2d or see above for accessubg the underlying logging.Logger object.
If you find yourself doing this, you'll inevitably find yourself reading the standard library logging documentation which of course goes against the very reason for log2d's existence - to offer simple, sane, and sensible logging without the pain of having to learn its innner workings. So if you've reached this point and need more control or sophistication with your logging, then log2d has probably served it's purpose and you're ready to move onwards and upwards!
I'd be delighted to hear any suggestions, bug reports, or comments in the form of a Github ISSUE, and if you've found log2d useful or merely interesting please do click the "Star" button. It really raises my spirits to see that kind of feedback.
If you're in paid employment and log2d has saved you even a few minutes' effort, please consider how much this gift is worth based on your daily/hourly earnings, and whether it might feel good to at least leave a nice comment on BuyMeaCoffee.com? Thank you.
Special Thanks to Mike Pollard for the .find() utility.