Configuring Logging

Logging

About Levels

Galah does its best to log in a manner that system administrators should really appreciate. Each log level mean something fairly distinct within Galah.

1. DEBUG: Possibly helpful noise. Messages that would not be useful to system administrators outside of trying to debug a very specific problem. Very high volume.
2. INFO: Helpful status messages. Messages that may be useful for system administrators monitoring the status of the system, and useful to have in the logs when problems are reported in order to gain context.
3. WARNING: Human intervention required, eventually. Similar to INFO, however, it means that human intervention is required. These will occur when an error condition that is predicted and acceptable occurs, but don't need to be addressed immediately. You should look through all warnings that occur as they are indicators of problems, but you can take your time getting to them.
4. ERROR: Human intervention required now. Similar to warning, but human intervention is required as soon as possible. These will occur when an error condition that was not predicted occurs and could indicated serious problems (they could also just indicate a benign bug in Galah, but it's impossible to know). All errors should be examined immediately and if any occur that cannot be fixed from the system administrator side of things, a bug report should be filed.
5. CRITICAL: The ship is sinking, SOS. This is the highest level message that Galah can log and is only done in the most dire of circumstances (as of this writing, only two conditions cause a critical message to be logged, and they are both very dire). Any critical messages need to be taken care of immediately. If possible, critical error messages should send a text immediately to the on-call system administrator. Hopefully you'll never see this.

The goal is for no WARNING, ERROR, or CRITICAL error messages to occur unless the system administrator needs to do something. If you see a log message of these levels occur that you can't do anything to solve or doesn't seem useful, please file a bug report because this is a bug. Logging is important and we want to do it right, please help us out.

Getting Logging Configured

The Python standard library's logging module is used for all of Galah's internal logging. You have extremely fine-grained power over Galah's logging thanks to Python's standard logging module which you have a chance to access within your configuration file. If you are well adept at navigating Python and this particular library go ahead and set up any log handlers that you'd like. Alternatively, if your needs are simple enough, you can choose from one of the common logging configurations below... These configurations are in addition to any configuration options that you already set in your configuration file. In other words, don't forget to create a config dictionary.

Logging to Standard Error

This configuration is only viable if you are in a development environment, executing Galah's components yourself through the terminal.

import logging
logger_galah = logging.getLogger("galah")
logger_galah.setLevel(logging.DEBUG)

streamhandler = logging.StreamHandler()
streamhandler.setFormatter(logging.Formatter(
    "[%(levelname)s;%(name)s;%(asctime)s]: %(message)s"
))
logger_galah.addHandler(streamhandler)

Sending to syslog

Sending log messages to syslog is simple and very standard. A known problem with this configuration is that very long stack traces are truncated which can make reporting errors and debugging painful. If you use syslog already however, this is probably the configuration you'll be most comfortable with.

import logging
import logging.handlers

# Explanation of this at http://serverfault.com/a/408491
class BOMLessFormatter(logging.Formatter):
    def format(self, record):
        return logging.Formatter.format(self, record).encode("utf-8")

logger = logging.getLogger("galah")
logger.setLevel(logging.DEBUG)
log_handler = logging.handlers.SysLogHandler(address = "/dev/log")

# Modify this formatter if you want to change the layout of log messages.
formatter = BOMLessFormatter("%(name)s[%(process)d]: %(message)s")
log_handler.setFormatter(formatter)

logger.addHandler(log_handler)

This will produce log messages from syslog that look like...

Dec  5 03:34:01 ip-10-100-199-217 galah.web.views.login[13918]: (user=test_student_1@testing.edu;ip=66.215.201.124;path=/login/) User test_student_1@testing.edu has succesfully logged in via internal auth.
Dec  5 03:34:11 ip-10-100-199-217 galah.web.views.logout[13918]: (user=test_student_1@testing.edu;ip=66.215.201.124;path=/logout) Succesfully logged out.

Sending to Sentry

Sentry is an amazing logging server and is the best known way of keeping track of Galah's log messages.

In order to use Sentry you need to set it up yourself (it is a server that runs completely seperately from Galah). Check out the quickstart on Sentry's documentation here.

Once you have it set up, you'll want to use a logging configuration in Galah similar to the following:

from raven import Client
from raven.handlers.logging import SentryHandler
from raven.conf import setup_logging
client = Client('http://huge:key@sentry-server:9000/2')
sentryhandler = SentryHandler(client)
setup_logging(sentryhandler)