New Log Order

15th June, Year of our $DEITY 2016 - Log Pragma Changes

• Where is my APPLICATION_NAME log?

It is probably in /var/log/runnable/<YYYY>/<MM>/<DD>/<HH>/<APPLICATION_NAME>.log.

• Why did this change?

This was easier than refactoring everything that sends NPM output into /var/log/APPLICATION_NAME.log.

• Why not have both in one?

tl;dr - various conditions race.

• What applications are we talking about?

Basically anything we deploy as a service that runs in a container, like api or api-socket or navi or eru or detention and so on. Further, the services we are upstarting on the Docks have long been using the bunyan-syslog library to handle communication to the local rsyslogd instance on a given Dock (since upstart services run at the host level, no volume mounting or port exposing magic is necessary).

Check out this cool trick!

• Try to log into your favorite server and type logtail <app_name> where <app_name> is something like "api" or "docker-listener."

ETA 2016/05/16 Added commands to global .bash_aliases:

• logtail: Usage: logail <app_name> - follow a log and format with Bunyan
• lograw: Usage: lograw <app_name> - follow a log in raw format
• loglast: Usage: loglast <app_name> [ <#_of_lines> ] - Display the last few (or # number of) lines with Bunyan
• greplog: Usage: greplog <app_name> <regexp> - grep for an expression and format in Bunyan.

Searching in for logs in Loggly

You want "json.name" for the app name and "tag" for the environment, e.g.: "production-delta" and so on.

Debuggery

PRO-TIP - try deploying with -t deploy,loggly!

• There is a <APPLICATION_NAME>.log but there's nothing in it?

or

• There is no /var/log/runnable.. ?!

A few things could be tripping up logs here, check that the log-level is set to something sane if the service is an upstart service by looking at the LOG_LEVEL set in /etc/init/<APPLICATION_NAME>.conf.

Check to make sure that rsyslog is running, and if it has picked up any recent config changes (simply restart the service sudo service rsyslog restart).

Make sure an appropriate rsyslog config exists for the service: /etc/rsyslog.d/21-rotated-<APPLICATION_NAME>.conf should exist. If not, try re-deploying using ansible-playlist -i <ENV> -e git_branch=<GIT_BRANCH> -t loggly <APPLICATION_NAME>.yml.

• Still, no logs!

Rsyslog is now taking input directly from the docker-engine syslog driver, which pipes STDOUT from the container into the system's /dev/log log "device" (it's really just a named pipe). There is no other way to get docker-engine to send logs out the log driver than to restart the container with new runtime options for logging, so you will have to re-deploy your app.

Check to make sure the app has a /etc/rsyslog.d/21-rotated-<APP_NAME>.conf file to tell rsyslog how to manage the log.

• No luck still?

Ping someone on #ops-chatter in Slack.

The Future is Now! And In The Future!

Now we will be able to do some fun things in Ansible using the new log pragma:

1. Move logs off / (eg, /docker/app_logs or some other path, or a shared log volume);
2. Filter logs based on log severity at the rsyslogd level.
3. Munge the log output format to BEND TO OUR EVIL WILL!!!