-
Notifications
You must be signed in to change notification settings - Fork 2
MistifyOS Logging
The logging setup of MistifyOS shall be seen, generally, as two distinct components:
- Log Emitters, such as hypervisor nodes and possibly its guests.
- Log Receivers, of which there can be several, depending on site requirements.
All supported logs generated within the broader MistifyOS Cluster will be received, in real-time, on dedicated Log Receiver nodes. The over-the-wire transport means may be via syslog over UDP, TCP, or RFC 3195 in RAW mode. While a single Log Receiver node is sufficient, multiple log receivers should be provisionable within a MistifyOS Cluster to attain fault-tolerance of the logging infrastructure.
A basic Log Receiver node shall ingest syslog or other logging protocol traffic directed to it and store the logs in a predicable filesystem directory structure. The received logs will reside on the configured Mistify ZFS pool and be compressed transparently by ZFS.
On both Log Emitters and Log Receivers, the open source rsyslog daemon will serve as the main collector, receiver, and router of logs. The rsyslog version chosen is 8.7.0.
rsyslog is a modern, performant, multi-threaded and fully-featured logging server which supports a multitude of protocols and endpoint types. It may receive logs via traditional syslog UDP and TCP, or by more advanced protocols such as RFC 3195, RELP, and systemd's journald subsystem.
On the output side, rsyslog can store received logs in traditional log files and send to UNIX pipes, or relay them directly to higher-level services such as ElasticSearch, Hadoop, Redis, Postgres, Oracle, and MySQL.
rsyslog may also act as a simple syslog relay and protocol convertor. An example use case for this would be integration of a MistifyOS Cluster into existing log processing and storage infrastructure.
rsyslog integration with Windows' EventLog is available via commercial third-party agent software. This can be an option for Windows guests operating within a MistifyOS Cluster, but it is one we will not directly support due to its commercial, third-party nature.
Configuration of rsyslogd is done through the /etc/rsyslog.conf file, with automatic inclusion of additional .conf files from the /etc/rsyslog.d directory.
A MistifyOS node and its various components log items in three main ways:
syslog()- writing directly to a file
-
stdoutandstderr
rsyslog will enable us to capture standard system syslog messages (ie, /var/log/messages, authlog, etc) and directly-written application logs, including those produced by the Mistify Agent and its sub-agents. For directly-written log files, rsyslog can be configured to monitor an entire directory, or files which match a regex, and automatically monitor them.
Services running under systemd may log to stdout and/or stderr, and these will be caught by systemd via its journald component and then directly read in by rsyslog.
Guests running within MistifyOS Cluster hypervisors may also be configured, via their own software, to log to the MistifyOS Cluster Log Receivers. The Log Receivers should be smart enough to differentiate between logs emitted from MistifyOS Cluster nodes, and those emitted from Guests running within the MistifyOS Cluster hypervisors.
The Mistify Agent will be responsible for configuring rsyslog to forward the logs it receives to the Log Receivers.
A basic Log Receiver setup will comprise of a MistifyOS node which is configured to operate in that role. This role is to be configured by the Mistify Agent and, in its basic form, should require no manual intervention to attain. This role should be exclusive to the node, however further study should be done to ascertain whether it would be prudent to allow a Log Receiver node to concurrently act in other Cluster roles, such as a Hypervisor or Image Server.
Basic storage of received logs will be done through simple text files in a directory structure that is centered around the node hostname. rsyslog can be programmed to automatically create any required directories and files through the use of its RainerScript templates, so manual intervention when adding new nodes is kept to minimum, if any at all.
An example rsyslog.conf template directive to do this is:
#####################################################
# Log everything to a per host daily logfile #
#####################################################
$template DailyPerHostLogs,"/mistify/logs/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/messages.log"
*.* -?DailyPerHostLogs
A separate cron job may run daily to remove log files older than N days.
The ability to configure more advanced storage mechanisms should be provided for, as MistifyOS Cluster users may wish to use ElasticSearch or Hadoop clusters to further process or store logs. These specific items, however, will have to be provided for and managed external to the MistifyOS Cluster.
Log rotation should be implemented on both the Log Emitter nodes and the Log Receiver nodes. Log rotation is built-in to rsyslog, and can be achieved in two ways, either via date-based rotation through dynamic, date-based log file paths, or via size, upon which a specified external action may be performed (eg; mv logfile1 logfile2)
rsyslog supports three configuration languages:
- Traditional
/etc/syslog.confsyntax - A slightly more flexible rsyslog dialect of the above
- An advanced and extensible "RainerScript" language
It is the goal of this project to implement all rsyslog logging directives in RainerScript. This will allow us the most flexibility, especially on Log Receiver nodes, to configure templates based on external information and apply actions and rules to certain logs.