Logger options in riak.conf

[martinsumner]

The current riak.conf schema allows the log format to modified within loggeer, as well as the destination.

The logger configuraiton that gets produced is not, though, backwards compatible with previous releases - i.e. crash.log etc.

There exists riak_logger which will allow for a backwards compatible configuration to be provided. So it would be good if it were possible to produce this output from the default riak.conf (and for it to be the default).

There is also the issue of backend logging, which can be relatively verbose when LSM trees are used. For eleveldb this logs to a file within the database itself, however the leveled backend prints to the erlang.log (using io:format/2). As part of release 3.4, leveled will log via logger - however, it should be configurable where it should log to i.e. there should be two options:

• log - log to console.log as with other logs;
• divert (default) - log to a backend.log (whose location is configurable) to keep these logs separate from other riak logs.

There will be a backend.policy for selecting these options.

Likewise, TictacAAE exchanges are relatively verbose - this was a conscious decision prompted by a frustration with the lack of logging within the old AAE system. So likewise these should support log/divert options.

A number of long-lived process have been added to Riak in the past few years, where these processes heartbeat into logs metrics on a regular basis. This was to fit-in with an observability policy at the customer funding the changes, but is not to everyone's liking. So there should also be a metric.policy - that supports the log and divert options, but also an ignore option to not log heartbeat metrics at all.

The WIP for changing riak_logger to support this is here.

This can then be added to riak.conf as here.

The basic approach is to add a log_type element to the log metadata, and use that log_type to include/exclude these logs from different handlers.

There are some outstanding questions:

• Is the configurability of log format helpful? It may make support easier if the log format is fixed.
• Should it be possible to ignore backend and tictcaaae logs.
• How to test different configurations, is it enough to expand out eunit tests that confirm the configuration outputted from the function is as expected? It is hard to test anything driven by cuttlefish in riak_test.
• Leveled can have its log level set separately to the rest of Riak (https://github.com/martinsumner/leveled/blob/develop-3.4/priv/leveled.schema#L83-L96), but it also has the ability to change the level of individual log references (so that one can set the level to warn and promote interesting info log references to warn at startup). This isn't configurable though (really hard to do in cuttlefish) - should it be supported (even if only via advanced.config)?