logging: consider moving away from cluster settings and into logging filters

[knz]

Today we have certain notable events for which we use a cluster setting to enable/disable the logging. This includes:

• create_statistics, controlled by sql.stats.post_events.enabled (see eventpb: document that CreateStatistics is only reported conditionally #57877)
• client_connection_start / client_connection_end, controlled by server.auth_log.sql_connections.enabled (see pgwire: migrate auth/conn logs to notable events #57839)
• client_session_end, client_authorization_*, controlled by server.auth_log.sql_sessions.enabled (see pgwire: migrate auth/conn logs to notable events #57839 )

These logging events are only logged conditionally because they are otherwise rather frequent, and we did not want them to "pollute" logs or incur excessive disk IOPS when logging to files.

However, this is inconvenient for CC logging, where we likely do want to see these events collected and instead filter them "elsewhere" (in a network log collector). For CC clusters, this is extra inconvenient because it requires some additional cluster settings to be set before a CC cluster becomes operational, which is error-prone (both due to human mistakes and additional software errors).

Instead, we could consider have the internal CockroachDB components log them unconditionally, and have a single general-purpose filtering option in the logging system:

• when logging to files exclusively, the filter would be pre-set to exclude these event types
• when logging to the network, the filter would not be pre-set so all events are routed to the network.

The filter could be like a list of logging events to exclude, per sink. For example in YAML:

sinks:
  file-groups:
    default:
      exclude-events: create_statistics, client_connection_start, client_connection_end

This additional mechanism would also make it possible for end-users to opt in and out of logging certain notable events without requiring us to add cluster settings for each of them every time.

@aaron-crl thoughts? @thtruo ?

Jira issue: CRDB-3472

[knz]

cc @logston I think you'll want to learn this

[knz]

In fact as per @tbg we may need to do so urgently, because in multi-tenant individual tenants cannot set cluster settings. So if we want these logs they can't be controlled by cluster settings for now.

[tbg]

What we can always do is override individual cluster settings from the env in start-sql.

[knz]

What we can always do is override individual cluster settings from the env in start-sql.

Is this override mechanism already implemented? If so, where can I find more information about it?

[tbg]

Settings have a .Override method, which is what I was referring to.
This isn't the right option if we start letting tenants listen to their settings table, of course. Then, nothing else would be required.

[knz]

ok so if nothing else we can implement a new override mechanism in the mt start-sql CLI flag handling. That'd be a temporary workaround, but it doesn't hurt.

[aaron-crl]

Logging filters sound like a good strategy. That could also potentially allow end users to tune it such that events we don't traditionally think of as security events go to a custom sink for their own consumption.

[thtruo]

Echoing Aaron's sentiment, moving towards logging filters and away from setting clusters sounds like a great approach. The usability improvement around managing logs in a single place (at log configuration) instead of managing knobs at multiple places will lead to a quality of life improvement for users. I also like your idea of an exclusion list too because it gives the end user more flexibility and control around what events go where.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!