Skip to content

Commit

Permalink
Merge pull request #7 from veqryn/replace-resolve-keys
Browse files Browse the repository at this point in the history 
Replace resolve keys
  • Loading branch information
@veqryn
veqryn authored Mar 21, 2024
2 parents 637ced9 + e470dd2 commit cebf505
Show file tree
Hide file tree
Showing 14 changed files with 882 additions and 354 deletions.
139 changes: 127 additions & 12 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,29 @@ Outputs:
"duplicated": "two"
}
```
With this in mind, this repo was created with several different ways of deduplicating the keys.
With this in mind, this repo was created with several different ways of deduplicating the keys: overwriting, ignoring, appending, incrementing.
For example, incrementing:
```json
{
"time": "2024-03-21T09:33:25Z",
"level": "INFO",
"msg": "this is the dedup incrementer handler",
"duplicated": "zero",
"duplicated#01": "one",
"duplicated#02": "two"
}
```

Additionally, this library includes convenience methods for formatting output to
match what is expected for various log aggregation tools (such as Graylog), as
well as cloud providers (such as Stackdriver / Google Cloud Operations / GCP Log Explorer).

### Other Great SLOG Utilities
- [slogctx](https://github.com/veqryn/slog-context): Add attributes to context and have them automatically added to all log lines. Work with a logger stored in context.
- [slogotel](https://github.com/veqryn/slog-context/tree/main/otel): Automatically extract and add [OpenTelemetry](https://opentelemetry.io/) TraceID's to all log lines.
- [slogdedup](https://github.com/veqryn/slog-dedup): Middleware that deduplicates and sorts attributes. Particularly useful for JSON logging.
- [slogbugsnag](https://github.com/veqryn/slog-bugsnag): Middleware that pipes Errors to [Bugsnag](https://www.bugsnag.com/).
- [slogjson](https://github.com/veqryn/slog-json): Formatter that uses the [JSON v2](https://github.com/golang/go/discussions/63397) [library](https://github.com/go-json-experiment/json), with optional single-line pretty-printing.

## Install
`go get github.com/veqryn/slog-dedup`
Expand All @@ -65,7 +81,7 @@ logger.Info("this is the dedup overwrite handler",
Outputs:
```json
{
"time": "2023-10-03T01:30:00Z",
"time": "2024-03-21T09:33:25Z",
"level": "INFO",
"msg": "this is the dedup overwrite handler",
"duplicated": "two"
Expand All @@ -84,7 +100,7 @@ logger.Info("this is the dedup ignore handler",
Outputs:
```json
{
"time": "2023-10-03T01:30:00Z",
"time": "2024-03-21T09:33:25Z",
"level": "INFO",
"msg": "this is the dedup ignore handler",
"duplicated": "zero"
Expand All @@ -103,7 +119,7 @@ logger.Info("this is the dedup incrementer handler",
Outputs:
```json
{
"time": "2023-10-03T01:30:00Z",
"time": "2024-03-21T09:33:25Z",
"level": "INFO",
"msg": "this is the dedup incrementer handler",
"duplicated": "zero",
Expand All @@ -124,7 +140,7 @@ logger.Info("this is the dedup appender handler",
Outputs:
```json
{
"time": "2023-10-03T01:30:00Z",
"time": "2024-03-21T09:33:25Z",
"level": "INFO",
"msg": "this is the dedup appender handler",
"duplicated": [
Expand All @@ -135,7 +151,35 @@ Outputs:
}
```

### Using ResolveKey and ReplaceAttr
#### Stackdriver (GCP Cloud Operations / Google Log Explorer)
```go
logger := slog.New(slogdedup.NewOverwriteHandler(
slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
AddSource: true,
ReplaceAttr: slogdedup.ReplaceAttrStackdriver(), // Needed for builtin's
}),
&slogdedup.OverwriteHandlerOptions{ResolveKey: slogdedup.ResolveKeyStackdriver()}, // Needed for everything else, and deduplication
))
logger.Warn("this is the main message", slog.String("duplicated", "zero"), slog.String("duplicated", "one"))
```
Outputs:
```json
{
"time": "2024-03-21T09:59:19.652284-06:00",
"severity": "WARNING",
"logging.googleapis.com/sourceLocation": {
"function": "main.main",
"file": "/go/src/github.com/veqryn/slog-dedup/cmd/replacers/cmd.go",
"line": "19"
},
"message": "this is the main message",
"duplicated": "one"
}
```

## Full Example Main File
### Basic Use of Each Handler
```go
package main

Expand All @@ -153,7 +197,7 @@ func main() {

/*
{
"time": "2023-10-03T01:30:00Z",
"time": "2024-03-21T09:33:25Z",
"level": "INFO",
"msg": "this is the dedup overwrite handler",
"duplicated": "two"
Expand All @@ -171,7 +215,7 @@ func main() {

/*
{
"time": "2023-10-03T01:30:00Z",
"time": "2024-03-21T09:33:25Z",
"level": "INFO",
"msg": "this is the dedup ignore handler",
"duplicated": "zero"
Expand All @@ -189,7 +233,7 @@ func main() {

/*
{
"time": "2023-10-03T01:30:00Z",
"time": "2024-03-21T09:33:25Z",
"level": "INFO",
"msg": "this is the dedup incrementer handler",
"duplicated": "zero",
Expand All @@ -209,7 +253,7 @@ func main() {

/*
{
"time": "2023-10-03T01:30:00Z",
"time": "2024-03-21T09:33:25Z",
"level": "INFO",
"msg": "this is the dedup appender handler",
"duplicated": [
Expand All @@ -227,8 +271,79 @@ func main() {
}
```

### Using ResolveKey and ReplaceAttr
```go
package main

import (
"log/slog"
"os"

slogdedup "github.com/veqryn/slog-dedup"
)

func main() {
// Example sending logs of Stackdriver (GCP Cloud Operations / Google Log Explorer)
// which then pipes the data to Graylog.

// First, create a function to resolve/replace attribute keys before deduplication,
// which will also ensure the builtin keys are unused by non-builtin attributes:
resolveKey := slogdedup.JoinResolveKey(
slogdedup.ResolveKeyStackdriver(),
slogdedup.ResolveKeyGraylog(),
)

// Second, create a function to replace the builtin record attributes
// (time, level, msg, source) with the appropriate keys and values for
// Stackdriver and Graylog:
replaceAttr := slogdedup.JoinReplaceAttr(
slogdedup.ReplaceAttrStackdriver(),
slogdedup.ReplaceAttrGraylog(),
)

// Next create the final handler (the sink), which is a json handler,
// using the replaceAttr function we just made:
jsonHandler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
AddSource: true,
Level: slog.LevelDebug,
ReplaceAttr: replaceAttr,
})

// Now create any one of the dedup middleware handlers, using the
// resolveKey function we just made, then set it as the default logger:
dedupHandler := slogdedup.NewOverwriteHandler(
jsonHandler,
&slogdedup.OverwriteHandlerOptions{
KeyCompare: slogdedup.CaseSensitiveCmp, // this is the default
ResolveKey: resolveKey, // use the key resolver for stackdriver/graylog
},
)
slog.SetDefault(slog.New(dedupHandler))

/*
{
"time": "2024-03-21T09:53:43Z",
"severity": "WARNING",
"logging.googleapis.com/sourceLocation": {
"function": "main.main",
"file": "/go/src/github.com/veqryn/slog-dedup/cmd/replacers/cmd.go",
"line": "56"
},
"message": "this is the main message",
"duplicated": "one"
}
*/
slog.Warn("this is the main message", slog.String("duplicated", "zero"), slog.String("duplicated", "one"))
}
```

## Breaking Changes
### O.1.0 -> 0.2.0
### O.3.x -> 0.4.0
`ResolveBuiltinKeyConflict`,`DoesBuiltinKeyConflict`, and `IncrementKeyName` have
all been unified into a single function: `ResolveKey`.


### O.1.x -> 0.2.0
Package renamed from `dedup` to `slogdedup`.
To fix, change this:
```go
Expand All @@ -248,7 +363,7 @@ This library has convenience methods that allow it to interoperate with [github.
in order to easily setup slog workflows such as pipelines, fanout, routing, failover, etc.
```go
slog.SetDefault(slog.New(slogmulti.
Pipe(slogcontext.NewMiddleware(&slogcontext.HandlerOptions{})).
Pipe(slogctx.NewMiddleware(&slogctx.HandlerOptions{})).
Pipe(slogdedup.NewOverwriteMiddleware(&slogdedup.OverwriteHandlerOptions{})).
Handler(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{})),
))
Expand All @@ -261,4 +376,4 @@ Using an overwrite handler allows a slightly different style of logging that is
These handlers will correctly deal with sub-loggers, whether using `WithAttrs()` or `WithGroup()`. It will even handle groups injected as attributes using `slog.Group()`. Due to the lack of a `slog.Slice` type/kind, the `AppendHandler` has a special case where groups that are inside of slices/arrays are turned into a `map[string]any{}` slog attribute before being passed to the final handler.

### The Built-In Fields (time, level, msg, source)
Because this handler is a middleware, it must pass a `slog.Record` to the final handler. The built-in attributes for time, level, msg, and source are treated separately, and have their own fields on the `slog.Record` struct. It would therefore be impossible to deduplicate these, if we didn't handle these as a special case. The increment handler considers that these four keys are always taken at the root level, and any attributes using those keys will start with the #01 increment on their key name. The other handlers can be customized using their options struct to either increment the name (default), overwrite, or allow the duplicates for the builtin keys. You can also customize this behavior by passing your own functions to the options struct (same for log handlers that use different keys for the built-in fields).
Because this handler is a middleware, it must pass a `slog.Record` to the final handler. The built-in attributes for time, level, msg, and source are treated separately, and have their own fields on the `slog.Record` struct. It would therefore be impossible to deduplicate these, if we didn't handle these as a special case. The increment handler considers that these four keys are always taken at the root level, and any attributes using those keys will start with the #01 increment on their key name. The other handlers can be customized using their options struct to either increment the name (default), drop old attributes using those keys (overwrite with the final slog.Record builtins), or allow the duplicates for the builtin keys. You can also customize this behavior by passing your own functions to the options struct (same for log handlers that use different keys for the built-in fields).
44 changes: 26 additions & 18 deletions append_handler.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package slogdedup
import (
"context"
"log/slog"
"slices"

"modernc.org/b/v2"
)
Expand All @@ -12,10 +13,17 @@ type AppendHandlerOptions struct {
// Comparison function to determine if two keys are equal
KeyCompare func(a, b string) int

// Function that will be called on all root level (not in a group) attribute keys.
// Returns the new key value to use, and true to keep the attribute or false to drop it.
// Can be used to drop, keep, or rename any attributes matching the builtin attributes.
ResolveBuiltinKeyConflict func(k string) (string, bool)
// Function that will be called on each attribute and group, to determine
// the key to use. Returns the new key value to use, and true to keep the
// attribute or false to drop it. Can be used to drop, keep, or rename any
// attributes matching the builtin attributes.
//
// The first argument is a list of currently open groups that contain the
// Attr. It must not be retained or modified.
//
// ResolveKey will not be called for the built-in fields on slog.Record
// (ie: time, level, msg, and source).
ResolveKey func(groups []string, key string, _ int) (string, bool)
}

// AppendHandler is a slog.Handler middleware that will deduplicate all attributes and
Expand All @@ -25,7 +33,7 @@ type AppendHandler struct {
next slog.Handler
goa *groupOrAttrs
keyCompare func(a, b string) int
getKey func(key string, depth int) (string, bool)
resolveKey func(groups []string, key string, _ int) (string, bool)
}

var _ slog.Handler = &AppendHandler{} // Assert conformance with interface
Expand Down Expand Up @@ -59,14 +67,14 @@ func NewAppendHandler(next slog.Handler, opts *AppendHandlerOptions) *AppendHand
if opts.KeyCompare == nil {
opts.KeyCompare = CaseSensitiveCmp
}
if opts.ResolveBuiltinKeyConflict == nil {
opts.ResolveBuiltinKeyConflict = IncrementIfBuiltinKeyConflict
if opts.ResolveKey == nil {
opts.ResolveKey = IncrementIfBuiltinKeyConflict
}

return &AppendHandler{
next: next,
keyCompare: opts.KeyCompare,
getKey: getKeyClosure(opts.ResolveBuiltinKeyConflict),
resolveKey: opts.ResolveKey,
}
}

Expand All @@ -89,7 +97,7 @@ func (h *AppendHandler) Handle(ctx context.Context, r slog.Record) error {

// Resolve groups and with-attributes
uniq := b.TreeNew[string, any](h.keyCompare)
h.createAttrTree(uniq, goas, 0)
h.createAttrTree(uniq, goas, nil)

// Add all attributes to new record (because old record has all the old attributes)
newR := &slog.Record{
Expand Down Expand Up @@ -121,16 +129,16 @@ func (h *AppendHandler) WithAttrs(attrs []slog.Attr) slog.Handler {

// createAttrTree recursively goes through all groupOrAttrs, resolving their attributes and creating subtrees as
// necessary, adding the results to the map
func (h *AppendHandler) createAttrTree(uniq *b.Tree[string, any], goas []*groupOrAttrs, depth int) {
func (h *AppendHandler) createAttrTree(uniq *b.Tree[string, any], goas []*groupOrAttrs, groups []string) {
if len(goas) == 0 {
return
}

// If a group is encountered, create a subtree for that group and all groupOrAttrs after it
if goas[0].group != "" {
if key, keep := h.getKey(goas[0].group, depth); keep {
if key, keep := h.resolveKey(groups, goas[0].group, 0); keep {
uniqGroup := b.TreeNew[string, any](h.keyCompare)
h.createAttrTree(uniqGroup, goas[1:], depth+1)
h.createAttrTree(uniqGroup, goas[1:], append(slices.Clip(groups), key))
// Ignore empty groups, otherwise put subtree into the map
if uniqGroup.Len() > 0 {
// Put calls func(oldValue, true) if key already exists, or func(oldValue, false) if it doesn't.
Expand All @@ -151,15 +159,15 @@ func (h *AppendHandler) createAttrTree(uniq *b.Tree[string, any], goas []*groupO
}

// Otherwise, set all attributes for this groupOrAttrs, and then call again for remaining groupOrAttrs's
h.resolveValues(uniq, goas[0].attrs, depth)
h.createAttrTree(uniq, goas[1:], depth)
h.resolveValues(uniq, goas[0].attrs, groups)
h.createAttrTree(uniq, goas[1:], groups)
}

// resolveValues iterates through the attributes, resolving them and putting them into the map.
// If a group is encountered (as an attribute), it will be separately resolved and added as a subtree.
// Since attributes are ordered from oldest to newest, it creates a slice whenever it detects the key already exists,
// appending the new attribute, then overwriting the key with that slice.
func (h *AppendHandler) resolveValues(uniq *b.Tree[string, any], attrs []slog.Attr, depth int) {
func (h *AppendHandler) resolveValues(uniq *b.Tree[string, any], attrs []slog.Attr, groups []string) {
var keep bool
for _, a := range attrs {
a.Value = a.Value.Resolve()
Expand All @@ -168,7 +176,7 @@ func (h *AppendHandler) resolveValues(uniq *b.Tree[string, any], attrs []slog.At
}

// Default situation: resolve the key and put it into the map
a.Key, keep = h.getKey(a.Key, depth)
a.Key, keep = h.resolveKey(groups, a.Key, 0)
if !keep {
continue
}
Expand All @@ -189,13 +197,13 @@ func (h *AppendHandler) resolveValues(uniq *b.Tree[string, any], attrs []slog.At

// Groups with empty keys are inlined
if a.Key == "" {
h.resolveValues(uniq, a.Value.Group(), depth)
h.resolveValues(uniq, a.Value.Group(), groups)
continue
}

// Create a subtree for this group
uniqGroup := b.TreeNew[string, any](h.keyCompare)
h.resolveValues(uniqGroup, a.Value.Group(), depth+1)
h.resolveValues(uniqGroup, a.Value.Group(), append(slices.Clip(groups), a.Key))

// Ignore empty groups, otherwise put subtree into the map
if uniqGroup.Len() > 0 {
Expand Down
Loading

0 comments on commit cebf505

Please sign in to comment.