Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(pkg/ottl) Adding an optional replacementFormat argument to the replace_pattern editors that specified the format of the replacement string #27686

Closed
wants to merge 2 commits into from
Closed

feat(pkg/ottl) Adding an optional replacementFormat argument to the replace_pattern editors that specified the format of the replacement string #27686

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
336226d
feat(pkg/ottl) Adding an optional replacement prefix argument to the …
rnishtala-sumo Oct 16, 2023
ee00bae
Change the replacement prefix string to be a format string, this allo…
rnishtala-sumo Oct 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .chloggen/ottl-replace-pattern.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ change_type: enhancement
component: pkg/ottl

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Add optional Converter parameters to replacement Editors
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this updating an existing unpublished feature?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there will be a new changelog for this. Will revert this change.

note: Add optional parameters to replacement Editors

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [27235]
Expand All @@ -21,6 +21,7 @@ subtext: |
- `replace_all_patterns`
- `replace_match`
- `replace_all_matches`
An optional replacement prefix argument can also be passed to `replace_pattern` and `replace_all_patterns` to be prepended to the replacement string.

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
Expand Down
13 changes: 7 additions & 6 deletions pkg/ottl/ottlfuncs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ Examples:

### replace_all_patterns

`replace_all_patterns(target, mode, regex, replacement, function)`
`replace_all_patterns(target, mode, regex, replacement, replacementFormat, function)`

The `replace_all_patterns` function replaces any segments in a string value or key that match the regex pattern with the replacement string.

Expand All @@ -175,7 +175,7 @@ The `replace_all_patterns` function replaces any segments in a string value or k

If one or more sections of `target` match `regex` they will get replaced with `replacement`.

The `replacement` string can refer to matched groups using [regexp.Expand syntax](https://pkg.go.dev/regexp#Regexp.Expand).
The `replacement` string can refer to matched groups using [regexp.Expand syntax](https://pkg.go.dev/regexp#Regexp.Expand). `replacementFormat` is an optional string argument that specifies the format of the replacement.

The `function` is an optional argument that can take in any Converter that accepts a (`replacement`) string and returns a string. An example is a hash function that replaces any matching regex pattern with the hash value of `replacement`.

Expand All @@ -188,7 +188,8 @@ Examples:
- `replace_all_patterns(attributes, "value", "/account/\\d{4}", "/account/{accountId}")`
- `replace_all_patterns(attributes, "key", "/account/\\d{4}", "/account/{accountId}")`
- `replace_all_patterns(attributes, "key", "^kube_([0-9A-Za-z]+_)", "k8s.$$1.")`
- `replace_all_patterns(attributes, "key", "^kube_([0-9A-Za-z]+_)", "k8s.$$1.", SHA256)`
- `replace_all_patterns(attributes, "key", "^kube_([0-9A-Za-z]+_)", "$$1.", "k8s.%s")`
- `replace_all_patterns(attributes, "key", "^kube_([0-9A-Za-z]+_)", "$$1.", "k8s.%s", SHA256)`

Note that when using OTTL within the collector's configuration file, `$` must be escaped to `$$` to bypass
environment variable substitution logic. To input a literal `$` from the configuration file, use `$$$`.
Expand Down Expand Up @@ -216,15 +217,15 @@ Examples:

### replace_pattern

`replace_pattern(target, regex, replacement, function)`
`replace_pattern(target, regex, replacement, replacementFormat, function)`

The `replace_pattern` function allows replacing all string sections that match a regex pattern with a new value.

`target` is a path expression to a telemetry field. `regex` is a regex string indicating a segment to replace. `replacement` is either a path expression to a string telemetry field or a literal string.

If one or more sections of `target` match `regex` they will get replaced with `replacement`.

The `replacement` string can refer to matched groups using [regexp.Expand syntax](https://pkg.go.dev/regexp#Regexp.Expand).
The `replacement` string can refer to matched groups using [regexp.Expand syntax](https://pkg.go.dev/regexp#Regexp.Expand). `replacementFormat` is an optional string argument that specifies the format of the replacement.

The `function` is an optional argument that can take in any Converter that accepts a (`replacement`) string and returns a string. An example is a hash function that replaces a matching regex pattern with the hash value of `replacement`.

Expand All @@ -236,7 +237,7 @@ Examples:

- `replace_pattern(resource.attributes["process.command_line"], "password\\=[^\\s]*(\\s?)", "password=***")`
- `replace_pattern(name, "^kube_([0-9A-Za-z]+_)", "k8s.$$1.")`
- `replace_pattern(name, "^kube_([0-9A-Za-z]+_)", "k8s.$$1.", SHA256)`
- `replace_pattern(name, "^kube_([0-9A-Za-z]+_)", "$$1.", "k8s.%s", SHA256)`

Note that when using OTTL within the collector's configuration file, `$` must be escaped to `$$` to bypass
environment variable substitution logic. To input a literal `$` from the configuration file, use `$$$`.
Expand Down
30 changes: 23 additions & 7 deletions pkg/ottl/ottlfuncs/func_replace_all_patterns.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"context"
"fmt"
"regexp"
"strings"

"go.opentelemetry.io/collector/pdata/pcommon"

Expand All @@ -19,11 +20,12 @@ const (
)

type ReplaceAllPatternsArguments[K any] struct {
Target ottl.PMapGetter[K]
Mode string
RegexPattern string
Replacement ottl.StringGetter[K]
Function ottl.Optional[ottl.FunctionGetter[K]]
Target ottl.PMapGetter[K]
Mode string
RegexPattern string
Replacement ottl.StringGetter[K]
ReplacementFormat ottl.Optional[string]
Function ottl.Optional[ottl.FunctionGetter[K]]
}

type replaceAllPatternFuncArgs[K any] struct {
Expand All @@ -41,10 +43,10 @@ func createReplaceAllPatternsFunction[K any](_ ottl.FunctionContext, oArgs ottl.
return nil, fmt.Errorf("ReplaceAllPatternsFactory args must be of type *ReplaceAllPatternsArguments[K]")
}

return replaceAllPatterns(args.Target, args.Mode, args.RegexPattern, args.Replacement, args.Function)
return replaceAllPatterns(args.Target, args.Mode, args.RegexPattern, args.Replacement, args.ReplacementFormat, args.Function)
}

func replaceAllPatterns[K any](target ottl.PMapGetter[K], mode string, regexPattern string, replacement ottl.StringGetter[K], fn ottl.Optional[ottl.FunctionGetter[K]]) (ottl.ExprFunc[K], error) {
func replaceAllPatterns[K any](target ottl.PMapGetter[K], mode string, regexPattern string, replacement ottl.StringGetter[K], replacementFormat ottl.Optional[string], fn ottl.Optional[ottl.FunctionGetter[K]]) (ottl.ExprFunc[K], error) {
compiledPattern, err := regexp.Compile(regexPattern)
if err != nil {
return nil, fmt.Errorf("the regex pattern supplied to replace_all_patterns is not a valid pattern: %w", err)
Expand Down Expand Up @@ -86,13 +88,27 @@ func replaceAllPatterns[K any](target ottl.PMapGetter[K], mode string, regexPatt
switch mode {
case modeValue:
if compiledPattern.MatchString(originalValue.Str()) {
if !replacementFormat.IsEmpty() { // If replacementFormat is not empty, add it to the replacement value
formatString := replacementFormat.Get()
if !strings.Contains(formatString, "%s") {
return false
}
replacementVal = fmt.Sprintf(replacementFormat.Get(), replacementVal)
}
updatedString := compiledPattern.ReplaceAllString(originalValue.Str(), replacementVal)
updated.PutStr(key, updatedString)
} else {
originalValue.CopyTo(updated.PutEmpty(key))
}
case modeKey:
if compiledPattern.MatchString(key) {
if !replacementFormat.IsEmpty() { // If replacementFormat is not empty, add it to the replacement value
formatString := replacementFormat.Get()
if !strings.Contains(formatString, "%s") {
return false
}
replacementVal = fmt.Sprintf(replacementFormat.Get(), replacementVal)
}
updatedKey := compiledPattern.ReplaceAllString(key, replacementVal)
originalValue.CopyTo(updated.PutEmpty(updatedKey))
} else {
Expand Down
103 changes: 79 additions & 24 deletions pkg/ottl/ottlfuncs/func_replace_all_patterns_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,13 +39,14 @@ func Test_replaceAllPatterns(t *testing.T) {
}

tests := []struct {
name string
target ottl.PMapGetter[pcommon.Map]
mode string
pattern string
replacement ottl.StringGetter[pcommon.Map]
function ottl.Optional[ottl.FunctionGetter[pcommon.Map]]
want func(pcommon.Map)
name string
target ottl.PMapGetter[pcommon.Map]
mode string
pattern string
replacement ottl.StringGetter[pcommon.Map]
replacementFormat ottl.Optional[string]
function ottl.Optional[ottl.FunctionGetter[pcommon.Map]]
want func(pcommon.Map)
}{
{
name: "replace only matches (with hash function)",
Expand All @@ -57,7 +58,8 @@ func Test_replaceAllPatterns(t *testing.T) {
return "hello {universe}", nil
},
},
function: optionalArg,
replacementFormat: ottl.Optional[string]{},
function: optionalArg,
want: func(expectedMap pcommon.Map) {
expectedMap.PutStr("test", "4804d6b7f03268e33f78c484977f3d81771220df07cc6aac4ad4868102141fad world")
expectedMap.PutStr("test2", "4804d6b7f03268e33f78c484977f3d81771220df07cc6aac4ad4868102141fad")
Expand All @@ -67,6 +69,44 @@ func Test_replaceAllPatterns(t *testing.T) {
expectedMap.PutBool("test6", true)
},
},
{
name: "replace only matches (with replacement format)",
target: target,
mode: modeValue,
pattern: "hello",
replacement: ottl.StandardStringGetter[pcommon.Map]{
Getter: func(context.Context, pcommon.Map) (interface{}, error) {
return "hello {universe}", nil
},
},
replacementFormat: ottl.NewTestingOptional[string]("passwd=%s"),
function: optionalArg,
want: func(expectedMap pcommon.Map) {
expectedMap.PutStr("test", "passwd=4804d6b7f03268e33f78c484977f3d81771220df07cc6aac4ad4868102141fad world")
expectedMap.PutStr("test2", "passwd=passwd=4804d6b7f03268e33f78c484977f3d81771220df07cc6aac4ad4868102141fad") // replacement format is applied twice
expectedMap.PutStr("test3", "goodbye world1 and world2")
expectedMap.PutInt("test4", 1234)
expectedMap.PutDouble("test5", 1234)
expectedMap.PutBool("test6", true)
},
},
{
name: "replace only matches (with invalid replacement format)",
target: target,
mode: modeValue,
pattern: "hello",
replacement: ottl.StandardStringGetter[pcommon.Map]{
Getter: func(context.Context, pcommon.Map) (interface{}, error) {
return "hello {universe}", nil
},
},
replacementFormat: ottl.NewTestingOptional[string]("passwd="),
function: optionalArg,
want: func(expectedMap pcommon.Map) {
expectedMap.PutEmpty("test")
expectedMap.Remove("test")
},
},
{
name: "replace only matches",
target: target,
Expand All @@ -77,7 +117,8 @@ func Test_replaceAllPatterns(t *testing.T) {
return "hello {universe}", nil
},
},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
replacementFormat: ottl.Optional[string]{},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
want: func(expectedMap pcommon.Map) {
expectedMap.PutStr("test", "hello {universe} world")
expectedMap.PutStr("test2", "hello {universe}")
Expand All @@ -97,7 +138,8 @@ func Test_replaceAllPatterns(t *testing.T) {
return "nothing {matches}", nil
},
},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
replacementFormat: ottl.Optional[string]{},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
want: func(expectedMap pcommon.Map) {
expectedMap.PutStr("test", "hello world")
expectedMap.PutStr("test2", "hello")
Expand All @@ -117,7 +159,8 @@ func Test_replaceAllPatterns(t *testing.T) {
return "**** ", nil
},
},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
replacementFormat: ottl.Optional[string]{},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
want: func(expectedMap pcommon.Map) {
expectedMap.PutStr("test", "hello **** ")
expectedMap.PutStr("test2", "hello")
Expand All @@ -137,7 +180,8 @@ func Test_replaceAllPatterns(t *testing.T) {
return "foo", nil
},
},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
replacementFormat: ottl.Optional[string]{},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
want: func(expectedMap pcommon.Map) {
expectedMap.Clear()
expectedMap.PutStr("test", "hello world")
Expand All @@ -158,7 +202,8 @@ func Test_replaceAllPatterns(t *testing.T) {
return "nothing {matches}", nil
},
},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
replacementFormat: ottl.Optional[string]{},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
want: func(expectedMap pcommon.Map) {
expectedMap.Clear()
expectedMap.PutStr("test", "hello world")
Expand All @@ -179,7 +224,8 @@ func Test_replaceAllPatterns(t *testing.T) {
return "test.", nil
},
},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
replacementFormat: ottl.Optional[string]{},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
want: func(expectedMap pcommon.Map) {
expectedMap.Clear()
expectedMap.PutStr("test.", "hello world")
Expand All @@ -200,7 +246,8 @@ func Test_replaceAllPatterns(t *testing.T) {
return "world-$1", nil
},
},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
replacementFormat: ottl.Optional[string]{},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
want: func(expectedMap pcommon.Map) {
expectedMap.Clear()
expectedMap.PutStr("test", "hello world")
Expand All @@ -221,7 +268,8 @@ func Test_replaceAllPatterns(t *testing.T) {
return "test-$1", nil
},
},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
replacementFormat: ottl.Optional[string]{},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
want: func(expectedMap pcommon.Map) {
expectedMap.PutStr("test", "hello world")
expectedMap.PutStr("test-2", "hello")
Expand All @@ -241,7 +289,8 @@ func Test_replaceAllPatterns(t *testing.T) {
return "$$world-$1", nil
},
},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
replacementFormat: ottl.Optional[string]{},
function: ottl.Optional[ottl.FunctionGetter[pcommon.Map]]{},
want: func(expectedMap pcommon.Map) {
expectedMap.Clear()
expectedMap.PutStr("test", "hello world")
Expand All @@ -258,7 +307,7 @@ func Test_replaceAllPatterns(t *testing.T) {
scenarioMap := pcommon.NewMap()
input.CopyTo(scenarioMap)

exprFunc, err := replaceAllPatterns[pcommon.Map](tt.target, tt.mode, tt.pattern, tt.replacement, tt.function)
exprFunc, err := replaceAllPatterns[pcommon.Map](tt.target, tt.mode, tt.pattern, tt.replacement, tt.replacementFormat, tt.function)
assert.NoError(t, err)

_, err = exprFunc(nil, scenarioMap)
Expand All @@ -284,9 +333,10 @@ func Test_replaceAllPatterns_bad_input(t *testing.T) {
return "{replacement}", nil
},
}
replacementFormat := ottl.Optional[string]{}
function := ottl.Optional[ottl.FunctionGetter[interface{}]]{}

exprFunc, err := replaceAllPatterns[interface{}](target, modeValue, "regexpattern", replacement, function)
exprFunc, err := replaceAllPatterns[interface{}](target, modeValue, "regexpattern", replacement, replacementFormat, function)
assert.Nil(t, err)

_, err = exprFunc(nil, input)
Expand All @@ -305,9 +355,10 @@ func Test_replaceAllPatterns_bad_function_input(t *testing.T) {
return nil, nil
},
}
replacementFormat := ottl.Optional[string]{}
function := ottl.Optional[ottl.FunctionGetter[interface{}]]{}

exprFunc, err := replaceAllPatterns[interface{}](target, modeValue, "regexp", replacement, function)
exprFunc, err := replaceAllPatterns[interface{}](target, modeValue, "regexp", replacement, replacementFormat, function)
assert.NoError(t, err)

result, err := exprFunc(nil, input)
Expand All @@ -334,9 +385,10 @@ func Test_replaceAllPatterns_bad_function_result(t *testing.T) {
},
Fact: StandardConverters[interface{}]()["IsString"],
}
replacementFormat := ottl.Optional[string]{}
function := ottl.NewTestingOptional[ottl.FunctionGetter[interface{}]](ottlValue)

exprFunc, err := replaceAllPatterns[interface{}](target, modeValue, "regexp", replacement, function)
exprFunc, err := replaceAllPatterns[interface{}](target, modeValue, "regexp", replacement, replacementFormat, function)
assert.NoError(t, err)

result, err := exprFunc(nil, input)
Expand All @@ -355,9 +407,10 @@ func Test_replaceAllPatterns_get_nil(t *testing.T) {
return "{anything}", nil
},
}
replacementFormat := ottl.Optional[string]{}
function := ottl.Optional[ottl.FunctionGetter[interface{}]]{}

exprFunc, err := replaceAllPatterns[interface{}](target, modeValue, "regexp", replacement, function)
exprFunc, err := replaceAllPatterns[interface{}](target, modeValue, "regexp", replacement, replacementFormat, function)
assert.NoError(t, err)

_, err = exprFunc(nil, nil)
Expand All @@ -376,10 +429,11 @@ func Test_replaceAllPatterns_invalid_pattern(t *testing.T) {
return "{anything}", nil
},
}
replacementFormat := ottl.Optional[string]{}
function := ottl.Optional[ottl.FunctionGetter[interface{}]]{}

invalidRegexPattern := "*"
exprFunc, err := replaceAllPatterns[interface{}](target, modeValue, invalidRegexPattern, replacement, function)
exprFunc, err := replaceAllPatterns[interface{}](target, modeValue, invalidRegexPattern, replacement, replacementFormat, function)
require.Error(t, err)
assert.ErrorContains(t, err, "error parsing regexp:")
assert.Nil(t, exprFunc)
Expand All @@ -397,10 +451,11 @@ func Test_replaceAllPatterns_invalid_model(t *testing.T) {
return "{anything}", nil
},
}
replacementFormat := ottl.Optional[string]{}
function := ottl.Optional[ottl.FunctionGetter[interface{}]]{}

invalidMode := "invalid"
exprFunc, err := replaceAllPatterns[interface{}](target, invalidMode, "regex", replacement, function)
exprFunc, err := replaceAllPatterns[interface{}](target, invalidMode, "regex", replacement, replacementFormat, function)
assert.Nil(t, exprFunc)
assert.Contains(t, err.Error(), "invalid mode")
}
Loading
Loading