Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Logs Filter by Attributes in Its Json Body #31375

Closed
DevProJSSV opened this issue Feb 21, 2024 · 13 comments
Closed

Logs Filter by Attributes in Its Json Body #31375

DevProJSSV opened this issue Feb 21, 2024 · 13 comments
Labels
enhancement New feature or request processor/filter Filter processor

Comments

@DevProJSSV
Copy link

DevProJSSV commented Feb 21, 2024 •
edited
Loading

Hello community,

I am trying to filter some logs based on the structure this log has. I am testing with a pod that outputs the following log every second :

{
          'timestamp': 'sometime',
          'indexType': 'exampleIndexType',
          'index.type': 'exampleIndexType',
          'message': 'This is a log message'
}

image

I have tried the following filters but none of them are giving us any results:

First attempt :

filter/platform_audit_logs:
      logs:	     
        log_record:	      
          - 'IsMatch (body["indexType"], "exampleIndexType")'

Second attempt :

filter/platform_audit_logs_2:
      logs:
        include:
          match_type: strict
          record_attributes:
            - key: index.type
              value: "exampleIndexType"

Third attempt :

 filter/platform_audit_logs_3:
      logs:
        include:
          match_type: strict
          bodies:
            - "This is a log message"

Fourth attempt :

filter/platform_audit_logs_4:
      logs:
        include:
          match_type: strict
          resource_attributes:
            - key: index.type
              value: "exampleIndexType"

The filter is only working if we pick one of the attributes that are added from opentelemetry collector configuration :

image

filter/platform_audit_logs_4:
      logs:
        include:
          match_type: strict
          resource_attributes:
            - key: k8s.container.name
              value: "log-generator"

As you can see in the image all the attributes or keys if we can refer to them as they come in the log json structure are mapped in the Splunk log entry, but when trying to use them in the filter we do not see any log error in the controller console. This seems to be a good syntax but I think maybe we are not pointing to the right way of getting the log content.
@TylerHelmuth TylerHelmuth transferred this issue from open-telemetry/opentelemetry-helm-charts Feb 21, 2024
@TylerHelmuth TylerHelmuth added enhancement New feature or request processor/filter Filter processor labels Feb 21, 2024
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners for processor/filter: @TylerHelmuth @boostchicken. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@TylerHelmuth
Copy link
Member

@DevProJSSV what does your log look like inside the collector? Is that message all in the log body as a map or string? You can use the debug exporter with verbosity: detailed to see how the collector represents the log internally, which is what OTTL depends on.

@DevProJSSV
Copy link
Author

DevProJSSV commented Feb 21, 2024 •
edited
Loading

Hello @TylerHelmuth I just updated my description with more details. I hope the screen captures can help. Thank you for the help.

@DevProJSSV
Copy link
Author

This is what I can see in body @TylerHelmuth
image

@TylerHelmuth
Copy link
Member

Your body is a string so you can't do IsMatch(body["indexType"], "exampleIndexType") because you can't index the body because it is a string. You can try something like IsMatch(body, "exampleIndexType") or something like ParseJSON(body)["indexType"] == "exampleIndexType"

@TylerHelmuth
Copy link
Member

List of OTTL functions: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs

@DevProJSSV
Copy link
Author

Hello @TylerHelmuth thank you so much for your suggestions.
Yes, based on the IsMatch log_records I sent above we can see an error about the formatting :

2024-02-22T16:00:34.074Z	error	filterprocessor@v0.91.0/logs.go:85	failed processing logs	{"kind": "processor", "name": "filter/platform_audit_logs", "pipeline": "logs/audit", "error": "failed to eval condition: IsMatch(body[\"indexType\"], \"exampleIndexType\"), error getting value in ottl.StandardStringLikeGetter[github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog.TransformContext]: log bodies of type Str cannot be indexed; failed to eval condition: IsMatch(body[\"indexType\"], \"exampleIndexType\"), error getting value in ottl.StandardStringLikeGetter[github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog.TransformContext]: log bodies of type Str cannot be indexed", "errorCauses": [{"error": "failed to eval condition: IsMatch(body[\"indexType\"], \"exampleIndexType\"), error getting value in ottl.StandardStringLikeGetter[github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog.TransformContext]: log bodies of type Str cannot be indexed"}, {"error": "failed to eval condition: IsMatch(body[\"indexType\"], \"exampleIndexType\"), error getting value in ottl.StandardStringLikeGetter[github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog.TransformContext]: log bodies of type Str cannot be indexed"}]}
github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor.(*filterLogProcessor).processLogs
	github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor@v0.91.0/logs.go:85
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:52
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:60
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:60
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:60
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/batchprocessor.(*batchLogs).export
	go.opentelemetry.io/collector/processor/batchprocessor@v0.91.0/batch_processor.go:489
go.opentelemetry.io/collector/processor/batchprocessor.(*shard).sendItems
	go.opentelemetry.io/collector/processor/batchprocessor@v0.91.0/batch_processor.go:256
go.opentelemetry.io/collector/processor/batchprocessor.(*shard).start
	go.opentelemetry.io/collector/processor/batchprocessor@v0.91.0/batch_processor.go:218

I tested all of the scenarios you suggested and some others based on your response but I am still not able to see the expected result. Here I summarize the tests :

Log sample from opentelemetry output:

LogRecord #1
ObservedTimestamp: 2024-02-22 16:15:53.402719124 +0000 UTC
Timestamp: 2024-02-22 16:15:53.368377965 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"timestamp": "sometime", "indexType": "exampleIndexType", "index.type": "exampleIndexType", "message": "This is a log message"})
Attributes:
     -> log.file.path: Str(/var/log/pods/default_log-generator_930c702a-b82f-4a27-95ef-4fe38798ae0c/log-generator/0.log)
     -> log.iostream: Str(stdout)
     -> logtag: Str(F)
     -> time: Str(2024-02-22T16:15:53.368377965Z)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #2
ObservedTimestamp: 2024-02-22 16:15:53.402760525 +0000 UTC
Timestamp: 2024-02-22 16:15:53.368392965 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"timestamp": "sometime", "indexType": "exampleIndexType", "index.type": "exampleIndexType", "message": "This is a log message"})
Attributes:
     -> log.file.path: Str(/var/log/pods/default_log-generator_930c702a-b82f-4a27-95ef-4fe38798ae0c/log-generator/0.log)
     -> logtag: Str(F)
     -> time: Str(2024-02-22T16:15:53.368392965Z)
     -> log.iostream: Str(stdout)
Trace ID: 
Span ID: 
Flags: 0

Test # 1 :

filter/platform_audit_logs:
      logs:	     
        log_record:	      
          - 'ParseJSON(body)["indexType"] == "exampleIndexType"'

Result: Error

2024-02-22T16:09:54.817Z	error	filterprocessor@v0.91.0/logs.go:85	failed processing logs	{"kind": "processor", "name": "filter/platform_audit_logs", "pipeline": "logs/audit", "error": "failed to eval condition: ParseJSON(body)[\"indexType\"] == \"exampleIndexType\", ReadMapCB: expect { or n, but found [, error found in #1 byte of ...|[7] 2024/02|..., bigger context ...|[7] 2024/02/22 16:09:54.526203 [INF] 10.19.148.6:62|...; failed to eval condition: ParseJSON(body)[\"indexType\"] == \"exampleIndexType\", ReadMapCB: expect { or n, but found [, error found in #1 byte of ...|[7] 2024/02|..., bigger context ...|[7] 2024/02/22 16:09:54.526280 [DBG] 10.19.148.6:62|...", "errorCauses": [{"error": "failed to eval condition: ParseJSON(body)[\"indexType\"] == \"exampleIndexType\", ReadMapCB: expect { or n, but found [, error found in #1 byte of ...|[7] 2024/02|..., bigger context ...|[7] 2024/02/22 16:09:54.526203 [INF] 10.19.148.6:62|..."}, {"error": "failed to eval condition: ParseJSON(body)[\"indexType\"] == \"exampleIndexType\", ReadMapCB: expect { or n, but found [, error found in #1 byte of ...|[7] 2024/02|..., bigger context ...|[7] 2024/02/22 16:09:54.526280 [DBG] 10.19.148.6:62|..."}]}
github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor.(*filterLogProcessor).processLogs
	github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor@v0.91.0/logs.go:85
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:52
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:60
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:60
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:60
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/batchprocessor.(*batchLogs).export
	go.opentelemetry.io/collector/processor/batchprocessor@v0.91.0/batch_processor.go:489
go.opentelemetry.io/collector/processor/batchprocessor.(*shard).sendItems
	go.opentelemetry.io/collector/processor/batchprocessor@v0.91.0/batch_processor.go:256
go.opentelemetry.io/collector/processor/batchprocessor.(*shard).start
	go.opentelemetry.io/collector/processor/batchprocessor@v0.91.0/batch_processor.go:218

Test # 2 :

filter/platform_audit_logs:
      logs:	     
        log_record:	      
          - 'IsMatch(ParseJSON(body)["indexType"], "exampleIndexType")'

Result:

2024-02-22T16:04:04.881Z	error	filterprocessor@v0.91.0/logs.go:85	failed processing logs	{"kind": "processor", "name": "filter/platform_audit_logs", "pipeline": "logs/audit", "error": "failed to eval condition: IsMatch(ParseJSON(body)[\"indexType\"], \"exampleIndexType\"), error getting value in ottl.StandardStringLikeGetter[github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog.TransformContext]: ReadMapCB: expect { or n, but found [, error found in #1 byte of ...|[7] 2024/02|..., bigger context ...|[7] 2024/02/22 16:04:04.678668 [INF] 10.19.148.5:21|...; failed to eval condition: IsMatch(ParseJSON(body)[\"indexType\"], \"exampleIndexType\"), error getting value in ottl.StandardStringLikeGetter[github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog.TransformContext]: ReadMapCB: expect { or n, but found [, error found in #1 byte of ...|[7] 2024/02|..., bigger context ...|[7] 2024/02/22 16:04:04.678728 [DBG] 10.19.148.5:21|...", "errorCauses": [{"error": "failed to eval condition: IsMatch(ParseJSON(body)[\"indexType\"], \"exampleIndexType\"), error getting value in ottl.StandardStringLikeGetter[github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog.TransformContext]: ReadMapCB: expect { or n, but found [, error found in #1 byte of ...|[7] 2024/02|..., bigger context ...|[7] 2024/02/22 16:04:04.678668 [INF] 10.19.148.5:21|..."}, {"error": "failed to eval condition: IsMatch(ParseJSON(body)[\"indexType\"], \"exampleIndexType\"), error getting value in ottl.StandardStringLikeGetter[github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog.TransformContext]: ReadMapCB: expect { or n, but found [, error found in #1 byte of ...|[7] 2024/02|..., bigger context ...|[7] 2024/02/22 16:04:04.678728 [DBG] 10.19.148.5:21|..."}]}
github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor.(*filterLogProcessor).processLogs
	github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor@v0.91.0/logs.go:85
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:52
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:60
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:60
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1
	go.opentelemetry.io/collector/processor@v0.91.0/processorhelper/logs.go:60
go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs
	go.opentelemetry.io/collector/consumer@v0.91.0/logs.go:25
go.opentelemetry.io/collector/processor/batchprocessor.(*batchLogs).export
	go.opentelemetry.io/collector/processor/batchprocessor@v0.91.0/batch_processor.go:489
go.opentelemetry.io/collector/processor/batchprocessor.(*shard).sendItems
	go.opentelemetry.io/collector/processor/batchprocessor@v0.91.0/batch_processor.go:256
go.opentelemetry.io/collector/processor/batchprocessor.(*shard).start
	go.opentelemetry.io/collector/processor/batchprocessor@v0.91.0/batch_processor.go:218

Test # 3 :

filter/platform_audit_logs:
      logs:	     
        log_record:	      
          - 'IsMatch(body, ".*exampleIndexType.*")'

Results: No errors but is not match anything as we do not get any log in the Splunk platform:

LogRecord #4
ObservedTimestamp: 2024-02-22 16:15:53.402877425 +0000 UTC
Timestamp: 2024-02-22 16:15:53.368404365 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"timestamp": "sometime", "indexType": "exampleIndexType", "index.type": "exampleIndexType", "message": "This is a log message"})
Attributes:
     -> log.iostream: Str(stdout)
     -> time: Str(2024-02-22T16:15:53.368404365Z)
     -> log.file.path: Str(/var/log/pods/default_log-generator_930c702a-b82f-4a27-95ef-4fe38798ae0c/log-generator/0.log)
     -> logtag: Str(F)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #5
ObservedTimestamp: 2024-02-22 16:15:53.402915625 +0000 UTC
Timestamp: 2024-02-22 16:15:53.368410065 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"timestamp": "sometime", "indexType": "exampleIndexType", "index.type": "exampleIndexType", "message": "This is a log message"})
Attributes:
     -> log.file.path: Str(/var/log/pods/default_log-generator_930c702a-b82f-4a27-95ef-4fe38798ae0c/log-generator/0.log)
     -> time: Str(2024-02-22T16:15:53.368410065Z)
     -> log.iostream: Str(stdout)
     -> logtag: Str(F)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #6
ObservedTimestamp: 2024-02-22 16:15:53.402935125 +0000 UTC
Timestamp: 2024-02-22 16:15:53.368415365 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str({"timestamp": "sometime", "indexType": "exampleIndexType", "index.type": "exampleIndexType", "message": "This is a log message"})
Attributes:
     -> logtag: Str(F)
     -> log.iostream: Str(stdout)
     -> log.file.path: Str(/var/log/pods/default_log-generator_930c702a-b82f-4a27-95ef-4fe38798ae0c/log-generator/0.log)
     -> time: Str(2024-02-22T16:15:53.368415365Z)
Trace ID: 
Span ID: 
Flags: 0

image

Thank you so much in advance for any suggestions based on the logs.

@TylerHelmuth
Copy link
Member

For test 3, if there is a match the data is dropped, so I wouldn't expect it to be exported.

@DevProJSSV
Copy link
Author

Why @TylerHelmuth ? I thought this way I would be able to only consider exporting the logs that match this condition. Is there an exclusion by default or then this not the right approach to filter the way I would like to?

filter/platform_audit_logs:
      logs:	     
        log_record:	      
          - 'IsMatch(body, ".*exampleIndexType.*")'

Should this filter include an "include" parameter or something ? I can not identify it here https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md

@TylerHelmuth
Copy link
Member

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor#configuration

@TylerHelmuth
Copy link
Member

If any condition is met, the telemetry is dropped (each condition is ORed together).

@TylerHelmuth
Copy link
Member

You can do - 'not IsMatch(body, ".*exampleIndexType.*")' if you want to drop everything where that condition is not true

@DevProJSSV
Copy link
Author

Thank you @TylerHelmuth this works for us !

This was referenced Feb 27, 2024
Closed
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request processor/filter Filter processor
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@TylerHelmuth @DevProJSSV