[exporter/loki] Automatically maps severity value to "level" attribute #14560

jtama · 2022-09-28T08:17:20Z

Description: Loki follows severity using the "level" attribute value. This allow automatic mapping from the log record "severity" value to the "attribute.level" value

Link to tracking Issue: Closes #14313

Testing: Test added in the pkg/translator/loki/convert_test.go#TestConvertAttributesAndMerge
Documentation: Not done for the moment

pkg/translator/loki/convert.go

kovrus

Can you maybe add one sentence in the readme of the Loki exporter mentioning this automatic mapping?

pkg/translator/loki/logs_to_loki.go

kovrus · 2022-10-06T10:29:14Z

pkg/translator/loki/convert.go

@@ -59,7 +58,15 @@ func convertAttributesToLabels(attributes pcommon.Map, attrsToSelect pcommon.Val
 		attr = strings.TrimSpace(attr)
 		av, ok := attributes.Get(attr) // do we need to trim this?
 		if ok {
-			out[model.LabelName(attr)] = model.LabelValue(av.AsString())
+			if attr == levelAttributeName {


lets revert this changes and do the conversion in logs_to_loki#handleSeverity method, smth like

level, _ := severityNumberToLevel[log.SeverityNumber().String()] ... log.Attributes().PutStr(levelAttributeName, level)

linting error: coz of _

kovrus

lgtm

kovrus · 2022-10-06T19:41:31Z

@jpkrohling do you want to take another look at the PR?

jpkrohling · 2022-10-10T20:35:21Z

I'll take another look.

jtama · 2022-10-18T07:18:06Z

I'll take another look.

@jpkrohling Does it looks good to you ?

exporter/lokiexporter/README.md

pkg/translator/loki/logs_to_loki.go

jpkrohling · 2022-10-20T11:33:51Z

Could you please fix the conflicts? This PR will then be ready to be merged. Thank you for your patience during this PR!

jpkrohling · 2022-10-20T11:41:59Z

If you prefer to have the simple version of the code (the original one you wrote), that's fine as well. We can optimize it later if we find it to be a problem.

jtama · 2022-10-20T13:08:00Z

If you prefer to have the simple version of the code (the original one you wrote), that's fine as well. We can optimize it later if we find it to be a problem.

Done. Also backported the code to LogsToLokiRequests

jpkrohling · 2022-10-20T15:44:17Z

The tests seem to be failing:

=== RUN   TestLogsToLokiRequestWithoutTenant/with_severity_to_label
    logs_to_loki_test.go:329: 
        	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/pkg/translator/loki/logs_to_loki_test.go:329
        	Error:      	Not equal: 
        	            	expected: "{exporter=\"OTLP\", level=\"DEBUG4\"}"
        	            	actual  : "{exporter=\"OTLP\"}"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-{exporter="OTLP", level="DEBUG4"}
        	            	+{exporter="OTLP"}
        	Test:       	TestLogsToLokiRequestWithoutTenant/with_severity_to_label
    logs_to_loki_test.go:333: 
        	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/pkg/translator/loki/logs_to_loki_test.go:333
        	Error:      	Not equal: 
        	            	expected: "{\"traceid\":\"01020304000000000000000000000000\"}"
        	            	actual  : "{\"traceid\":\"01000000000000000000000000000000\"}"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-{"traceid":"01020304000000000000000000000000"}
        	            	+{"traceid":"01000000000000000000000000000000"}
        	Test:       	TestLogsToLokiRequestWithoutTenant/with_severity_to_label
    logs_to_loki_test.go:333: 
        	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/pkg/translator/loki/logs_to_loki_test.go:333
        	Error:      	Not equal: 
        	            	expected: "{\"traceid\":\"01020304050000000000000000000000\"}"
        	            	actual  : "{\"traceid\":\"02000000000000000000000000000000\"}"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-{"traceid":"01020304050000000000000000000000"}
        	            	+{"traceid":"02000000000000000000000000000000"}
        	Test:       	TestLogsToLokiRequestWithoutTenant/with_severity_to_label
    logs_to_loki_test.go:333: 
        	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/pkg/translator/loki/logs_to_loki_test.go:333
        	Error:      	Not equal: 
        	            	expected: "{\"traceid\":\"01020304050600000000000000000000\"}"
        	            	actual  : "{\"traceid\":\"03000000000000000000000000000000\"}"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-{"traceid":"01020304050600000000000000000000"}
        	            	+{"traceid":"03000000000000000000000000000000"}
        	Test:       	TestLogsToLokiRequestWithoutTenant/with_severity_to_label
=== RUN   TestLogsToLokiRequestWithoutTenant/with_severity,_already_existing_level
--- FAIL: TestLogsToLokiRequestWithoutTenant (0.00s)
    --- PASS: TestLogsToLokiRequestWithoutTenant/with_attribute_to_label_and_regular_attribute (0.00s)
    --- PASS: TestLogsToLokiRequestWithoutTenant/with_resource_to_label_and_regular_resource (0.00s)
    --- FAIL: TestLogsToLokiRequestWithoutTenant/with_severity_to_label (0.00s)
    --- FAIL: TestLogsToLokiRequestWithoutTenant/with_severity,_already_existing_level (0.00s)
panic: runtime error: index out of range [1] with length 1 [recovered]
	panic: runtime error: index out of range [1] with length 1

goroutine 51 [running]:
testing.tRunner.func1.2({0x12af040, 0xc0000e8dc8})
	/opt/hostedtoolcache/go/1.18.7/x64/src/testing/testing.go:1389 +0x366
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.18.7/x64/src/testing/testing.go:1392 +0x5d2
panic({0x12af040, 0xc0000e8dc8})
	/opt/hostedtoolcache/go/1.18.7/x64/src/runtime/panic.go:844 +0x258
go.opentelemetry.io/collector/pdata/plog.LogRecordSlice.At(...)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/pdata@v0.62.2-0.20221017171445-6313054b642c/plog/generated_logs.go:469
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki.TestLogsToLokiRequestWithoutTenant.func1(0x0?)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/pkg/translator/loki/logs_to_loki_test.go:298 +0x14a9
testing.tRunner(0xc0002cd040, 0xc0002da8e0)
	/opt/hostedtoolcache/go/1.18.7/x64/src/testing/testing.go:1439 +0x214
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.18.7/x64/src/testing/testing.go:1486 +0x725
FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki	0.062s
FAIL
make[2]: *** [../../../Makefile.Common:57: test] Error 1

jtama · 2022-11-04T08:31:06Z

@jpkrohling, @kovrus Is it good for you ?

jtama · 2022-11-09T14:38:23Z

Why is this blocked ? It would be much simpler for me to have this feature ;)

kovrus · 2022-11-09T14:56:09Z

@jtama I think it is good to be merged.

@jpkrohling can you please merge it?

open-telemetry#14560) Automatically maps severity value to "level" attribute

lieberlois · 2024-11-12T10:50:55Z

@jtama Is there any guidance on how to extract the severity field into a log attribute using the OTEL Collector?

jtama · 2024-11-12T12:43:09Z

Hi,
I just saw your comment on the grafana project. Do you want to achieve the same only with configuration ? I don't know if this is even possible.

jtama requested a review from a team September 28, 2022 08:17

jtama requested a review from jpkrohling as a code owner September 28, 2022 08:17

github-actions bot assigned djaglowski Sep 28, 2022

djaglowski changed the title ~~Automatically maps severity value to "level" attribute~~ [exporter/loki] Automatically maps severity value to "level" attribute Sep 28, 2022

djaglowski added the exporter/loki Loki Exporter label Sep 28, 2022

kovrus reviewed Oct 5, 2022

View reviewed changes

pkg/translator/loki/convert.go Outdated Show resolved Hide resolved

pkg/translator/loki/convert.go Outdated Show resolved Hide resolved

kovrus reviewed Oct 6, 2022

View reviewed changes

kovrus approved these changes Oct 6, 2022

View reviewed changes

jpkrohling reviewed Oct 19, 2022

View reviewed changes

exporter/lokiexporter/README.md Outdated Show resolved Hide resolved

pkg/translator/loki/logs_to_loki.go Show resolved Hide resolved

jpkrohling approved these changes Oct 20, 2022

View reviewed changes

Automatically maps severity value to "level" attribute

9546cdb

djaglowski merged commit e3fccb7 into open-telemetry:main Nov 9, 2022

shalper2 pushed a commit to shalper2/opentelemetry-collector-contrib that referenced this pull request Dec 6, 2022

[exporter/loki] Automatically maps severity value to "level" attribute (

6bdcfac

open-telemetry#14560) Automatically maps severity value to "level" attribute

plantfansam mentioned this pull request Jul 21, 2023

disregard #24427

Closed

crobert-1 mentioned this pull request Mar 1, 2024

[exporter/loki] Documentation incorrect on mapping severity to level #31333

Closed

svennergr mentioned this pull request Nov 12, 2024

Loki: Implement detected_level in logs volume grafana/grafana#87564

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporter/loki] Automatically maps severity value to "level" attribute #14560

[exporter/loki] Automatically maps severity value to "level" attribute #14560

jtama commented Sep 28, 2022 •

edited by jpkrohling

Loading

kovrus left a comment •

edited

Loading

kovrus Oct 6, 2022

kovrus Oct 6, 2022

kovrus left a comment

kovrus commented Oct 6, 2022

jpkrohling commented Oct 10, 2022

jtama commented Oct 18, 2022

jpkrohling commented Oct 20, 2022 •

edited

Loading

jpkrohling commented Oct 20, 2022

jtama commented Oct 20, 2022

jpkrohling commented Oct 20, 2022

jtama commented Nov 4, 2022 •

edited

Loading

jtama commented Nov 9, 2022 •

edited

Loading

kovrus commented Nov 9, 2022 •

edited

Loading

lieberlois commented Nov 12, 2024

jtama commented Nov 12, 2024

[exporter/loki] Automatically maps severity value to "level" attribute #14560

[exporter/loki] Automatically maps severity value to "level" attribute #14560

Conversation

jtama commented Sep 28, 2022 • edited by jpkrohling Loading

kovrus left a comment • edited Loading

Choose a reason for hiding this comment

kovrus Oct 6, 2022

Choose a reason for hiding this comment

kovrus Oct 6, 2022

Choose a reason for hiding this comment

kovrus left a comment

Choose a reason for hiding this comment

kovrus commented Oct 6, 2022

jpkrohling commented Oct 10, 2022

jtama commented Oct 18, 2022

jpkrohling commented Oct 20, 2022 • edited Loading

jpkrohling commented Oct 20, 2022

jtama commented Oct 20, 2022

jpkrohling commented Oct 20, 2022

jtama commented Nov 4, 2022 • edited Loading

jtama commented Nov 9, 2022 • edited Loading

kovrus commented Nov 9, 2022 • edited Loading

lieberlois commented Nov 12, 2024

jtama commented Nov 12, 2024

jtama commented Sep 28, 2022 •

edited by jpkrohling

Loading

kovrus left a comment •

edited

Loading

jpkrohling commented Oct 20, 2022 •

edited

Loading

jtama commented Nov 4, 2022 •

edited

Loading

jtama commented Nov 9, 2022 •

edited

Loading

kovrus commented Nov 9, 2022 •

edited

Loading