[pkg/stanza] Add mute option for operators' error Handler

[ChrsMark]

Component(s)

pkg/stanza

Describe the issue you're reporting

At the moment if an operator fails to parse an entry an error is logged by

opentelemetry-collector-contrib/pkg/stanza/operator/helper/transformer.go

Line 97 in 2170757

func (t *TransformerOperator) HandleEntryError(ctx context.Context, entry *entry.Entry, err error) error {

.

For example given the following configuration:

operators:
      ...
      - type: move
        id: best_effort_move
        from: attributes.if_key
        to: attributes.val

where we want to apply the move on a best effort approach only if the if_key exists in the attributes, we will get the following error in the collector's logs:

2024-04-03T17:26:41.565+0300	error	helper/transformer.go:98	Failed to process entry	{"kind": "receiver", "name": "filelog", "data_type": "logs", "operator_id": "edo", "operator_type": "move", "error": "move: field does not exist: attributes.if_key", "action": "send"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/helper/transformer.go:98
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/helper/transformer.go:90
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/move.(*Transformer).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/transformer/move/move.go:69
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/helper/transformer.go:92
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/move.(*Transformer).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/transformer/move/move.go:69
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine.(*Transformer).flushSource
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/transformer/recombine/recombine.go:331
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/recombine.(*Transformer).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/transformer/recombine/recombine.go:253
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/helper/parser.go:122
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/container.(*Parser).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/parser/container/container.go:166
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/file.(*Input).emit
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/operator/input/file/file.go:52
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/fileconsumer/internal/reader/reader.go:89
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.97.0/fileconsumer/file.go:181

However, there might be cases where users want to use an operator in a best effort way, and hence having an actual error log for each failure is sth that can bug the users and flood their logs as well.

There are 2 options that I can think here:

1. log on debug level if OnError setting is SendOnError, otherwise if we DropOnError we can log on error level.
2. add a new muteErrorLogs setting which will be decoupled by the OnError setting in order to have a specific way how to handle the verbosity of the operator's errors.

[github-actions]

Pinging code owners:

• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

There are 2 options that I can think here:

1. log on debug level if OnError setting is SendOnError, otherwise if we DropOnError we can log on error level.
2. add a new muteErrorLogs setting which will be decoupled by the OnError setting in order to have a specific way how to handle the verbosity of the operator's errors.

A muteErrorLogs setting seems to imply that it would affect log levels across the entire operator, which I don't think should be the intention here. Maybe with a different name this could work but even then it could be very confusing to have a second setting which trigger under exactly the same circumstances as on_error.

I typically don't like enumerating combinations but given the above, maybe it would be most intuitive to just add new options to on_error e.g. send_quiet and drop_quiet.

[ChrsMark]

I typically don't like enumerating combinations but given the above, maybe it would be most intuitive to just add new options to on_error e.g. send_quiet and drop_quiet.

I think that would serve the purpose without making the configuration confusing.

[crobert-1]

Removing needs triage based on code owner's response.

[djaglowski]

Closed by #32220