Send fluentd logs through otel #109
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This sends logs collected by fluentd to otelcol via fluentforward. There are some limitations noted by TODOs that I will file issues to track but they should not affect the common cases. It's mostly around configuring of hec_exporter TLS settings. It still attaches k8s metadata on the fluentd side as it uses various annotations to construct source/sourcetype in some cases. May not be worth trying to fix with move to filelog receiver.
dmitryax
reviewed
Apr 22, 2021
| # set the index field to the value found in the pod splunk.com/index annotations. if not set, use namespace annotation, or default to the default_index | ||
| index ${record.dig("kubernetes", "annotations", "splunk.com/index") ? record.dig("kubernetes", "annotations", "splunk.com/index") : record.dig("kubernetes", "namespace_annotations", "splunk.com/index") ? (record["kubernetes"]["namespace_annotations"]["splunk.com/index"]) : ("{{ .Values.logsBackend.hec.indexName | default "main"}}")} | ||
| index ${record.dig("kubernetes", "annotations", "splunk.com/index") ? record.dig("kubernetes", "annotations", "splunk.com/index") : record.dig("kubernetes", "namespace_annotations", "splunk.com/index")} |
There was a problem hiding this comment.
Look like main default value is missed?
There was a problem hiding this comment.
I tried to consolidate setting of index default in the filter ** section:
com.splunk.index ${record.dig("index") ? record.dig("index") : "{{ .Values.logsBackend.hec.indexName | default "main"}}"}
I'll have to test more to be completely confident it's working.
There was a problem hiding this comment.
So this will be updated, right?
There was a problem hiding this comment.
If index isn't set or is null then it falls back to setting default value here. No need to set default everywhere.
There was a problem hiding this comment.
Ok found it in com.splunk.index
dmitryax
approved these changes
Apr 23, 2021
Comment on lines
+292
to
+307
| <filter **> | ||
| @type record_transformer | ||
| enable_ruby | ||
| <record> | ||
| com.splunk.sourcetype ${record.dig("sourcetype") ? record.dig("sourcetype") : ""} | ||
| com.splunk.source ${record.dig("source") ? record.dig("source") : ""} | ||
| com.splunk.index ${record.dig("index") ? record.dig("index") : "{{ .Values.logsBackend.hec.indexName | default "main"}}"} | ||
| </record> | ||
| remove_keys denylist,docker,kubernetes,source,sourcetype,index | ||
| </filter> |
There was a problem hiding this comment.
Can we remove this extra filter and use com.splunk.index instead of index from the beginning? a follow up PR is fine
There was a problem hiding this comment.
Yep index can, source and sourcetype can't because of the jq .record.source = ... craziness.
|
Merging to otel-logs for followup PRs before merging to main. |
Merged
jrcamp
added a commit
that referenced
this pull request
May 21, 2021
Send fluentd logs through otel (#109) This sends logs collected by fluentd to otelcol via fluentforward. There are some limitations noted by TODOs that I will file issues to track but they should not affect the common cases. It's mostly around configuring of hec_exporter TLS settings. It still attaches k8s metadata on the fluentd side as it uses various annotations to construct source/sourcetype in some cases. May not be worth trying to fix with move to filelog receiver. * Fluentd cleanup (#123) * Remove hec token * remove ingestHost, ingestPort, ingestProtocol * Disable receivers and pipelines if telemetry type disabled (#127) - Fix changelog ordering - Don't include fluentd configmap when agent not enabled - Enable http-forwarder for all telemetry types since signalfx exporter sends metadata updates * remove fluentforward from service (no longer configured in gateway) * use <ingest>/v1/log for hec endpoint
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This sends logs collected by fluentd to otelcol via fluentforward. There are
some limitations noted by TODOs that I will file issues to track but they
should not affect the common cases. It's mostly around configuring of
hec_exporter TLS settings.
It still attaches k8s metadata on the fluentd side as it uses various
annotations to construct source/sourcetype in some cases. May not be worth
trying to fix with move to filelog receiver.
Tested with agent sending directly and through gateway.