Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a config to decode json-encoded strings in attribute values #2827

Merged
merged 3 commits into from
Mar 25, 2021
Merged

Add a config to decode json-encoded strings in attribute values #2827

merged 3 commits into from
Mar 25, 2021

Conversation

pxaws
Copy link
Contributor

@pxaws pxaws commented Mar 23, 2021

Description:

This commit adds a config parse_json_encoded_attr_values which is the list of attributes whose values are json-encoded strings. The emf exporter will convert those json-encoded strings as json structures before putting them into the emf log events.

This is to support the aws container insight use case in which json objects are used as field values. For example, like the kubernetes and Sources field in the following emf log event:

{
    "AutoScalingGroupName": "eks-a6bb9db9-267c-401c-db55-df8ef645b06f",
    "ClusterName": "ci-demo",
    "InstanceId": "i-01bf9fb097cbf3205",
    "InstanceType": "t2.xlarge",
    "Namespace": "aws-otel-eks",
    "NodeName": "ip-192-168-12-170.ec2.internal",
    "PodName": "aws-otel-eks-ci",
    "Sources": [
        "cadvisor",
        "calculated"
    ],
    "Type": "PodNet",
    "Version": "0",
    "interface": "eth0",
    "kubernetes": {
        "host": "ip-192-168-12-170.ec2.internal",
        "labels": {
            "controller-revision-hash": "66bb8fc7f4",
            "name": "aws-otel-eks-ci",
            "pod-template-generation": "1"
        },
        "namespace_name": "aws-otel-eks",
        "pod_id": "1810eef4-9121-4e93-8b0a-020c54f7924b",
        "pod_name": "aws-otel-eks-ci-xp66r",
        "pod_owners": [
            {
                "owner_kind": "DaemonSet",
                "owner_name": "aws-otel-eks-ci"
            }
        ]
    },
    "pod_interface_network_rx_bytes": 3166.8867430387045,
    "pod_interface_network_rx_dropped": 0,
    "pod_interface_network_rx_errors": 0,
    "pod_interface_network_rx_packets": 3.7814973119633524,
    "pod_interface_network_total_bytes": 4809.434331265391,
    "pod_interface_network_tx_bytes": 1642.5475882266862,
    "pod_interface_network_tx_dropped": 0,
    "pod_interface_network_tx_errors": 0,
    "pod_interface_network_tx_packets": 2.990021130389628
}

Since the api metric.LabelsMap() in https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/awsemfexporter/datapoint.go#L106 returns a key value string map, a json object like this

{
        "host": "ip-192-168-12-170.ec2.internal",
        "labels": {
            "controller-revision-hash": "66bb8fc7f4",
            "name": "aws-otel-eks-ci",
            "pod-template-generation": "1"
        },
        "namespace_name": "aws-otel-eks",
        "pod_id": "1810eef4-9121-4e93-8b0a-020c54f7924b",
        "pod_name": "aws-otel-eks-ci-xp66r",
        "pod_owners": [
            {
                "owner_kind": "DaemonSet",
                "owner_name": "aws-otel-eks-ci"
            }
        ]
    }

can't be used directly. To work around the limitation, we stringify the json value and save it as string attribute value in receiver and then restore the original json in emf exporter.

Link to tracking Issue:
#2307

Testing:
Updated the unit tests.

Documentation:
Updated the readme to include the new config.

@pxaws pxaws requested a review from anuraaga as a code owner March 23, 2021 04:46
@pxaws pxaws requested a review from a team March 23, 2021 04:46
@codecov
Copy link

codecov bot commented Mar 23, 2021

Codecov Report

Merging #2827 (89c7fca) into main (4df9244) will increase coverage by 1.06%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2827      +/-   ##
==========================================
+ Coverage   90.51%   91.58%   +1.06%     
==========================================
  Files         449      450       +1     
  Lines       22116    22213      +97     
==========================================
+ Hits        20019    20344     +325     
+ Misses       1632     1397     -235     
- Partials      465      472       +7     
Flag Coverage Δ
integration 69.09% <ø> (?)
unit 90.52% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
exporter/awsemfexporter/config.go 100.00% <ø> (ø)
exporter/awsemfexporter/emf_exporter.go 100.00% <100.00%> (ø)
exporter/awsemfexporter/factory.go 100.00% <100.00%> (ø)
exporter/awsemfexporter/metric_translator.go 98.17% <100.00%> (+0.25%) ⬆️
internal/common/testing/container/container.go 73.68% <0.00%> (ø)
receiver/jmxreceiver/receiver.go 94.80% <0.00%> (+16.88%) ⬆️
receiver/dockerstatsreceiver/docker.go 92.30% <0.00%> (+39.05%) ⬆️
receiver/dockerstatsreceiver/receiver.go 96.72% <0.00%> (+47.54%) ⬆️
receiver/jmxreceiver/subprocess/subprocess.go 96.55% <0.00%> (+74.48%) ⬆️
receiver/redisreceiver/receiver.go 87.50% <0.00%> (+87.50%) ⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4df9244...89c7fca. Read the comment docs.

@bogdandrutu bogdandrutu merged commit 48c806c into open-telemetry:main Mar 25, 2021
@pxaws pxaws deleted the decode-json-data branch April 21, 2021 17:50
pmatyjasek-sumo referenced this pull request in pmatyjasek-sumo/opentelemetry-collector-contrib Apr 28, 2021
* decoding json-encoded attribute values

* update config unit tests

* cover exception cases
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants