Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix service name source being added multiple times for multiple injections #254

Merged
merged 3 commits into from
Oct 23, 2024

Conversation

nathalapooja
Copy link
Contributor

@nathalapooja nathalapooja commented Oct 21, 2024

Issue #, if available:
With the current code, if the customer pod has multiple injections , the service name source is being added multiple times and with invalid value from the second injection.

OTEL_RESOURCE_ATTRIBUTES: com.amazonaws.cloudwatch.entity.internal.service.name.source=K8sWorkload,k8s.container.name=petclinic,k8s.deployment.name=petclinic-1,k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.replicaset.name=petclinic-1-6bb849b4d5,service.version=latest,**com.amazonaws.cloudwatch.entity.internal.service.name.source=Instrumentation,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME)**

Description of changes:
added code to check the existing OTEL resource attributes and add the attribute only when the existing value is NULL

Tested using Unit Tests

Manual Testing on petclinic pod without custom service name

Before

OTEL_RESOURCE_ATTRIBUTES:   k8s.container.name=petclinic-container,k8s.daemonset.name=petclinic-pod,k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),service.vers │
│ ion=latest,com.amazonaws.cloudwatch.entity.internal.service.name.source=K8sWorkload,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),com.amazonaws.cloudwatch.entity.internal.service.name.source=Instrumentation

After

OTEL_RESOURCE_ATTRIBUTES:                        com.amazonaws.cloudwatch.entity.internal.service.name.source=K8sWorkload,k8s.container.name=petclinic-container,k8s.daemonset.name=petclinic-pod,k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_ │
│ NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),service.version=latest   

With Custom service name:

After

OTEL_RESOURCE_ATTRIBUTES:                        deployment.environment=compass-env,com.amazonaws.cloudwatch.entity.internal.service.name.source=Instrumentation,k8s.container.name=petclinic-container,k8s.daemonset.name=petclinic-pod,k8s.namespace.name=default,k8 │
│ s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),service.version=latest  

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@@ -263,8 +263,22 @@ func (i *sdkInjector) injectCommonSDKConfig(ctx context.Context, otelinst v1alph
}
}

// get existing resources env var and parse it into a map
existingRes := map[string]string{}
existingResourceEnvIdx := getIndexOfEnv(pod.Spec.Containers[appIndex].Env, constants.EnvOTELResourceAttrs)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like this code borrows the first few lines of createResourceMap...can we prevent duplicate code by creating a helper function for the following:

// get existing resources env var and parse it into a map
	existingRes := map[string]bool{}
	existingResourceEnvIdx := getIndexOfEnv(pod.Spec.Containers[index].Env, constants.EnvOTELResourceAttrs)
	if existingResourceEnvIdx > -1 {
		existingResArr := strings.Split(pod.Spec.Containers[index].Env[existingResourceEnvIdx].Value, ",")
		for _, kv := range existingResArr {
			keyValueArr := strings.Split(strings.TrimSpace(kv), "=")
			if len(keyValueArr) != 2 {
				continue
			}
			existingRes[keyValueArr[0]] = true
		}
	}

	res := map[string]string{}
	for k, v := range otelinst.Spec.Resource.Attributes {
		if !existingRes[k] {
			res[k] = v
		}
	}

and then using this in createResourceMap?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using map[string]bool in createResourceMap and in the new code we use map[string]string{}.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't need to be a map[string]string{} right? Because we are just checking if it exists.

I don't see us using the value of existingRes at all in the code

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could actually just modify createResourceMap to return the existing map. Seems wasteful to execute the same code twice.

// Some attributes might be empty, we should get them via k8s downward API
if resourceMap[string(semconv.K8SPodNameKey)] == "" {
if existingRes[string(semconv.K8SPodNameKey)] == "" && resourceMap[string(semconv.K8SPodNameKey)] == "" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand...why isn't resourceMap sufficient to determine if the k8s pod name key is missing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for k, v := range otelinst.Spec.Resource.Attributes {
if !existingRes[k] {
res[k] = v
}
}
If the attribute doesn't exist in existingRes, then the res will be updated. So when the second injection happens, res doesn't have the semconv.K8SPodNameKey key in it as it was existing value. So just checking the resourceMap[string(semconv.K8SPodNameKey)] == "" will be true and the key is added again

@nathalapooja nathalapooja merged commit 79a21d9 into main-feature Oct 23, 2024
1 check passed
@nathalapooja nathalapooja deleted the fix-multiple-inject branch October 23, 2024 16:08
lisguo added a commit that referenced this pull request Oct 28, 2024
…er (#262)

* added agent server port as default port on CW agent (#234)

* Attach service name source as resource attribute to operator (#235)

* Add service name source for instrumentation and workload

* Change service name source attribute naming

* Fix unit tests for pod mutators

* Simplify service name source determination

* Fix incorrect refactor of source constants

* Use replicaset name as part of service name fallback (#249)

* Use replicaset name as part of service name fallback

* Add chooseServiceName unit test

* Change import statement ordering

* Fix service name source being added multiple times for multiple injections (#254)

* added fix to not override the service name source for multiple auto-inject

---------

Co-authored-by: POOJA REDDY NATHALA <poojardy@amazon.com>
Co-authored-by: zhihonl <61301537+zhihonl@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants