Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions enrichments/trace/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,10 @@ type ElasticTransactionConfig struct {
// ElasticSpanConfig configures the enrichment attributes for the spans
// which are NOT identified as elastic transaction.
type ElasticSpanConfig struct {
Name AttributeConfig `mapstructure:"name"`
EventOutcome AttributeConfig `mapstructure:"event_outcome"`
ServiceTarget AttributeConfig `mapstructure:"service_target"`
Name AttributeConfig `mapstructure:"name"`
EventOutcome AttributeConfig `mapstructure:"event_outcome"`
ServiceTarget AttributeConfig `mapstructure:"service_target"`
DestinationService AttributeConfig `mapstructure:"destination_service"`
}

// AttributeConfig is the configuration options for each attribute.
Expand All @@ -79,9 +80,10 @@ func Enabled() Config {
EventOutcome: AttributeConfig{Enabled: true},
},
Span: ElasticSpanConfig{
Name: AttributeConfig{Enabled: true},
EventOutcome: AttributeConfig{Enabled: true},
ServiceTarget: AttributeConfig{Enabled: true},
Name: AttributeConfig{Enabled: true},
EventOutcome: AttributeConfig{Enabled: true},
ServiceTarget: AttributeConfig{Enabled: true},
DestinationService: AttributeConfig{Enabled: true},
},
}
}
17 changes: 9 additions & 8 deletions enrichments/trace/internal/elastic/attributes.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,13 @@ const (
AttributeServiceFrameworkVersion = "service.framework.version"

// span attributes
AttributeTransactionRoot = "transaction.root"
AttributeTransactionName = "transaction.name"
AttributeTransactionType = "transaction.type"
AttributeTransactionResult = "transaction.result"
AttributeSpanName = "span.name"
AttributeEventOutcome = "event.outcome"
AttributeServiceTargetType = "service.target.type"
AttributeServiceTargetName = "service.target.name"
AttributeTransactionRoot = "transaction.root"
AttributeTransactionName = "transaction.name"
AttributeTransactionType = "transaction.type"
AttributeTransactionResult = "transaction.result"
AttributeSpanName = "span.name"
AttributeEventOutcome = "event.outcome"
AttributeServiceTargetType = "service.target.type"
AttributeServiceTargetName = "service.target.name"
AttributeSpanDestinationServiceResource = "span.destination.service.resource"
)
40 changes: 40 additions & 0 deletions enrichments/trace/internal/elastic/span.go
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,9 @@ func (s *spanEnrichmentContext) enrichSpan(
if cfg.ServiceTarget.Enabled {
s.setServiceTarget(span)
}
if cfg.DestinationService.Enabled {
s.setDestinationService(span)
}
}

// normalizeAttributes sets any dependent attributes that
Expand Down Expand Up @@ -300,6 +303,40 @@ func (s *spanEnrichmentContext) setServiceTarget(span ptrace.Span) {
}
}

func (s *spanEnrichmentContext) setDestinationService(span ptrace.Span) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[For reviewers] There is quite a bit of discrepancy between the destination resource and the service target fields (service.target.{name, type}) - you can check this by the test cases in the PR. The current code keeps the logic similar to what is present in apm-data for the most part -- any points of differences should be highlighted in the other comments.

var destnResource string
if s.peerService != "" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, peer.service is a very interesting attribute.

SHOULD be equal to the actual service.name resource attribute of the remote service if any.

This is THE ultimate attribute describing a dependency 🎉 - if I understand the spec correctly, this literally gives us the service name of the callee.

So the difference between taking this as the higher prio attribute for span.destination.service.resource vs. not taking it as the higher one for service.target.name comes from apm-data, right?

I suspect the thing here is the following: while this attribute in theory is very useful, in reality, that is probably never used. How could one populate this attribute? I can't imagine an agent is able to figure out the service.name of the service that it's calling. That's something that the user sets on the other side (or is auto populated based on the executable name). Looking at the history it seems that code was there since the very first commit (haven't checked the pre-history in apm-server).

So I think the code path that uses peer.service is basically never triggered in real life.

If this would be a new mapping, I'd say that there should not be a difference here - if peer.service is set, then service.target.name should take it as well, because it's more useful to see the service name as e.g. www.foo.bar:443. But given we have so many moving parts, I agree, let's not go into this right now - just keep as is.

At the same time, I see there are lots of tests focusing on this - it's ok to have those, just keep in mind, if my theory about basically never having peer.service in real life is correct, then those tests are not very relevant for production. It's more likely that part is always skipped and these fields are populated based on other attributes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the difference between taking this as the higher prio attribute for span.destination.service.resource vs. not taking it as the higher one for service.target.name comes from apm-data, right?

Yes, you are right.

Thanks for the insight on the peer.service, I was wondering about the peer.service before and your point does make sense. However, I wonder if we should give higher priority to peer.service for service.target.name deduction in an event it is actually set... Maybe something to discuss outside this PR and long term.

But given we have so many moving parts, I agree, let's not go into this right now - just keep as is.

Agreed 👍

At the same time, I see there are lots of tests focusing on this - it's ok to have those, just keep in mind, if my theory about basically never having peer.service in real life is correct, then those tests are not very relevant for production. It's more likely that part is always skipped and these fields are populated based on other attributes.

Hmm, good point. I think I added peer.service for a lot of test cases. I will revisit them separately and see if I need to edit something, will create a followup if required.

destnResource = s.peerService
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't this be just an if-else and then dropping the lots of destnResource == "" checks for better readability?

Something like:

	if s.peerService != "" {
		destnResource = s.peerService
	} else {
		switch {
		case s.isDB:
			if s.dbType != "" {
				destnResource = s.dbType
			}
		case s.isMessaging:
			if s.messagingSystem != "" {
				destnResource = s.messagingSystem
			}
			// For parity with apm-data, destn resource does not handle
			// temporary destination flag. However, it is handled by
			// service.target fields and we might want to do the same here.
			if s.messagingDestinationName != "" {
				destnResource += "/" + s.messagingDestinationName
			}
		case s.isRPC, s.isHTTP:
			if res := getHostPort(s.urlFull, s.urlDomain, s.urlPort); res != "" {
				destnResource = res
			}
		}
	}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or separate the derivation from setting the attribute, so we can return early.

Copy link
Contributor Author

@lahsivjar lahsivjar Aug 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

		// For parity with apm-data, destn resource does not handle
		// temporary destination flag. However, it is handled by
		// service.target fields and we might want to do the same here.
		if destnResource != "" && s.messagingDestinationName != "" {
			destnResource += "/" + s.messagingDestinationName
		}

This is the problematic bit, the destination resource is updated based on the messagingDestinationName. I could make it return early with some conditions but, I think, that would make the code a bit harder to follow through and maintain. No strong opinions though but a simple if/else or a case in the switch statement itself for s.peerService != "" will not work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case it was not clear in the above statement, the messaging destination is appended to the destination resource if it is not empty. Atleast, this is what parity with apm-data would be. This would mean that if peer.service is testsvc and messagingDestinationName is topic1 then the destination resource would be testsvc/topic1 -- this is also covered in the test case TestElasticSpanEnrich/messaging_destination

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I was thinking we could do something like this:

if s.messagingSystem != "" {
    destnResource := s.messagingSystem
    if s.messagingDestinationName != "" {
        destnResource += "/" + s.messagingDestinationName
    }
    return destnResource
}

I realise now that's not quite equivalent to what you've implemented though, because as it's written we might append the messaging.destination.name to peer.service. Is that a valid thing to do though?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that a valid thing to do though?

I spent a bit of time reasoning about this when I was reading the apm-data code but I couldn't think of this NOT being valid so I carried over the logic. Technically, if we know the destination name of a topic and we have peer.service then appending the topic would result in an useful valid value. However, I am not sure how this would look practically for ex: will the peer.service and messaging.destination.name be populated together?

I am not too sure overall what to do here. I would suggest we do this separate to this PR, I have created a GH issue to discuss this and make any changes to this. Let me know if it sounds good.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems fine to me, thanks.


switch {
case s.isDB:
if destnResource == "" && s.dbType != "" {
destnResource = s.dbType
}
case s.isMessaging:
if destnResource == "" && s.messagingSystem != "" {
destnResource = s.messagingSystem
}
// For parity with apm-data, destn resource does not handle
// temporary destination flag. However, it is handled by
// service.target fields and we might want to do the same here.
if destnResource != "" && s.messagingDestinationName != "" {
destnResource += "/" + s.messagingDestinationName
}
Comment on lines +321 to +326
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[For reviewers] Check the code comment.

case s.isRPC, s.isHTTP:
if destnResource == "" {
if res := getHostPort(s.urlFull, s.urlDomain, s.urlPort); res != "" {
destnResource = res
}
}
}

if destnResource != "" {
span.Attributes().PutStr(AttributeSpanDestinationServiceResource, destnResource)
}
}

func isTraceRoot(span ptrace.Span) bool {
return span.ParentSpanID().IsEmpty()
}
Expand All @@ -319,6 +356,9 @@ func isElasticTransaction(span ptrace.Span) bool {
return false
}

// getHostPort derives the host:port value from url.* attributes. Unlike
// apm-data, the current code does NOT fallback to net.* or http.*
// attributes as most of these are now deprecated.
func getHostPort(urlFull *url.URL, urlDomain string, urlPort int64) string {
if urlFull != nil {
return urlFull.Host
Expand Down
108 changes: 60 additions & 48 deletions enrichments/trace/internal/elastic/span_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -265,10 +265,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetName: "testsvc",
AttributeServiceTargetType: "",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetName: "testsvc",
AttributeServiceTargetType: "",
AttributeSpanDestinationServiceResource: "testsvc",
},
},
{
Expand All @@ -285,10 +286,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "http",
AttributeServiceTargetName: "testsvc",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "http",
AttributeServiceTargetName: "testsvc",
AttributeSpanDestinationServiceResource: "testsvc",
},
},
{
Expand All @@ -311,10 +313,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "http",
AttributeServiceTargetName: "www.foo.bar:443",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "http",
AttributeServiceTargetName: "www.foo.bar:443",
AttributeSpanDestinationServiceResource: "testsvc",
},
},
{
Expand All @@ -335,10 +338,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "http",
AttributeServiceTargetName: "www.foo.bar:443",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "http",
AttributeServiceTargetName: "www.foo.bar:443",
AttributeSpanDestinationServiceResource: "testsvc",
},
},
{
Expand All @@ -355,10 +359,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "grpc",
AttributeServiceTargetName: "testsvc",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "grpc",
AttributeServiceTargetName: "testsvc",
AttributeSpanDestinationServiceResource: "testsvc",
},
},
{
Expand All @@ -372,10 +377,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "xmlrpc",
AttributeServiceTargetName: "testsvc",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "xmlrpc",
AttributeServiceTargetName: "testsvc",
AttributeSpanDestinationServiceResource: "testsvc",
},
},
{
Expand All @@ -391,10 +397,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "external",
AttributeServiceTargetName: "service.Test",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "external",
AttributeServiceTargetName: "service.Test",
AttributeSpanDestinationServiceResource: "testsvc",
},
},
{
Expand All @@ -408,10 +415,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "kafka",
AttributeServiceTargetName: "testsvc",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "kafka",
AttributeServiceTargetName: "testsvc",
AttributeSpanDestinationServiceResource: "testsvc",
},
},
{
Expand All @@ -425,10 +433,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "messaging",
AttributeServiceTargetName: "t1",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "messaging",
AttributeServiceTargetName: "t1",
AttributeSpanDestinationServiceResource: "testsvc/t1",
},
},
{
Expand All @@ -443,10 +452,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "messaging",
AttributeServiceTargetName: "testsvc",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "messaging",
AttributeServiceTargetName: "testsvc",
AttributeSpanDestinationServiceResource: "testsvc/t1",
},
},
{
Expand All @@ -467,10 +477,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "elasticsearch",
AttributeServiceTargetName: "testsvc",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "elasticsearch",
AttributeServiceTargetName: "testsvc",
AttributeSpanDestinationServiceResource: "testsvc",
},
},
{
Expand All @@ -496,10 +507,11 @@ func TestElasticSpanEnrich(t *testing.T) {
}(),
config: config.Enabled().Span,
enrichedAttrs: map[string]any{
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "cassandra",
AttributeServiceTargetName: "testsvc",
AttributeSpanName: "testspan",
AttributeEventOutcome: "success",
AttributeServiceTargetType: "cassandra",
AttributeServiceTargetName: "testsvc",
AttributeSpanDestinationServiceResource: "testsvc",
},
},
} {
Expand Down