Distributed Tracing CPU Load Optimization

[iamemilio]

Customers have been reporting 30% higher CPU when using distributed tracing. GTSE has pinpointed this issue to the when the GetMetadataFromContext() function calls CreateDistributedTracePayload(). Their profiling suggests that the problematic line is:

hdrs.Set(DistributedTraceNewRelicHeader, p.NRHTTPSafe())

https://github.com/newrelic/go-agent/blob/master/v3/newrelic/distributed_tracing.go#L139-L153

Is there a way that we can reduce the performance impact of this code? Is this impacted by the memory leak?

[nr-swilloughby]

I pushed a fix for this to the unreleased branch json_speed_issue_505. In my preliminary testing this code runs 2x the speed of the previous code. If you want, you can try running some testcases using the agent code in that branch and see how the performance compares to what you were seeing before.