Details for peer service on spans of kind SERVER

[spencerwilson]

What are you trying to achieve?
We have two web applications which interact with a backend service over HTTP and WebSocket. On the backend's SERVER spans it would be valuable to have data that indicate "it was version 1.2.3 of Web App 1" that made this request. Pursuing this, two issues arise.

Problem 1: How to store this data on SERVER spans

@anuraaga added the span attribute peer.service in #652. Originally proposed as peer.service.name and subsequently simplified, this attribute is what its old name suggests: the service.name property on the peer Service.

If a SERVER span wanted to indicate other information about the client Resource (namespace, instance id, version), how should it do so? This is of use particularly in systems where any spans generated on the client would be in a separate trace (say, the request crosses a trust boundary and so for security reasons we design the trace context to be reset). Being in a separate trace,

• its sampling decision is independent from that of the backend trace, and so the client-side spans may not have been persisted
• the very useful queries "group this set of SERVER spans by their parent span's peer.service", or "get me the SERVER spans whose parent span's peer.service is X", are difficult or impossible in most trace backends today

As for motivation, more than anything we're keen on being able to know which version of which of our client-side programs are making requests that fit a certain criteria. E.g., identify that Web App 2 version abcdef123 is the exclusive sender of some kind of invalid query.

Problem 2: How to convey this data

I've seen data like this be conveyed via User-Agent, by just appending it to a typical UA like so:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1 Safari/605.1.15 WebApp1/1.2.3

Unfortunately, propagation based on HTTP request headers isn't a universal fix in the system considered here, for two reasons:

• Chromium-based browsers have a bug where they don’t support customizing User-Agent (see issue 571722)
• More generally, the browser’s WebSocket API doesn’t support adding custom headers at all (corroboration)

What did you expect to see?
Suggestions on how to include peer Resource information in a span, beyond just the peer's service.name.

Additional context.

Our current plan is to do the following: convey the data in W3C baggage in a manner depending on the communication protocol.

• HTTP: Use the W3C baggage request header
• WebSocket: Use whatever context propagation affordance the subprotocol being used provides—assuming it provides one. In our case the subprotocol is graphql-ws, the ubiquitous but unmaintained protocol for GraphQL communication over a WebSocket, commonly used to power GraphQL subscriptions. The client begins by sending a GQL_CONNECTION_INIT message which includes an arbitrary JSON-serializable object. Context could be injected and extracted out of this object. The successor to graphql-ws, graphql-transport-ws, has an analogous ConnectionInit message that also supports an arbitrary-object payload.

In baggage and in SERVER span attributes, use attributes

• peer.service (in spec currently), or peer.service.name (duplicates peer.service, but more consistent with the following three attributes)
• peer.service.namespace
• peer.service.instance.id
• peer.service.version

1. Which of the above should be used for service name? Both?
2. Should the latter three be put into the trace semantic conventions?
3. Would it be reasonable for HTTP instrumentations to support configurable stripping/replacing peer.* items from baggage?
4. Do semantic conventions apply to baggage too, or is baggage item key naming fully left for application owners to choose?