From 8ec82379a7e7f5186871ee9d754a575ae0a722b5 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Fri, 25 Aug 2023 12:44:45 +0200 Subject: [PATCH 01/19] Integrate OTEP-220: "Receive" operations --- docs/messaging/messaging-spans.md | 157 +++++++++++++++--------------- 1 file changed, 81 insertions(+), 76 deletions(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index 8faceefe39..f17b0eca62 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -196,12 +196,6 @@ Examples: * `AuthenticationRequest-Conversations process` * `(anonymous) publish` (`(anonymous)` being a stable identifier for an unnamed destination) -### Span kind - -A producer of a message should set the span kind to `PRODUCER` unless it synchronously waits for a response: then it should use `CLIENT`. -The processor of the message should set the kind to `CONSUMER`, unless it always sends back a reply that is directed to the producer of the message -(as opposed to e.g., a queue on which the producer happens to listen): then it should use `SERVER`. - ### Operation names The following operations related to messages are defined for these semantic conventions: @@ -212,6 +206,16 @@ The following operations related to messages are defined for these semantic conv | `receive` | A message is received from a destination by a message consumer/server. | | `process` | A message that was previously received from a destination is processed by a message consumer/server. | +### Span kind + +[Span kinds](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#spankind) +SHOULD be set according to the following table, based on the operation a span describes. + +| Operation name | Span kind| +|----------------|-------------| +| `publish` | `PRODUCER` | +| `receive` | `CONSUMER` | + ## Messaging attributes @@ -346,7 +350,14 @@ under the namespace `messaging.destination_publish.*` the broker does not have such notion, the original destination name SHOULD uniquely identify the broker. -The *receive* span is used to track the time used for receiving the message(s), whereas the *process* span(s) track the time for processing the message(s). +"Receive" spans SHOULD be created for operations of passing messages to the application when those operations are initiated by the application code. + +"Receive" spans MUST NOT be created for messages which are not forwarded to the caller, but are pre-fetched or cached by messaging libraries or SDKs. + +A single "Receive" span can account for a single message, for multiple messages (in case messages are passed for processing as batches), or for no message at all (if it is signalled that no messages were received). For each message it accounts for, the "Receive" span SHOULD link to the message's creation context. In addition, if it is possible the creation context MAY be +set as a parent of the "Receive" span. + +The "Receive" span is used to track the time used for receiving the message(s), whereas the "Process" span(s) track the time for processing the message(s). Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.operation` = `receive`. The distinction between receiving and processing of messages is not always of particular interest or sometimes hidden away in a framework (see the [Message consumption](#message-consumption) section above) and therefore the attribute can be left out. For batch receiving and processing (see the [Batch receiving](#batch-receiving) and [Batch processing](#batch-processing) examples below) in particular, the attribute SHOULD be set. @@ -358,7 +369,7 @@ Instead span kind should be set to either `CONSUMER` or `SERVER` according to th All messaging operations (`publish`, `receive`, `process`, or others not covered by this specification) can describe both single and/or batch of messages. Attributes in the `messaging.message` or `messaging.{system}.message` namespace describe individual messages. For single-message operations they SHOULD be set on corresponding span. -For batch operations, per-message attributes are usually different and cannot be set on the corresponding span. In such cases the attributes MAY be set on links. See [Batch Receiving](#batch-receiving) and [Batch Processing](#batch-processing) for more information on correlation using links. +For batch operations, per-message attributes are usually different and cannot be set on the corresponding span. In such cases the attributes SHOULD be set on links. See [Batch Receiving](#batch-receiving) and [Batch Processing](#batch-processing) for more information on correlation using links. Some messaging systems (e.g., Kafka, Azure EventGrid) allow publishing a single batch of messages to different topics. In such cases, the attributes in `messaging.destination` MAY be set on links. Instrumentations MAY set destination attributes on the span if all messages in the batch share the same destination. @@ -376,29 +387,44 @@ All attributes that are specific for a messaging system SHOULD be populated in ` ### Topic with multiple consumers -Given is a process P, that publishes a message to a topic T on messaging system MS, and two processes CA and CB, which both receive the message and process it. - -``` -Process P: | Span Prod1 | --- -Process CA: | Span CA1 | --- -Process CB: | Span CB1 | +Given is a process P, that publishes a message to a topic T on messaging system MS, and two processes CA and CB, which both receive the message. + +```mermaid +flowchart LR; + subgraph PRODUCER + direction TB + P[Span Publish A] + end + subgraph CONSUMER1 + direction TB + R1[Span Receive A 1] + end + subgraph CONSUMER2 + direction TB + R2[Span Receive A 2] + end + P-. link .-R1; + P-. link .-R2; + + classDef normal fill:green + class P,R1,R2 normal + linkStyle 0,1 color:green,stroke:green ``` -| Field or Attribute | Span Prod1 | Span CA1 | Span CB1 | + +| Field or Attribute | Span Publish A | Span Receive A 1| Span Receive A 2 | |-|-|-|-| -| Span name | `"T publish"` | `"T process"` | `"T process"` | -| Parent | | Span Prod1 | Span Prod1 | -| Links | | | | +| Span name | `T publish` | `T receive` | `T receive` | +| Parent | | | | +| Links | | `T publish` | `T publish` | | SpanKind | `PRODUCER` | `CONSUMER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | | `server.address` | `"ms"` | `"ms"` | `"ms"` | | `server.port` | `1234` | `1234` | `1234` | | `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination.name` | `"T"` | `"T"` | `"T"` | -| `messaging.operation` | | `"process"` | `"process"` | -| `messaging.message.id` | `"a1"` | `"a1"`| `"a1"` | +| `messaging.operation` | | `"receive"` | `"receive"` | +| `messaging.message.id` | `"a"` | `"a"`| `"a"` | ### Batch receiving @@ -406,62 +432,41 @@ Given is a process P, that publishes two messages to a queue Q on messaging syst Since a span can only have one parent and the propagated trace and span IDs are not known when the receiving span is started, the receiving span will have no parent and the processing spans are correlated with the producing spans using links. -``` -Process P: | Span Prod1 | Span Prod2 | --- -Process C: | Span Recv1 | - | Span Proc1 | - | Span Proc2 | +```mermaid +flowchart LR; + subgraph PRODUCER + direction TB + PA[Span Publish A] + PB[Span Publish B] + end + subgraph CONSUMER1 + direction TB + R1[Span Receive A B] + end + PA-. link .-R1; + PB-. link .-R1; + + classDef normal fill:green + class PA,PB,R1 normal + linkStyle 0,1 color:green,stroke:green ``` -| Field or Attribute | Span Prod1 | Span Prod2 | Span Recv1 | Span Proc1 | Span Proc2 | -|-|-|-|-|-|-| -| Span name | `"Q publish"` | `"Q publish"` | `"Q receive"` | `"Q process"` | `"Q process"` | -| Parent | | | | Span Recv1 | Span Recv1 | -| Links | | | | Span Prod1 | Span Prod2 | -| SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | `CONSUMER` | `CONSUMER` | -| Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | -| `server.address` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | -| `server.port` | `1234` | `1234` | `1234` | `1234` | `1234` | -| `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | -| `messaging.destination.name` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | -| `messaging.operation` | | | `"receive"` | `"process"` | `"process"` | -| `messaging.message.id` | `"a1"` | `"a2"` | | `"a1"` | `"a2"` | -| `messaging.batch.message_count` | | | 2 | | | - -### Batch processing - -Given is a process P, that publishes two messages to a queue Q on messaging system MS, and a process C, which receives them separately in two different operations (Span Recv1 and Recv2) and processes both messages in one batch (Span Proc1). - -Since each span can only have one parent, C3 should not choose a random parent out of C1 and C2, but rather rely on the implicitly selected parent as defined by the [tracing API spec](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/trace/api.md). -Depending on the implementation, the producing spans might still be available in the meta data of the messages and should be added to C3 as links. -The client library or application could also add the receiver span's SpanContext to the data structure it returns for each message. In this case, C3 could also add links to the receiver spans C1 and C2. - -The status of the batch processing span is selected by the application. Depending on the semantics of the operation. A span status `Ok` could, for example, be set only if all messages or if just at least one were properly processed. - -``` -Process P: | Span Prod1 | Span Prod2 | --- -Process C: | Span Recv1 | Span Recv2 | - | Span Proc1 | -``` - -| Field or Attribute | Span Prod1 | Span Prod2 | Span Recv1 | Span Recv2 | Span Proc1 | -|-|-|-|-|-|-| -| Span name | `"Q publish"` | `"Q publish"` | `"Q receive"` | `"Q receive"` | `"Q process"` | -| Parent | | | Span Prod1 | Span Prod2 | | -| Links | | | | | [Span Prod1, Span Prod2 ] | -| Link attributes | | | | | Span Prod1: `messaging.message.id`: `"a1"` | -| | | | | | Span Prod2: `messaging.message.id`: `"a2"` | -| SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | `CONSUMER` | `CONSUMER` | -| Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | -| `server.address` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | -| `server.port` | `1234` | `1234` | `1234` | `1234` | `1234` | -| `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | -| `messaging.destination.name` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | -| `messaging.operation` | | | `"receive"` | `"receive"` | `"process"` | -| `messaging.message.id` | `"a1"` | `"a2"` | `"a1"` | `"a2"` | | -| `messaging.batch.message_count` | | | 1 | 1 | 2 | +| Field or Attribute | Span Publish A | Span Publish B | Span Receive A B | +|-|-|-|-| +| Span name | `Q publish` | `Q publish` | `Q receive` | +| Parent | | | | +| Links | | | Span Publish A, Span Publish B | +| Link attributes | | | Span Publish A: `messaging.message.id`: `"a1"` | +| | | | Span Publish B: `messaging.message.id`: `"a2"` | +| SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | +| Status | `Ok` | `Ok` | `Ok` | +| `server.address` | `"ms"` | `"ms"` | `"ms"` | +| `server.port` | `1234` | `1234` | `1234` | +| `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | +| `messaging.destination.name` | `"Q"` | `"Q"` | `"Q"` | +| `messaging.operation` | | | `"receive"` | +| `messaging.message.id` | `"a1"` | `"a2"` | | +| `messaging.batch.message_count` | | | 2 | ## Semantic Conventions for specific messaging technologies From 1906c73c653b34915cd622382c7d269f00a70adb Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Wed, 13 Sep 2023 15:31:52 +0200 Subject: [PATCH 02/19] Update docs/messaging/messaging-spans.md Co-authored-by: Liudmila Molkova --- docs/messaging/messaging-spans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index f17b0eca62..ec15b8def8 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -352,7 +352,7 @@ the broker does not have such notion, the original destination name SHOULD uniqu "Receive" spans SHOULD be created for operations of passing messages to the application when those operations are initiated by the application code. -"Receive" spans MUST NOT be created for messages which are not forwarded to the caller, but are pre-fetched or cached by messaging libraries or SDKs. +"Receive" spans MUST NOT be created for messages until they are forwarded to the caller, but are pre-fetched or cached by messaging libraries or SDKs. A single "Receive" span can account for a single message, for multiple messages (in case messages are passed for processing as batches), or for no message at all (if it is signalled that no messages were received). For each message it accounts for, the "Receive" span SHOULD link to the message's creation context. In addition, if it is possible the creation context MAY be set as a parent of the "Receive" span. From 6e5dac1e8f4fcc72105b942faa8712b7034bd5d2 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Thu, 14 Sep 2023 15:15:55 +0200 Subject: [PATCH 03/19] Add deliver and create spans --- docs/messaging/messaging-spans.md | 76 +++++++++++++++++++++---------- 1 file changed, 53 insertions(+), 23 deletions(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index ec15b8def8..61a18b63c8 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -18,8 +18,9 @@ - [Conventions](#conventions) * [Context propagation](#context-propagation) * [Span name](#span-name) - * [Span kind](#span-kind) * [Operation names](#operation-names) + * [Span kind](#span-kind) + * [Trace structure](#trace-structure) - [Messaging attributes](#messaging-attributes) * [Attribute namespaces](#attribute-namespaces) * [Consumer attributes](#consumer-attributes) @@ -202,9 +203,10 @@ The following operations related to messages are defined for these semantic conv | Operation name | Description | | -------------- | ----------- | -| `publish` | A message is sent to a destination by a message producer/client. | -| `receive` | A message is received from a destination by a message consumer/server. | -| `process` | A message that was previously received from a destination is processed by a message consumer/server. | +| `publish` | One or more messages are provided for publishing to an intermediary. | +| `create` | A message is created. | +| `receive` | One or more messages are requested by a consumer. | +| `deliver` | One or more messages are passed to a consumer. | ### Span kind @@ -213,8 +215,46 @@ SHOULD be set according to the following table, based on the operation a span de | Operation name | Span kind| |----------------|-------------| -| `publish` | `PRODUCER` | +| `publish` | `PRODUCER`, if no `create` spans are present. | +| `create` | `PRODUCER`, | | `receive` | `CONSUMER` | +| `deliver` | `CONSUMER` | + +### Trace structure + +#### Producer spans + +"Publish" spans SHOULD be created for operations of providing messages for +sending or publishing to an intermediary. A single "Publish" span can account +for a single message, or for multiple messages (in the case of providing +messages in batches). "Create" spans MAY be created. A single "Create" span +SHOULD account only for a single message. "Create" spans SHOULD either be +children or links of the related "Publish" span. + +If a "Create" span exists for a message, its context SHOULD be injected into +the message. If no "Create" span exists, the context of the related "Publish" +span SHOULD be injected into the message. + +#### Consumer spans + +"Deliver" spans SHOULD be created for operations of passing messages to the +application when those operations are not initiated by the application code +(push-based scenarios). + +"Receive" spans SHOULD be created for operations of passing messages to the +application when those operations are initiated by the application code +(pull-based scenarios). + +"Deliver" or "Receive" spans MUST NOT be created for messages that are +pre-fetched or cached by messaging libraries or SDKs until they are forwarded +to the caller. + +A single "Deliver" or "Receive" span can account for a single message, for +multiple messages (in case messages are passed for processing as batches), or +for no message at all (if it is signalled that no messages were received). For +each message it accounts for, the "Deliver" or "Receive" span SHOULD link to +the message's creation context. In addition, if it is possible the creation +context MAY be set as a parent of the "Deliver" or "Receive" span. ## Messaging attributes @@ -350,19 +390,9 @@ under the namespace `messaging.destination_publish.*` the broker does not have such notion, the original destination name SHOULD uniquely identify the broker. -"Receive" spans SHOULD be created for operations of passing messages to the application when those operations are initiated by the application code. - -"Receive" spans MUST NOT be created for messages until they are forwarded to the caller, but are pre-fetched or cached by messaging libraries or SDKs. - -A single "Receive" span can account for a single message, for multiple messages (in case messages are passed for processing as batches), or for no message at all (if it is signalled that no messages were received). For each message it accounts for, the "Receive" span SHOULD link to the message's creation context. In addition, if it is possible the creation context MAY be -set as a parent of the "Receive" span. - -The "Receive" span is used to track the time used for receiving the message(s), whereas the "Process" span(s) track the time for processing the message(s). Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.operation` = `receive`. The distinction between receiving and processing of messages is not always of particular interest or sometimes hidden away in a framework (see the [Message consumption](#message-consumption) section above) and therefore the attribute can be left out. For batch receiving and processing (see the [Batch receiving](#batch-receiving) and [Batch processing](#batch-processing) examples below) in particular, the attribute SHOULD be set. -Even though in that case one might think that the processing span's kind should be `INTERNAL`, that kind MUST NOT be used. -Instead span kind should be set to either `CONSUMER` or `SERVER` according to the rules defined above. ### Per-message attributes @@ -426,11 +456,11 @@ flowchart LR; | `messaging.operation` | | `"receive"` | `"receive"` | | `messaging.message.id` | `"a"` | `"a"`| `"a"` | -### Batch receiving +### Batch delivering -Given is a process P, that publishes two messages to a queue Q on messaging system MS, and a process C, which receives both of them in one batch (Span Recv1) and processes each message separately (Spans Proc1 and Proc2). +Given is a process P, that publishes two messages to a queue Q on messaging system MS, and a process C, which gets both of them delivered in one batch (Span Recv1) and processes each message separately. -Since a span can only have one parent and the propagated trace and span IDs are not known when the receiving span is started, the receiving span will have no parent and the processing spans are correlated with the producing spans using links. +Since a span can only have one parent, the `deliver` span will have no parent will be correlated with the producing spans using links. ```mermaid flowchart LR; @@ -441,19 +471,19 @@ flowchart LR; end subgraph CONSUMER1 direction TB - R1[Span Receive A B] + D1[Span Deliver A B] end - PA-. link .-R1; - PB-. link .-R1; + PA-. link .-D1; + PB-. link .-D1; classDef normal fill:green class PA,PB,R1 normal linkStyle 0,1 color:green,stroke:green ``` -| Field or Attribute | Span Publish A | Span Publish B | Span Receive A B | +| Field or Attribute | Span Publish A | Span Publish B | Span Deliver A B | |-|-|-|-| -| Span name | `Q publish` | `Q publish` | `Q receive` | +| Span name | `Q publish` | `Q publish` | `Q deliver` | | Parent | | | | | Links | | | Span Publish A, Span Publish B | | Link attributes | | | Span Publish A: `messaging.message.id`: `"a1"` | From 9b3bf4519bcd2fb6bea037f8c6076e88a60c8921 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Thu, 14 Sep 2023 15:20:57 +0200 Subject: [PATCH 04/19] Fix message operation attribute --- docs/messaging/messaging-spans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index 61a18b63c8..b1a5c2bf75 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -494,7 +494,7 @@ flowchart LR; | `server.port` | `1234` | `1234` | `1234` | | `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination.name` | `"Q"` | `"Q"` | `"Q"` | -| `messaging.operation` | | | `"receive"` | +| `messaging.operation` | | | `"deliver"` | | `messaging.message.id` | `"a1"` | `"a2"` | | | `messaging.batch.message_count` | | | 2 | From 5bb94512d3f49debeed6acb294c00aaf56b13045 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Thu, 14 Sep 2023 15:34:54 +0200 Subject: [PATCH 05/19] Fix linter issues --- docs/messaging/messaging-spans.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index b1a5c2bf75..9a80f66aa7 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -28,8 +28,7 @@ * [Attributes specific to certain messaging systems](#attributes-specific-to-certain-messaging-systems) - [Examples](#examples) * [Topic with multiple consumers](#topic-with-multiple-consumers) - * [Batch receiving](#batch-receiving) - * [Batch processing](#batch-processing) + * [Batch delivering](#batch-delivering) - [Semantic Conventions for specific messaging technologies](#semantic-conventions-for-specific-messaging-technologies) @@ -390,10 +389,6 @@ under the namespace `messaging.destination_publish.*` the broker does not have such notion, the original destination name SHOULD uniquely identify the broker. -Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.operation` = `receive`. -The distinction between receiving and processing of messages is not always of particular interest or sometimes hidden away in a framework (see the [Message consumption](#message-consumption) section above) and therefore the attribute can be left out. -For batch receiving and processing (see the [Batch receiving](#batch-receiving) and [Batch processing](#batch-processing) examples below) in particular, the attribute SHOULD be set. - ### Per-message attributes All messaging operations (`publish`, `receive`, `process`, or others not covered by this specification) can describe both single and/or batch of messages. @@ -441,7 +436,6 @@ flowchart LR; linkStyle 0,1 color:green,stroke:green ``` - | Field or Attribute | Span Publish A | Span Receive A 1| Span Receive A 2 | |-|-|-|-| | Span name | `T publish` | `T receive` | `T receive` | From c67760aba71fc0dd5108196316497c58737c020e Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Thu, 14 Sep 2023 15:39:14 +0200 Subject: [PATCH 06/19] Linter checks --- docs/messaging/messaging-spans.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index 9a80f66aa7..194bf8a4a2 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -21,6 +21,8 @@ * [Operation names](#operation-names) * [Span kind](#span-kind) * [Trace structure](#trace-structure) + + [Producer spans](#producer-spans) + + [Consumer spans](#consumer-spans) - [Messaging attributes](#messaging-attributes) * [Attribute namespaces](#attribute-namespaces) * [Consumer attributes](#consumer-attributes) @@ -394,7 +396,7 @@ the broker does not have such notion, the original destination name SHOULD uniqu All messaging operations (`publish`, `receive`, `process`, or others not covered by this specification) can describe both single and/or batch of messages. Attributes in the `messaging.message` or `messaging.{system}.message` namespace describe individual messages. For single-message operations they SHOULD be set on corresponding span. -For batch operations, per-message attributes are usually different and cannot be set on the corresponding span. In such cases the attributes SHOULD be set on links. See [Batch Receiving](#batch-receiving) and [Batch Processing](#batch-processing) for more information on correlation using links. +For batch operations, per-message attributes are usually different and cannot be set on the corresponding span. In such cases the attributes SHOULD be set on links. See [Batch Delivering](#batch-delivering) for more information on correlation using links. Some messaging systems (e.g., Kafka, Azure EventGrid) allow publishing a single batch of messages to different topics. In such cases, the attributes in `messaging.destination` MAY be set on links. Instrumentations MAY set destination attributes on the span if all messages in the batch share the same destination. From 6c771d787d5a75d23d0aa6d963c2995a6b620c89 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Tue, 26 Sep 2023 12:08:43 +0200 Subject: [PATCH 07/19] Update docs/messaging/messaging-spans.md Co-authored-by: Liudmila Molkova --- docs/messaging/messaging-spans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index 194bf8a4a2..0f421ebfe0 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -217,7 +217,7 @@ SHOULD be set according to the following table, based on the operation a span de | Operation name | Span kind| |----------------|-------------| | `publish` | `PRODUCER`, if no `create` spans are present. | -| `create` | `PRODUCER`, | +| `create` | `PRODUCER` | | `receive` | `CONSUMER` | | `deliver` | `CONSUMER` | From 8acf50d8a928a0928ad2507b526ca434e0ac2410 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Tue, 26 Sep 2023 14:28:59 +0200 Subject: [PATCH 08/19] Apply suggestions from code review Co-authored-by: Liudmila Molkova --- docs/messaging/messaging-spans.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index 0f421ebfe0..35966ddfaf 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -449,7 +449,7 @@ flowchart LR; | `server.port` | `1234` | `1234` | `1234` | | `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination.name` | `"T"` | `"T"` | `"T"` | -| `messaging.operation` | | `"receive"` | `"receive"` | +| `messaging.operation` | `"publish"` | `"receive"` | `"receive"` | | `messaging.message.id` | `"a"` | `"a"`| `"a"` | ### Batch delivering @@ -490,7 +490,7 @@ flowchart LR; | `server.port` | `1234` | `1234` | `1234` | | `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination.name` | `"Q"` | `"Q"` | `"Q"` | -| `messaging.operation` | | | `"deliver"` | +| `messaging.operation` | `"publish"` | `"publish"` | `"deliver"` | | `messaging.message.id` | `"a1"` | `"a2"` | | | `messaging.batch.message_count` | | | 2 | From b83847ea488f196caf2e62c559ddd11a07bf3451 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Thu, 28 Sep 2023 17:46:25 +0200 Subject: [PATCH 09/19] Update docs/messaging/messaging-spans.md Co-authored-by: Joao Grassi --- docs/messaging/messaging-spans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index 35966ddfaf..7a102ff54b 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -456,7 +456,7 @@ flowchart LR; Given is a process P, that publishes two messages to a queue Q on messaging system MS, and a process C, which gets both of them delivered in one batch (Span Recv1) and processes each message separately. -Since a span can only have one parent, the `deliver` span will have no parent will be correlated with the producing spans using links. +Since a span can only have one parent, the `deliver` span will have no parent and will be correlated with the producing spans using links. ```mermaid flowchart LR; From 7561b25c3152e285cfd2601a99ab750e2b1ee996 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Thu, 5 Oct 2023 15:52:20 +0200 Subject: [PATCH 10/19] PR comments --- docs/messaging/messaging-spans.md | 35 ++++++++++++++++++++----------- 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index 7a102ff54b..6b0544bd58 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -216,11 +216,22 @@ SHOULD be set according to the following table, based on the operation a span de | Operation name | Span kind| |----------------|-------------| -| `publish` | `PRODUCER`, if no `create` spans are present. | -| `create` | `PRODUCER` | +| `publish` | `PRODUCER`, if the context of the "Publish" span is used as creation context. | +| `create` | `PRODUCER`, | | `receive` | `CONSUMER` | | `deliver` | `CONSUMER` | +For cases not covered by the table above, the span kind should be set according +to the [generic specification about span kinds](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#spankind), +e. g. it should be set to CLIENT for the "Publish" span if its context is not +used as creation context and if the "Publish" span models a synchronous call to +the intermediary. + +Setting span kinds according to this table ensures that span links between +consumers and producers always exist between a PRODUCER span on the producer +side and a CONSUMER span on the consumer side. This allows analysis tools to +interpret linked traces without the need for additional semantic hints. + ### Trace structure #### Producer spans @@ -250,12 +261,12 @@ application when those operations are initiated by the application code pre-fetched or cached by messaging libraries or SDKs until they are forwarded to the caller. -A single "Deliver" or "Receive" span can account for a single message, for -multiple messages (in case messages are passed for processing as batches), or -for no message at all (if it is signalled that no messages were received). For -each message it accounts for, the "Deliver" or "Receive" span SHOULD link to -the message's creation context. In addition, if it is possible the creation -context MAY be set as a parent of the "Deliver" or "Receive" span. +A single "Deliver" or "Receive" span can account for a single message, for a +batch of messages, or for no message at all (if it is signalled that no +messages were received). For each message it accounts for, the "Deliver" or +"Receive" span SHOULD link to the message's creation context. In addition, if +it is possible the creation context MAY be set as a parent of the "Deliver" or +"Receive" span. ## Messaging attributes @@ -414,7 +425,7 @@ All attributes that are specific for a messaging system SHOULD be populated in ` ### Topic with multiple consumers -Given is a process P, that publishes a message to a topic T on messaging system MS, and two processes CA and CB, which both receive the message. +Given is a publisher that publishes a message to a queue "T" on RabbitMQ, and two consumers which both receive the message. ```mermaid flowchart LR; @@ -454,7 +465,7 @@ flowchart LR; ### Batch delivering -Given is a process P, that publishes two messages to a queue Q on messaging system MS, and a process C, which gets both of them delivered in one batch (Span Recv1) and processes each message separately. +Given is a publisher that publishes two messages to a topic "Q" on Kafka, and a consumer which gets both messages delivered in one batch. Since a span can only have one parent, the `deliver` span will have no parent and will be correlated with the producing spans using links. @@ -473,7 +484,7 @@ flowchart LR; PB-. link .-D1; classDef normal fill:green - class PA,PB,R1 normal + class PA,PB,D1 normal linkStyle 0,1 color:green,stroke:green ``` @@ -488,7 +499,7 @@ flowchart LR; | Status | `Ok` | `Ok` | `Ok` | | `server.address` | `"ms"` | `"ms"` | `"ms"` | | `server.port` | `1234` | `1234` | `1234` | -| `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | +| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | | `messaging.destination.name` | `"Q"` | `"Q"` | `"Q"` | | `messaging.operation` | `"publish"` | `"publish"` | `"deliver"` | | `messaging.message.id` | `"a1"` | `"a2"` | | From 7527f00b0177633ea1548236b759ca027881b3af Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Thu, 5 Oct 2023 16:10:32 +0200 Subject: [PATCH 11/19] PR comments --- docs/messaging/messaging-spans.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index 6b0544bd58..80bc852710 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -204,10 +204,10 @@ The following operations related to messages are defined for these semantic conv | Operation name | Description | | -------------- | ----------- | -| `publish` | One or more messages are provided for publishing to an intermediary. | -| `create` | A message is created. | -| `receive` | One or more messages are requested by a consumer. | -| `deliver` | One or more messages are passed to a consumer. | +| `publish` | One or more messages are provided for publishing to an intermediary. If a single message is published, the context of the "Publish" span can be used as the creation context and no "Create" span needs to be created. | +| `create` | A message is created. "Create" spans always refer to a single message and are used to provide a unique creation context for messages in batch publishing scenarios. | +| `receive` | One or more messages are requested by a consumer. This operation refers to pull-based scenarios, where consumers explicitly call methods of messaging SDKs to receive messages. | +| `deliver` | One or more messages are passed to a consumer. This operation refers to push-based scenarios, where consumer register callbacks which get called by messaging SDKs. | ### Span kind @@ -244,8 +244,9 @@ SHOULD account only for a single message. "Create" spans SHOULD either be children or links of the related "Publish" span. If a "Create" span exists for a message, its context SHOULD be injected into -the message. If no "Create" span exists, the context of the related "Publish" -span SHOULD be injected into the message. +the message. If no "Create" span exists and no custom creation context is +injected into the message, the context of the related "Publish" span SHOULD be +injected into the message. #### Consumer spans From 76bc55874b7ef2c98360102bc70cff77e27ac018 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Thu, 5 Oct 2023 17:31:57 +0200 Subject: [PATCH 12/19] Remove mentions of optional parent/child relationships --- docs/messaging/messaging-spans.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index 80bc852710..a628320e23 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -265,9 +265,7 @@ to the caller. A single "Deliver" or "Receive" span can account for a single message, for a batch of messages, or for no message at all (if it is signalled that no messages were received). For each message it accounts for, the "Deliver" or -"Receive" span SHOULD link to the message's creation context. In addition, if -it is possible the creation context MAY be set as a parent of the "Deliver" or -"Receive" span. +"Receive" span SHOULD link to the message's creation context. ## Messaging attributes @@ -468,8 +466,6 @@ flowchart LR; Given is a publisher that publishes two messages to a topic "Q" on Kafka, and a consumer which gets both messages delivered in one batch. -Since a span can only have one parent, the `deliver` span will have no parent and will be correlated with the producing spans using links. - ```mermaid flowchart LR; subgraph PRODUCER From cfa9fc48ed463ca6b8e2884ccac94cff3080a7da Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Tue, 17 Oct 2023 11:20:21 +0200 Subject: [PATCH 13/19] PR comments --- docs/messaging/messaging-spans.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index a628320e23..61d3102502 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -216,8 +216,8 @@ SHOULD be set according to the following table, based on the operation a span de | Operation name | Span kind| |----------------|-------------| -| `publish` | `PRODUCER`, if the context of the "Publish" span is used as creation context. | -| `create` | `PRODUCER`, | +| `publish` | `PRODUCER` if the context of the "Publish" span is used as creation context. | +| `create` | `PRODUCER` | | `receive` | `CONSUMER` | | `deliver` | `CONSUMER` | @@ -424,7 +424,7 @@ All attributes that are specific for a messaging system SHOULD be populated in ` ### Topic with multiple consumers -Given is a publisher that publishes a message to a queue "T" on RabbitMQ, and two consumers which both receive the message. +Given is a publisher that publishes a message to a topic "T" on Kafka, and two consumers which both get the message delivered. ```mermaid flowchart LR; @@ -434,11 +434,11 @@ flowchart LR; end subgraph CONSUMER1 direction TB - R1[Span Receive A 1] + R1[Span Deliver A 1] end subgraph CONSUMER2 direction TB - R2[Span Receive A 2] + R2[Span Deliver A 2] end P-. link .-R1; P-. link .-R2; @@ -448,23 +448,23 @@ flowchart LR; linkStyle 0,1 color:green,stroke:green ``` -| Field or Attribute | Span Publish A | Span Receive A 1| Span Receive A 2 | +| Field or Attribute | Span Publish A | Span Deliver A 1| Span Deliver A 2 | |-|-|-|-| -| Span name | `T publish` | `T receive` | `T receive` | +| Span name | `T publish` | `T deliver` | `T deliver` | | Parent | | | | | Links | | `T publish` | `T publish` | | SpanKind | `PRODUCER` | `CONSUMER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | | `server.address` | `"ms"` | `"ms"` | `"ms"` | | `server.port` | `1234` | `1234` | `1234` | -| `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | +| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | | `messaging.destination.name` | `"T"` | `"T"` | `"T"` | -| `messaging.operation` | `"publish"` | `"receive"` | `"receive"` | +| `messaging.operation` | `"publish"` | `"deliver"` | `"deliver"` | | `messaging.message.id` | `"a"` | `"a"`| `"a"` | ### Batch delivering -Given is a publisher that publishes two messages to a topic "Q" on Kafka, and a consumer which gets both messages delivered in one batch. +Given is a publisher that publishes two messages to a queue "Q" on RabbitMQ, and a consumer which receives both messages in one batch. ```mermaid flowchart LR; @@ -475,7 +475,7 @@ flowchart LR; end subgraph CONSUMER1 direction TB - D1[Span Deliver A B] + D1[Span Receive A B] end PA-. link .-D1; PB-. link .-D1; @@ -485,9 +485,9 @@ flowchart LR; linkStyle 0,1 color:green,stroke:green ``` -| Field or Attribute | Span Publish A | Span Publish B | Span Deliver A B | +| Field or Attribute | Span Publish A | Span Publish B | Span Receive A B | |-|-|-|-| -| Span name | `Q publish` | `Q publish` | `Q deliver` | +| Span name | `Q publish` | `Q publish` | `Q receive` | | Parent | | | | | Links | | | Span Publish A, Span Publish B | | Link attributes | | | Span Publish A: `messaging.message.id`: `"a1"` | @@ -496,9 +496,9 @@ flowchart LR; | Status | `Ok` | `Ok` | `Ok` | | `server.address` | `"ms"` | `"ms"` | `"ms"` | | `server.port` | `1234` | `1234` | `1234` | -| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | +| `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination.name` | `"Q"` | `"Q"` | `"Q"` | -| `messaging.operation` | `"publish"` | `"publish"` | `"deliver"` | +| `messaging.operation` | `"publish"` | `"publish"` | `"receive"` | | `messaging.message.id` | `"a1"` | `"a2"` | | | `messaging.batch.message_count` | | | 2 | From dcffdf753880b773d1eea5a98845c653bf35cade Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Tue, 17 Oct 2023 11:24:09 +0200 Subject: [PATCH 14/19] PR comments --- docs/messaging/messaging-spans.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index 61d3102502..f1895b709c 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -30,7 +30,7 @@ * [Attributes specific to certain messaging systems](#attributes-specific-to-certain-messaging-systems) - [Examples](#examples) * [Topic with multiple consumers](#topic-with-multiple-consumers) - * [Batch delivering](#batch-delivering) + * [Batch receiving](#batch-receiving) - [Semantic Conventions for specific messaging technologies](#semantic-conventions-for-specific-messaging-technologies) @@ -462,7 +462,7 @@ flowchart LR; | `messaging.operation` | `"publish"` | `"deliver"` | `"deliver"` | | `messaging.message.id` | `"a"` | `"a"`| `"a"` | -### Batch delivering +### Batch receiving Given is a publisher that publishes two messages to a queue "Q" on RabbitMQ, and a consumer which receives both messages in one batch. From 21504639285e98f7a4f6befbf6ec50926a49308f Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Tue, 17 Oct 2023 16:43:57 +0200 Subject: [PATCH 15/19] PR comments --- docs/messaging/messaging-spans.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index f1895b709c..d7857f6825 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -424,7 +424,7 @@ All attributes that are specific for a messaging system SHOULD be populated in ` ### Topic with multiple consumers -Given is a publisher that publishes a message to a topic "T" on Kafka, and two consumers which both get the message delivered. +Given is a publisher that publishes a message to a topic exchange "T" on RabbitMQ, and two consumers which both get the message delivered. ```mermaid flowchart LR; @@ -457,14 +457,14 @@ flowchart LR; | Status | `Ok` | `Ok` | `Ok` | | `server.address` | `"ms"` | `"ms"` | `"ms"` | | `server.port` | `1234` | `1234` | `1234` | -| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | +| `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination.name` | `"T"` | `"T"` | `"T"` | | `messaging.operation` | `"publish"` | `"deliver"` | `"deliver"` | | `messaging.message.id` | `"a"` | `"a"`| `"a"` | ### Batch receiving -Given is a publisher that publishes two messages to a queue "Q" on RabbitMQ, and a consumer which receives both messages in one batch. +Given is a publisher that publishes two messages to a topic "Q" on Kafka, and a consumer which receives both messages in one batch. ```mermaid flowchart LR; @@ -496,7 +496,7 @@ flowchart LR; | Status | `Ok` | `Ok` | `Ok` | | `server.address` | `"ms"` | `"ms"` | `"ms"` | | `server.port` | `1234` | `1234` | `1234` | -| `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | +| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | | `messaging.destination.name` | `"Q"` | `"Q"` | `"Q"` | | `messaging.operation` | `"publish"` | `"publish"` | `"receive"` | | `messaging.message.id` | `"a1"` | `"a2"` | | From b6397c47e9736d3fa15d45c097be91fac0a1b609 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Tue, 24 Oct 2023 11:52:32 +0200 Subject: [PATCH 16/19] PR comments --- docs/messaging/messaging-spans.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index d7857f6825..53e85a082d 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -243,10 +243,13 @@ messages in batches). "Create" spans MAY be created. A single "Create" span SHOULD account only for a single message. "Create" spans SHOULD either be children or links of the related "Publish" span. -If a "Create" span exists for a message, its context SHOULD be injected into -the message. If no "Create" span exists and no custom creation context is -injected into the message, the context of the related "Publish" span SHOULD be -injected into the message. +If a user provides a custom creation context in a message, this context SHOULD +NOT be modified, the "Create" span SHOULD NOT be created, and "Publish" span +SHOULD link to the custom creation context. Otherwise, if a "Create" span +exists for a message, its context SHOULD be injected into the message. If no +"Create" span exists and no custom creation context is injected into the +message, the context of the related "Publish" span SHOULD be injected into the +message. #### Consumer spans From bcd108c046775dda0398786e3b2dfc3325281cbd Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Tue, 24 Oct 2023 11:54:27 +0200 Subject: [PATCH 17/19] PR comments --- docs/messaging/messaging-spans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index 53e85a082d..a90b8f6b88 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -244,7 +244,7 @@ SHOULD account only for a single message. "Create" spans SHOULD either be children or links of the related "Publish" span. If a user provides a custom creation context in a message, this context SHOULD -NOT be modified, the "Create" span SHOULD NOT be created, and "Publish" span +NOT be modified, a "Create" span SHOULD NOT be created, and the "Publish" span SHOULD link to the custom creation context. Otherwise, if a "Create" span exists for a message, its context SHOULD be injected into the message. If no "Create" span exists and no custom creation context is injected into the From 0ec3aed18a0bf13fc05001706a6f7faded6c4305 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Wed, 25 Oct 2023 13:33:41 +0200 Subject: [PATCH 18/19] PR comments --- docs/messaging/messaging-spans.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index a90b8f6b88..622542f515 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -255,7 +255,8 @@ message. "Deliver" spans SHOULD be created for operations of passing messages to the application when those operations are not initiated by the application code -(push-based scenarios). +(push-based scenarios). A "Deliver" span covers the duration of such an +operation, which is usually a callback or handler. "Receive" spans SHOULD be created for operations of passing messages to the application when those operations are initiated by the application code From e872d0cf0a8039aa1e490a9789af349c01f0f4d4 Mon Sep 17 00:00:00 2001 From: Johannes Tax Date: Mon, 30 Oct 2023 17:20:40 +0100 Subject: [PATCH 19/19] Fix link --- docs/messaging/messaging-spans.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/messaging/messaging-spans.md b/docs/messaging/messaging-spans.md index a27dfac945..6ff2bc03a3 100644 --- a/docs/messaging/messaging-spans.md +++ b/docs/messaging/messaging-spans.md @@ -399,7 +399,7 @@ the broker doesn't have such notion, the original destination name SHOULD unique All messaging operations (`publish`, `receive`, `process`, or others not covered by this specification) can describe both single and/or batch of messages. Attributes in the `messaging.message` or `messaging.{system}.message` namespace describe individual messages. For single-message operations they SHOULD be set on corresponding span. -For batch operations, per-message attributes are usually different and cannot be set on the corresponding span. In such cases the attributes SHOULD be set on links. See [Batch Delivering](#batch-delivering) for more information on correlation using links. +For batch operations, per-message attributes are usually different and cannot be set on the corresponding span. In such cases the attributes SHOULD be set on links. See [Batch receiving](#batch-receiving) for more information on correlation using links. Some messaging systems (e.g., Kafka, Azure EventGrid) allow publishing a single batch of messages to different topics. In such cases, the attributes in `messaging.destination` MAY be set on links. Instrumentations MAY set destination attributes on the span if all messages in the batch share the same destination.