Restructured messaging semconv

specification/README.md

@@ -14,6 +14,7 @@ Semantic Conventions are defined for the following areas:
 * [HTTP](http/README.md): Semantic Conventions for HTTP client and server operations.
 * [Database](database/README.md): Semantic Conventions for database operations.
 * [RPC](rpc/README.md): Semantic Conventions for RPC client and server operations.
+* [Messaging](messaging/README.md): Semantic Conventions for Messaging operations and systems.
 * [System](system/README.md): System Semantic Conventions.
 * *Other areas can be found in the signal specific Semantic Conventions below*
 

specification/faas/aws-lambda.md

@@ -114,9 +114,9 @@ added as a link to the span. This means the span may have as many links as messa
 See [compatibility](../../supplementary-guidelines/compatibility/aws.md#context-propagation) for more info.
 
 - [`faas.trigger`][faas] MUST be set to `pubsub`.
-- [`messaging.operation`](/specification/trace/semantic_conventions/messaging.md) MUST be set to `process`.
-- [`messaging.system`](/specification/trace/semantic_conventions/messaging.md) MUST be set to `AmazonSQS`.
-- [`messaging.destination.kind` or `messaging.source.kind`](/specification/trace/semantic_conventions/messaging.md#messaging-attributes) MUST be set to `queue`.
+- [`messaging.operation`](/specification/messaging/messaging-spans.md) MUST be set to `process`.
+- [`messaging.system`](/specification/messaging/messaging-spans.md) MUST be set to `AmazonSQS`.
+- [`messaging.destination.kind` or `messaging.source.kind`](/specification/messaging/messaging-spans.md#messaging-attributes) MUST be set to `queue`.
 
 ### SQS Message
 
@@ -128,10 +128,10 @@ added as a link to the span.
 See [compatibility](../../supplementary-guidelines/compatibility/aws.md#context-propagation) for more info.
 
 - [`faas.trigger`][faas] MUST be set to `pubsub`.
-- [`messaging.operation`](/specification/trace/semantic_conventions/messaging.md#messaging-attributes) MUST be set to `process`.
-- [`messaging.system`](/specification/trace/semantic_conventions/messaging.md#messaging-attributes) MUST be set to `AmazonSQS`.
+- [`messaging.operation`](/specification/messaging/messaging-spans.md#messaging-attributes) MUST be set to `process`.
+- [`messaging.system`](/specification/messaging/messaging-spans.md#messaging-attributes) MUST be set to `AmazonSQS`.
 
-Other [Messaging attributes](/specification/trace/semantic_conventions/messaging.md#messaging-attributes) SHOULD be set based on the available information in the SQS message
+Other [Messaging attributes](/specification/messaging/messaging-spans.md#messaging-attributes) SHOULD be set based on the available information in the SQS message
 event.
 
 Note that `AWSTraceHeader` is the only supported mechanism for propagating `Context` in instrumentation for SQS

specification/faas/faas-spans.md

@@ -217,7 +217,8 @@ The function responsibility is to provide an answer to an inbound HTTP request.
 
 A function is set to be executed when messages are sent to a messaging system.
 In this case, multiple messages could be batch and forwarded at once to the same function invocation.
-Therefore, a different root span of type `faas` MUST be created for each message processed by the function, following the [Messaging systems semantic conventions](/specification/trace/semantic_conventions/messaging.md).
+Therefore, a different root span of type `faas` MUST be created for each message processed by the function,
+following the [Messaging systems semantic conventions](/specification/messaging/messaging-spans.md).
 This way, it is possible to correlate each individual message with its invocation sender.
 
 ### Timer

specification/general/trace-general.md

@@ -19,7 +19,7 @@ The following semantic conventions for spans are defined:
 * [HTTP](/specification/http/http-spans.md): For HTTP client and server spans.
 * [Database](/specification/database/database-spans.md): For SQL and NoSQL client call spans.
 * [RPC/RMI](/specification/rpc/rpc-spans.md): For remote procedure call (e.g., gRPC) spans.
-* [Messaging](/specification/trace/semantic_conventions/messaging.md): For messaging systems (queues, publish/subscribe, etc.) spans.
+* [Messaging](/specification/messaging/messaging-spans.md): For messaging systems (queues, publish/subscribe, etc.) spans.
 * [FaaS](/specification/faas/faas-spans.md): For [Function as a Service](https://en.wikipedia.org/wiki/Function_as_a_service) (e.g., AWS Lambda) spans.
 * [Exceptions](/specification/exceptions/exceptions-spans.md): For recording exceptions associated with a span.
 * [Compatibility](trace-compatibility.md): For spans generated by compatibility components, e.g. OpenTracing Shim layer.

specification/messaging/README.md

@@ -0,0 +1,17 @@
+# Semantic conventions for Messaging systems
+
+**Status**: [Experimental][DocumentStatus]
+
+This document defines semantic conventions for messaging systems spans, metrics and logs.
+
+Semantic conventions for messaging systems are defined for the following signals:
+
+* [Messaging Spans](messaging-spans.md): Semantic Conventions for messaging *spans*.
+
+Technology specific semantic conventions are defined for the following messaging systems:
+
+* [Kafka](kafka.md): Semantic Conventions for *Apache Kafka*.
+* [RabbitMQ](rabbitmq.md): Semantic Conventions for *RabbitMQ*.
+* [RocketMQ](rocketmq.md): Semantic Conventions for *Apache RocketMQ*.
+
+[DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/blob/v1.21.0/specification/document-status.md

specification/metrics/semantic_conventions/instrumentation/kafka.md → specification/messaging/kafka.md

@@ -2,21 +2,97 @@
 linkTitle: Kafka
 --->
 
-# Instrumenting Kafka
+# Semantic Conventions for Kafka
 
 **Status**: [Experimental][DocumentStatus]
 
-This document defines how to apply semantic conventions when instrumenting Kafka.
-
 <!-- toc -->
 
-- [Kafka Metrics](#kafka-metrics)
-- [Kafka Producer Metrics](#kafka-producer-metrics)
-- [Kafka Consumer Metrics](#kafka-consumer-metrics)
+- [Span attributes](#span-attributes)
+- [Examples](#examples)
+  * [Apache Kafka with Quarkus or Spring Boot Example](#apache-kafka-with-quarkus-or-spring-boot-example)
+- [Metrics](#metrics)
+  * [General Metrics](#general-metrics)
+  * [Producer Metrics](#producer-metrics)
+  * [Consumer Metrics](#consumer-metrics)
 
 <!-- tocstop -->
 
-## Kafka Metrics
+The Semantic Conventions for [Apache Kafka](https://kafka.apache.org/) extend and override the [Messaging Semantic Conventions](README.md)
+that describe common messaging operations attributes in addition to the Semantic Conventions
+described on this page.
+
+`messaging.system` MUST be set to `"kafka"`.
+
+## Span attributes
+
+For Apache Kafka, the following additional attributes are defined:
+
+<!-- semconv messaging.kafka -->
+| Attribute  | Type | Description  | Examples  | Requirement Level |
+|---|---|---|---|---|
+| `messaging.kafka.message.key` | string | Message keys in Kafka are used for grouping alike messages to ensure they're processed on the same partition. They differ from `messaging.message.id` in that they're not unique. If the key is `null`, the attribute MUST NOT be set. [1] | `myKey` | Recommended |
+| `messaging.kafka.consumer.group` | string | Name of the Kafka Consumer Group that is handling the message. Only applies to consumers, not producers. | `my-group` | Recommended |
+| `messaging.kafka.destination.partition` | int | Partition the message is sent to. | `2` | Recommended |
+| `messaging.kafka.message.offset` | int | The offset of a record in the corresponding Kafka partition. | `42` | Recommended |
+| `messaging.kafka.message.tombstone` | boolean | A boolean that is true if the message is a tombstone. |  | Conditionally Required: [2] |
+
+**[1]:** If the key type is not string, it's string representation has to be supplied for the attribute. If the key has no unambiguous, canonical string form, don't include its value.
+
+**[2]:** If value is `true`. When missing, the value is assumed to be `false`.
+<!-- endsemconv -->
+
+For Apache Kafka producers, [`peer.service`](/specification/general/general-attributes.md#general-remote-service-attributes) SHOULD be set to the name of the broker or service the message will be sent to.
+The `service.name` of a Consumer's Resource SHOULD match the `peer.service` of the Producer, when the message is directly passed to another service.
+If an intermediary broker is present, `service.name` and `peer.service` will not be the same.
+
+`messaging.client_id` SHOULD be set to the `client-id` of consumers, or to the `client.id` property of producers.
+
+## Examples
+
+### Apache Kafka with Quarkus or Spring Boot Example
+
+Given is a process P, that publishes a message to a topic T1 on Apache Kafka.
+One process, CA, receives the message and publishes a new message to a topic T2 that is then received and processed by CB.
+
+Frameworks such as Quarkus and Spring Boot separate processing of a received message from producing subsequent messages out.
+For this reason, receiving (Span Rcv1) is the parent of both processing (Span Proc1) and producing a new message (Span Prod2).
+The span representing message receiving (Span Rcv1) should not set `messaging.operation` to `receive`,
+as it does not only receive the message but also converts the input message to something suitable for the processing operation to consume and creates the output message from the result of processing.
+
+```
+Process P:  | Span Prod1 |
+--
+Process CA:              | Span Rcv1 |
+                                | Span Proc1 |
+                                  | Span Prod2 |
+--
+Process CB:                           | Span Rcv2 |
+```
+
+| Field or Attribute | Span Prod1 | Span Rcv1 | Span Proc1 | Span Prod2 | Span Rcv2
+|-|-|-|-|-|-|
+| Span name | `"T1 publish"` | `"T1 receive"` | `"T1 process"` | `"T2 publish"` | `"T2 receive`" |
+| Parent |  | Span Prod1 | Span Rcv1 | Span Rcv1 | Span Prod2 |
+| Links |  |  | |  |  |
+| SpanKind | `PRODUCER` | `CONSUMER` | `CONSUMER` | `PRODUCER` | `CONSUMER` |
+| Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` |
+| `peer.service` | `"myKafka"` |  |  | `"myKafka"` |  |
+| `service.name` |  | `"myConsumer1"` | `"myConsumer1"` |  | `"myConsumer2"` |
+| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` |
+| `messaging.destination.name` | `"T1"` | `"T1"` | `"T1"` | `"T2"` | `"T2"` |
+| `messaging.operation` |  |  | `"process"` |  | `"receive"` |
+| `messaging.client_id` |  | `"5"` | `"5"` | `"5"` | `"8"` |
+| `messaging.kafka.message.key` | `"myKey"` | `"myKey"` | `"myKey"` | `"anotherKey"` | `"anotherKey"` |
+| `messaging.kafka.consumer.group` |  | `"my-group"` | `"my-group"` |  | `"another-group"` |
+| `messaging.kafka.partition` | `"1"` | `"1"` | `"1"` | `"3"` | `"3"` |
+| `messaging.kafka.message.offset` | `"12"` | `"12"` | `"12"` | `"32"` | `"32"` |
+
+## Metrics
+
+This section defines how to apply semantic conventions when collecting Kafka metrics.
+
+### General Metrics
 
 **Description:** General Kafka metrics.
 
@@ -46,7 +122,7 @@ This document defines how to apply semantic conventions when instrumenting Kafka
 | messaging.kafka.partition.replicas.in_sync   | UpDownCounter | Int64      | replicas | `{replica}` | Number of synchronized replicas of partition | `topic` | The ID (integer) of a topic |
 |                                              |               |            |          |              |                                              | `partition` | The number (integer) of the partition|
 
-## Kafka Producer Metrics
+### Producer Metrics
 
 **Description:** Kafka Producer level metrics.
 
@@ -65,7 +141,7 @@ This document defines how to apply semantic conventions when instrumenting Kafka
 | messaging.kafka.producer.record-sent.rate     | Gauge         | Double     | records sent rate | `{record_sent}/s` | The average number of records sent per second for a specific topic.  | `client-id` | `client-id` value  |
 |                                               |               |            |                   |                    |                                                                      | `topic`     | topic name         |
 
-## Kafka Consumer Metrics
+### Consumer Metrics
 
 **Description:** Kafka Consumer level metrics.