From 471faf48626ea8e2887a9a1eaf6ac18ff9e0d4af Mon Sep 17 00:00:00 2001 From: Drew Robbins Date: Wed, 15 Nov 2023 16:14:33 +0900 Subject: [PATCH 1/3] Add openai metrics --- docs/ai/openai-metrics.md | 326 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 326 insertions(+) create mode 100644 docs/ai/openai-metrics.md diff --git a/docs/ai/openai-metrics.md b/docs/ai/openai-metrics.md new file mode 100644 index 0000000000..18424875a2 --- /dev/null +++ b/docs/ai/openai-metrics.md @@ -0,0 +1,326 @@ + + +# Semantic Conventions for OpenAI Metrics + +**Status**: [Experimental][DocumentStatus] + +This document defines semantic conventions for OpenAI client metrics. + + + + + +- [Chat completions](#chat-completions) + * [Metric: `openai.chat_completions.tokens`](#metric-openaichat_completionstokens) + * [Metric: `openai.chat_completions.choices`](#metric-openaichat_completionschoices) + * [Metric: `openai.chat_completions.duration`](#metric-openaichat_completionsduration) +- [Embeddings](#embeddings) + * [Metric: `openai.embeddings.tokens`](#metric-openaiembeddingstokens) + * [Metric: `openai.embeddings.vector_size`](#metric-openaiembeddingsvector_size) + * [Metric: `openai.embeddings.duration`](#metric-openaiembeddingsduration) +- [Image generation](#image-generation) + * [Metric: `openai.image_generations.duration`](#metric-openaiimage_generationsduration) + + + +## Chat completions + +### Metric: `openai.chat_completions.tokens` + +**Status**: [Experimental][DocumentStatus] + +This metric is required. + + +| Name | Instrument Type | Unit (UCUM) | Description | +| -------- | --------------- | ----------- | -------------- | +| `openai.chat_completions.tokens` | Counter | `{token}` | Number of tokens used in prompt and completions | + + + +| Attribute | Type | Description | Examples | Requirement Level | +|---|---|---|---|---| +| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | +| `openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt`; `completion` | Required | +| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | + +**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality. +Instrumentations SHOULD document the list of errors they report. + +The cardinality of `error.type` within one instrumentation library SHOULD be low, but +telemetry consumers that aggregate data from multiple instrumentation libraries and applications +should be prepared for `error.type` to have high cardinality at query time, when no +additional filters are applied. + +If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`. + +**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent +the server address behind any intermediaries (e.g. proxies) if it's available. + +`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used. + +| Value | Description | +|---|---| +| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. | + + +### Metric: `openai.chat_completions.choices` + +**Status**: [Experimental][DocumentStatus] + +This metric is required. + + +| Name | Instrument Type | Unit (UCUM) | Description | +| -------- | --------------- | ----------- | -------------- | +| `openai.chat_completions.choices` | Counter | `{choice}` | Number of choices returned by chat completions call | + + + +| Attribute | Type | Description | Examples | Requirement Level | +|---|---|---|---|---| +| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | +| `openai.choice.finish_reason` | string | Finish reason for single chat completion choice | `stop`; `length`; `content_filter` | Conditionally Required: if and only if it was returned. | +| `openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt`; `completion` | Required | +| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | + +**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality. +Instrumentations SHOULD document the list of errors they report. + +The cardinality of `error.type` within one instrumentation library SHOULD be low, but +telemetry consumers that aggregate data from multiple instrumentation libraries and applications +should be prepared for `error.type` to have high cardinality at query time, when no +additional filters are applied. + +If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`. + +**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent +the server address behind any intermediaries (e.g. proxies) if it's available. + +`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used. + +| Value | Description | +|---|---| +| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. | + + + +### Metric: `openai.chat_completions.duration` + +**Status**: [Experimental][DocumentStatus] + +This metric is required. + +This metric SHOULD be specified with +[`ExplicitBucketBoundaries`](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/metrics/api.md#instrument-advice) +of `[ 0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10 ]`. + + +| Name | Instrument Type | Unit (UCUM) | Description | +| -------- | --------------- | ----------- | -------------- | +| `openai.chat_completions.duration` | Histogram | `s` | Duration of chat completion operation | + + + +| Attribute | Type | Description | Examples | Requirement Level | +|---|---|---|---|---| +| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | +| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | + +**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality. +Instrumentations SHOULD document the list of errors they report. + +The cardinality of `error.type` within one instrumentation library SHOULD be low, but +telemetry consumers that aggregate data from multiple instrumentation libraries and applications +should be prepared for `error.type` to have high cardinality at query time, when no +additional filters are applied. + +If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`. + +**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent +the server address behind any intermediaries (e.g. proxies) if it's available. + +`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used. + +| Value | Description | +|---|---| +| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. | + + +## Embeddings + +### Metric: `openai.embeddings.tokens` + +**Status**: [Experimental][DocumentStatus] + +This metric is required. + + +| Name | Instrument Type | Unit (UCUM) | Description | +| -------- | --------------- | ----------- | -------------- | +| `openai.embeddings.tokens` | Counter | `{token}` | Number of tokens used in prompt. | + + + +| Attribute | Type | Description | Examples | Requirement Level | +|---|---|---|---|---| +| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | +| `openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt` | Recommended | +| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | + +**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality. +Instrumentations SHOULD document the list of errors they report. + +The cardinality of `error.type` within one instrumentation library SHOULD be low, but +telemetry consumers that aggregate data from multiple instrumentation libraries and applications +should be prepared for `error.type` to have high cardinality at query time, when no +additional filters are applied. + +If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`. + +**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent +the server address behind any intermediaries (e.g. proxies) if it's available. + +`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used. + +| Value | Description | +|---|---| +| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. | + + +### Metric: `openai.embeddings.vector_size` + +**Status**: [Experimental][DocumentStatus] + +This metric is required. + + +| Name | Instrument Type | Unit (UCUM) | Description | +| -------- | --------------- | ----------- | -------------- | +| `openai.embeddings.vector_size` | Counter | `{element}` | The size of returned vector. | + + + +| Attribute | Type | Description | Examples | Requirement Level | +|---|---|---|---|---| +| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | +| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | + +**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality. +Instrumentations SHOULD document the list of errors they report. + +The cardinality of `error.type` within one instrumentation library SHOULD be low, but +telemetry consumers that aggregate data from multiple instrumentation libraries and applications +should be prepared for `error.type` to have high cardinality at query time, when no +additional filters are applied. + +If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`. + +**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent +the server address behind any intermediaries (e.g. proxies) if it's available. + +`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used. + +| Value | Description | +|---|---| +| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. | + + +### Metric: `openai.embeddings.duration` + +**Status**: [Experimental][DocumentStatus] + +This metric is required. + +This metric SHOULD be specified with +[`ExplicitBucketBoundaries`](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/metrics/api.md#instrument-advice) +of `[ 0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10 ]`. + + +| Name | Instrument Type | Unit (UCUM) | Description | +| -------- | --------------- | ----------- | -------------- | +| `openai.embeddings.duration` | Histogram | `s` | Duration of embeddings operation | + + + +| Attribute | Type | Description | Examples | Requirement Level | +|---|---|---|---|---| +| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | +| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | + +**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality. +Instrumentations SHOULD document the list of errors they report. + +The cardinality of `error.type` within one instrumentation library SHOULD be low, but +telemetry consumers that aggregate data from multiple instrumentation libraries and applications +should be prepared for `error.type` to have high cardinality at query time, when no +additional filters are applied. + +If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`. + +**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent +the server address behind any intermediaries (e.g. proxies) if it's available. + +`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used. + +| Value | Description | +|---|---| +| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. | + + +[DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/document-status.md + +## Image generation + +### Metric: `openai.image_generations.duration` + +**Status**: [Experimental][DocumentStatus] + +This metric is required. + +This metric SHOULD be specified with +[`ExplicitBucketBoundaries`](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/metrics/api.md#instrument-advice) +of `[ 0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10 ]`. + + +| Name | Instrument Type | Unit (UCUM) | Description | +| -------- | --------------- | ----------- | -------------- | +| `openai.image_generations.duration` | Histogram | `s` | Duration of image generations operation | + + + +| Attribute | Type | Description | Examples | Requirement Level | +|---|---|---|---|---| +| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | +| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | + +**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality. +Instrumentations SHOULD document the list of errors they report. + +The cardinality of `error.type` within one instrumentation library SHOULD be low, but +telemetry consumers that aggregate data from multiple instrumentation libraries and applications +should be prepared for `error.type` to have high cardinality at query time, when no +additional filters are applied. + +If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`. + +**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent +the server address behind any intermediaries (e.g. proxies) if it's available. + +`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used. + +| Value | Description | +|---|---| +| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. | + + +[DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/document-status.md \ No newline at end of file From 53c4fb5ecaf3d6740e4cc3eedaca2affc0ef4469 Mon Sep 17 00:00:00 2001 From: Drew Robbins Date: Wed, 15 Nov 2023 16:14:48 +0900 Subject: [PATCH 2/3] Updates to openai spans --- docs/ai/openai.md | 103 +++++++++++++++++++++++++++------------------- 1 file changed, 61 insertions(+), 42 deletions(-) diff --git a/docs/ai/openai.md b/docs/ai/openai.md index f96a728469..b95d4557a4 100644 --- a/docs/ai/openai.md +++ b/docs/ai/openai.md @@ -2,81 +2,100 @@ linkTitle: OpenAI ---> -# Semantic Conventions for OpenAI +# Semantic Conventions for OpenAI Spans **Status**: [Experimental][DocumentStatus] -The Semantic Conventions for [OpenAI](https://platform.openai.com/) extend the [LLM Semantic Conventions](llm-spans.md) -that describe common LLM request attributes in addition to the Semantic Conventions -described on this page. +This document outlines the Semantic Conventions specific to +[OpenAI](https://platform.openai.com/) spans, extending the general semantics +found in the [LLM Semantic Conventions](llm-spans.md). These conventions are +designed to standardize telemetry data for OpenAI interactions, particularly +focusing on the `/chat/completions` endpoint. By following to these guidelines, +developers can ensure consistent, meaningful, and easily interpretable telemetry +data across different applications and platforms. -## OpenAI LLM request attributes +## Chat Completions -### Chat completion attributes +The span name for OpenAI chat completions SHOULD be `openai_chat_completion` +to maintain consistency and clarity in telemetry data. -These are additional attributes when instrumenting OpenAI LLM requests with the `/chat` endpoint. +## Request Attributes + +These are the attributes when instrumenting OpenAI LLM requests with the +`/chat/completions` endpoint. | Attribute | Type | Description | Examples | Requirement Level | |---|---|---|---|---| -| `llm.openai.messages..role` | string | The assigned role for a given OpenAI request, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `system` | Required | -| `llm.openai.messages..message` | string | The message for a given OpenAI request, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `You are an AI system that tells jokes about OpenTelemetry.` | Required | -| `llm.openai.messages..name` | string | If present, the message for a given OpenAI request, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `You are an AI system that tells jokes about OpenTelemetry.` | Required | -| `llm.openai.messages..function_call.name` | string | If present, name of an OpenAI function for a given OpenAI request, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `get_weather_forecast` | Required | -| `llm.openai.messages..function_call.arguments` | string | If present, the arguments to call a function call with for an OpenAI function for a given OpenAI request, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `{"type": "object", "properties": {}}` | Required | -| `llm.openai.functions..name` | string | If present, name of an OpenAI function for a given OpenAI request, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `get_weather_forecast` | Required | -| `llm.openai.functions..parameters` | string | If present, JSON-encoded string of the parameter object of an OpenAI function for a given OpenAI request, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `{"type": "object", "properties": {}}` | Required | -| `llm.openai.functions..description` | string | If present, description of an OpenAI function for a given OpenAI request, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `Gets the weather forecast.` | Required | -| `llm.openai.n` | int | If present, the number of messages an OpenAI request responds with. | `2` | Recommended | +| `llm.model` | string | The name of OpenAI Model a request is being made to. | `gpt-4` | Required | +| `llm.max_tokens` | int | The maximum number of tokens the LLM generates for a request. | `100` | Recommended | +| `llm.temperature` | float | The temperature setting for the LLM request. | `0.0` | Recommended | +| `llm.top_p` | float | The top_p sampling setting for the LLM request. | `1.0` | Recommended | +| `llm.stream` | bool | Whether the LLM responds with a stream. | `false` | Recommended | +| `llm.stop_sequences` | array | Array of strings the LLM uses as a stop sequence. | `["stop1"]` | Recommended | | `llm.openai.presence_penalty` | float | If present, the `presence_penalty` used in an OpenAI request. Value is between -2.0 and 2.0. | `-0.5` | Recommended | | `llm.openai.frequency_penalty` | float | If present, the `frequency_penalty` used in an OpenAI request. Value is between -2.0 and 2.0. | `-0.5` | Recommended | | `llm.openai.logit_bias` | string | If present, the JSON-encoded string of a `logit_bias` used in an OpenAI request. | `{2435:-100, 640:-100}` | Recommended | -| `llm.openai.user` | string | If present, the `user` used in an OpenAI request. | `bob` | Recommended | +| `llm.openai.user` | string | If present, the `user` used in an OpenAI request. | `bob` | Opt-in | +| `llm.openai.response_format` | string | An object specifying the format that the model must output. Either `text` or `json_object` | `text` | Recommended | +| `llm.openai.seed` | integer | Seed used in request to improve determinism. | `1234` | Recommended | + +### Request Events + +Recording details about Messages and Tools for each request MAY be included as +Span Events. + +Instrumentations SHOULD require an explicit configuration for which events to +record. + +#### Message Events -### Completion attributes +Message event name SHOULD be `llm.openai.message`. -These are additional attributes when instrumenting OpenAI LLM requests with the `/completion` endpoint. +| `role` | string | The role of the messages author, can be one of `system`, `user`, `assistant`, or `tool` | `system` | Required | +| `content` | string | The content for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `Why did the developer stop using OpenTelemetry? Because they couldn't trace their steps!` | Required | +| `tool_call_id` | string | If role is `tool`, then this tool call that this message is responding to. | `call_BP08xxEhU60txNjnz3z9R4h9` | Conditionally Required: If `role` is `tool`. | -## OpenAI LLM response attributes +### Tools Events -These are additional attributes when instrumenting OpenAI LLM responses. +Tools event name SHOULD be `llm.openai.tool`. -### Chat completion attributes +| `type` | string | They type of the tool. Currently, only `function` is supported. | `function` | Required | +| `function.name` | string | The name of the function to be called. | `get_weather` | Required ! +| `function.description` | string | A description of what the function does, used by the model to choose when and how to call the function. | `` | Required | +| `function.parameters` | string | JSON-encoded string of the parameter object for the function. | `{"type": "object", "properties": {}}` | Required | -These are the attributes for a full chat completion (no streaming). +### OpenAI Chat completion response attributes + +Attributes for chat completion responses SHOULD follow these conventions: | Attribute | Type | Description | Examples | Requirement Level | |---|---|---|---|---| -| `llm.openai.choices..role` | string | The assigned role for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `system` | Required | -| `llm.openai.choices..content` | string | The content for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `Why did the developer stop using OpenTelemetry? Because they couldn't trace their steps!` | Required | -| `llm.openai.choices..function_call.name` | string | If exists, the name of a function call for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `get_weather_report` | Required | -| `llm.openai.choices..function_call.arguments` | string | If exists, the arguments to call a function call with for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `{"type": "object", "properties": {"some":"data"}}` | Required | | `llm.openai.finish_reason` | string | The reason the OpenAI model stopped generating tokens | `stop` | Recommended | | `llm.openai.id` | string | The unique identifier for the chat completion. | `chatcmpl-123` | Recommended | | `llm.openai.created` | int | The UNIX timestamp (in seconds) if when the completion was created. | `1677652288` | Recommended | -| `llm.openai.model` | string | The name of the model used for the completion. | `gpt-3.5-turbo` | Recommended | | `llm.openai.usage.prompt_tokens` | int | The number of tokens in the prompt passed as input. | `500` | Recommended | | `llm.openai.usage.completion_tokens` | int | The number of tokens generated in the completion. | `100` | Recommended | | `llm.openai.usage.total_tokens` | int | The total number of tokens used in both the prompt and the generated completion. | `600` | Recommended | +| `llm.openai.system_fingerprint` | string | This fingerprint represents the backend configuration that the model runs with. | asdf987123 | Recommended | -### Chat completion chunk attributes +### Choice Events -These are the attributes for a chunk of data in a streamed chat completion. +Recording details about Choices in each response MAY be included as +Span Events. -todo +Choice event name SHOULD be `llm.openai.choice`. - -| Attribute | Type | Description | Examples | Requirement Level | -|---|---|---|---|---| -| `llm.openai.choices..delta.role` | string | The assigned role for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `system` | Required | -| `llm.openai.choices..delta.content` | string | The content for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `Why did the developer` | Required | -| `llm.openai.choices..delta.function_call.name` | string | If exists, the name of a function call for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `get_weather_report` | Required | -| `llm.openai.choices..delta.function_call.arguments` | string | If exists, the arguments to call a function call with for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `{"type": "object",` | Required | -| `llm.openai.choices..finish_reason` | string | The reason the OpenAI model stopped generating tokens for this chunk. | `stop` | Recommended | -| `llm.openai.id` | string | The unique identifier for the chat completion. | `chatcmpl-123` | Recommended | -| `llm.openai.created` | int | The UNIX timestamp (in seconds) if when the completion was created. | `1677652288` | Recommended | -| `llm.openai.model` | string | The name of the model used for the completion. | `gpt-3.5-turbo` | Recommended | +If there is more than one `tool_call`, separate events SHOULD be used. +| `type` | string | Either `delta` or `message`. | `message` | Required | +| `finish_reason` | string | The reason the OpenAI model stopped generating tokens for this chunk. | `stop` | Recommended | +| `role` | string | The assigned role for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `system` | Required | +| `content` | string | The content for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `Why did the developer stop using OpenTelemetry? Because they couldn't trace their steps!` | Required | +| `tool_call.id` | string | If exists, the ID of the tool call. | `call_BP08xxEhU60txNjnz3z9R4h9` | Required | +| `tool_call.type` | string | Currently only `function` is supported. | `function` | Required | +| `tool_call.function.name` | string | If exists, the name of a function call for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `get_weather_report` | Required | +| `tool_call.function.arguments` | string | If exists, the arguments to call a function call with for a given OpenAI response, denoted by ``. The value for `` starts with 0, where 0 is the first message. | `{"type": "object", "properties": {"some":"data"}}` | Required | [DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/document-status.md From 5a7ef2db4c78ba80d297b20d6928122baf9ef057 Mon Sep 17 00:00:00 2001 From: Drew Robbins Date: Wed, 15 Nov 2023 16:27:37 +0900 Subject: [PATCH 3/3] Prefix llm to be consistent with spans --- docs/ai/openai-metrics.md | 64 +++++++++++++++++++-------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/docs/ai/openai-metrics.md b/docs/ai/openai-metrics.md index 18424875a2..9d52f3a12e 100644 --- a/docs/ai/openai-metrics.md +++ b/docs/ai/openai-metrics.md @@ -13,21 +13,21 @@ This document defines semantic conventions for OpenAI client metrics. - [Chat completions](#chat-completions) - * [Metric: `openai.chat_completions.tokens`](#metric-openaichat_completionstokens) - * [Metric: `openai.chat_completions.choices`](#metric-openaichat_completionschoices) - * [Metric: `openai.chat_completions.duration`](#metric-openaichat_completionsduration) + * [Metric: `llm.openai.chat_completions.tokens`](#metric-openaichat_completionstokens) + * [Metric: `llm.openai.chat_completions.choices`](#metric-openaichat_completionschoices) + * [Metric: `llm.openai.chat_completions.duration`](#metric-openaichat_completionsduration) - [Embeddings](#embeddings) - * [Metric: `openai.embeddings.tokens`](#metric-openaiembeddingstokens) - * [Metric: `openai.embeddings.vector_size`](#metric-openaiembeddingsvector_size) - * [Metric: `openai.embeddings.duration`](#metric-openaiembeddingsduration) + * [Metric: `llm.openai.embeddings.tokens`](#metric-openaiembeddingstokens) + * [Metric: `llm.openai.embeddings.vector_size`](#metric-openaiembeddingsvector_size) + * [Metric: `llm.openai.embeddings.duration`](#metric-openaiembeddingsduration) - [Image generation](#image-generation) - * [Metric: `openai.image_generations.duration`](#metric-openaiimage_generationsduration) + * [Metric: `llm.openai.image_generations.duration`](#metric-openaiimage_generationsduration) ## Chat completions -### Metric: `openai.chat_completions.tokens` +### Metric: `llm.openai.chat_completions.tokens` **Status**: [Experimental][DocumentStatus] @@ -36,15 +36,15 @@ This metric is required. | Name | Instrument Type | Unit (UCUM) | Description | | -------- | --------------- | ----------- | -------------- | -| `openai.chat_completions.tokens` | Counter | `{token}` | Number of tokens used in prompt and completions | +| `llm.openai.chat_completions.tokens` | Counter | `{token}` | Number of tokens used in prompt and completions | | Attribute | Type | Description | Examples | Requirement Level | |---|---|---|---|---| -| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `llm.openai.model` | string | Model name | `text-davinci-003` | Required | | `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | -| `openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt`; `completion` | Required | +| `llm.openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt`; `completion` | Required | | [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | **[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality. @@ -67,7 +67,7 @@ the server address behind any intermediaries (e.g. proxies) if it's available. | `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. | -### Metric: `openai.chat_completions.choices` +### Metric: `llm.openai.chat_completions.choices` **Status**: [Experimental][DocumentStatus] @@ -76,16 +76,16 @@ This metric is required. | Name | Instrument Type | Unit (UCUM) | Description | | -------- | --------------- | ----------- | -------------- | -| `openai.chat_completions.choices` | Counter | `{choice}` | Number of choices returned by chat completions call | +| `llm.openai.chat_completions.choices` | Counter | `{choice}` | Number of choices returned by chat completions call | | Attribute | Type | Description | Examples | Requirement Level | |---|---|---|---|---| -| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `llm.openai.model` | string | Model name | `text-davinci-003` | Required | | `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | -| `openai.choice.finish_reason` | string | Finish reason for single chat completion choice | `stop`; `length`; `content_filter` | Conditionally Required: if and only if it was returned. | -| `openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt`; `completion` | Required | +| `llm.openai.choice.finish_reason` | string | Finish reason for single chat completion choice | `stop`; `length`; `content_filter` | Conditionally Required: if and only if it was returned. | +| `llm.openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt`; `completion` | Required | | [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | **[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality. @@ -109,7 +109,7 @@ the server address behind any intermediaries (e.g. proxies) if it's available. -### Metric: `openai.chat_completions.duration` +### Metric: `llm.openai.chat_completions.duration` **Status**: [Experimental][DocumentStatus] @@ -122,13 +122,13 @@ of `[ 0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, | Name | Instrument Type | Unit (UCUM) | Description | | -------- | --------------- | ----------- | -------------- | -| `openai.chat_completions.duration` | Histogram | `s` | Duration of chat completion operation | +| `llm.openai.chat_completions.duration` | Histogram | `s` | Duration of chat completion operation | | Attribute | Type | Description | Examples | Requirement Level | |---|---|---|---|---| -| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `llm.openai.model` | string | Model name | `text-davinci-003` | Required | | `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | | [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | @@ -154,7 +154,7 @@ the server address behind any intermediaries (e.g. proxies) if it's available. ## Embeddings -### Metric: `openai.embeddings.tokens` +### Metric: `llm.openai.embeddings.tokens` **Status**: [Experimental][DocumentStatus] @@ -163,15 +163,15 @@ This metric is required. | Name | Instrument Type | Unit (UCUM) | Description | | -------- | --------------- | ----------- | -------------- | -| `openai.embeddings.tokens` | Counter | `{token}` | Number of tokens used in prompt. | +| `llm.openai.embeddings.tokens` | Counter | `{token}` | Number of tokens used in prompt. | | Attribute | Type | Description | Examples | Requirement Level | |---|---|---|---|---| -| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `llm.openai.model` | string | Model name | `text-davinci-003` | Required | | `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | -| `openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt` | Recommended | +| `llm.openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt` | Recommended | | [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | **[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality. @@ -194,7 +194,7 @@ the server address behind any intermediaries (e.g. proxies) if it's available. | `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. | -### Metric: `openai.embeddings.vector_size` +### Metric: `llm.openai.embeddings.vector_size` **Status**: [Experimental][DocumentStatus] @@ -203,13 +203,13 @@ This metric is required. | Name | Instrument Type | Unit (UCUM) | Description | | -------- | --------------- | ----------- | -------------- | -| `openai.embeddings.vector_size` | Counter | `{element}` | The size of returned vector. | +| `llm.openai.embeddings.vector_size` | Counter | `{element}` | The size of returned vector. | | Attribute | Type | Description | Examples | Requirement Level | |---|---|---|---|---| -| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `llm.openai.model` | string | Model name | `text-davinci-003` | Required | | `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | | [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | @@ -233,7 +233,7 @@ the server address behind any intermediaries (e.g. proxies) if it's available. | `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. | -### Metric: `openai.embeddings.duration` +### Metric: `llm.openai.embeddings.duration` **Status**: [Experimental][DocumentStatus] @@ -246,13 +246,13 @@ of `[ 0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, | Name | Instrument Type | Unit (UCUM) | Description | | -------- | --------------- | ----------- | -------------- | -| `openai.embeddings.duration` | Histogram | `s` | Duration of embeddings operation | +| `llm.openai.embeddings.duration` | Histogram | `s` | Duration of embeddings operation | | Attribute | Type | Description | Examples | Requirement Level | |---|---|---|---|---| -| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `llm.openai.model` | string | Model name | `text-davinci-003` | Required | | `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | | [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required | @@ -280,7 +280,7 @@ the server address behind any intermediaries (e.g. proxies) if it's available. ## Image generation -### Metric: `openai.image_generations.duration` +### Metric: `llm.openai.image_generations.duration` **Status**: [Experimental][DocumentStatus] @@ -293,13 +293,13 @@ of `[ 0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, | Name | Instrument Type | Unit (UCUM) | Description | | -------- | --------------- | ----------- | -------------- | -| `openai.image_generations.duration` | Histogram | `s` | Duration of image generations operation | +| `llm.openai.image_generations.duration` | Histogram | `s` | Duration of image generations operation | | Attribute | Type | Description | Examples | Requirement Level | |---|---|---|---|---| -| `openai.model` | string | Model name | `text-davinci-003` | Required | +| `llm.openai.model` | string | Model name | `text-davinci-003` | Required | | `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error | | [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required |