Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Azure OpenAI semconv prototype #513

Closed
wants to merge 4 commits into from
Closed

Azure OpenAI semconv prototype #513

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f741ea4
OpenAI early draft
lmolkova Sep 14, 2023
a2a9cec
add index to filter result
lmolkova Sep 14, 2023
1ad21c4
merge tokens counter
lmolkova Sep 14, 2023
17f4bc7
attribute clean up and choices metric
lmolkova Sep 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
326 changes: 326 additions & 0 deletions docs/openai/openai-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,326 @@
<!--- Hugo front matter used to generate the website version of this page:
linkTitle: MEtrics
--->

# Semantic Conventions for OpenAI Matrics

**Status**: [Experimental][DocumentStatus]

This document defines semantic conventions for OpenAI client metrics.

<!-- Re-generate TOC with `markdown-toc --no-first-h1 -i` -->

<!-- toc -->

- [Chat completions](#chat-completions)
* [Metric: `openai.chat_completions.tokens`](#metric-openaichat_completionstokens)
* [Metric: `openai.chat_completions.choices`](#metric-openaichat_completionschoices)
* [Metric: `openai.chat_completions.duration`](#metric-openaichat_completionsduration)
- [Embeddings](#embeddings)
* [Metric: `openai.embeddings.tokens`](#metric-openaiembeddingstokens)
* [Metric: `openai.embeddings.vector_size`](#metric-openaiembeddingsvector_size)
* [Metric: `openai.embeddings.duration`](#metric-openaiembeddingsduration)
- [Image generation](#image-generation)
* [Metric: `openai.image_generations.duration`](#metric-openaiimage_generationsduration)

<!-- tocstop -->

## Chat completions

### Metric: `openai.chat_completions.tokens`

**Status**: [Experimental][DocumentStatus]

This metric is required.

<!-- semconv metric.openai.chat_completions.tokens(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `openai.chat_completions.tokens` | Counter | `{token}` | Number of tokens used in prompt and completions |
<!-- endsemconv -->

<!-- semconv metric.openai.chat_completions.tokens(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `openai.model` | string | Model name | `text-davinci-003` | Required |
| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error |
| `openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt`; `completion` | Required |
| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required |

**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality.
Instrumentations SHOULD document the list of errors they report.

The cardinality of `error.type` within one instrumentation library SHOULD be low, but
telemetry consumers that aggregate data from multiple instrumentation libraries and applications
should be prepared for `error.type` to have high cardinality at query time, when no
additional filters are applied.

If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`.

**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent
the server address behind any intermediaries (e.g. proxies) if it's available.

`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

| Value | Description |
|---|---|
| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. |
<!-- endsemconv -->

### Metric: `openai.chat_completions.choices`

**Status**: [Experimental][DocumentStatus]

This metric is required.

<!-- semconv metric.openai.chat_completions.choices(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `openai.chat_completions.choices` | Counter | `{choice}` | Number of choices returned by chat completions call |
<!-- endsemconv -->

<!-- semconv metric.openai.chat_completions.choices(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `openai.model` | string | Model name | `text-davinci-003` | Required |
| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error |
| `openai.choice.finish_reason` | string | Finish reason for single chat completion choice | `stop`; `length`; `content_filter` | Conditionally Required: if and only if it was returned. |
| `openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt`; `completion` | Required |
| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required |

**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality.
Instrumentations SHOULD document the list of errors they report.

The cardinality of `error.type` within one instrumentation library SHOULD be low, but
telemetry consumers that aggregate data from multiple instrumentation libraries and applications
should be prepared for `error.type` to have high cardinality at query time, when no
additional filters are applied.

If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`.

**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent
the server address behind any intermediaries (e.g. proxies) if it's available.

`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

| Value | Description |
|---|---|
| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. |
<!-- endsemconv -->


### Metric: `openai.chat_completions.duration`

**Status**: [Experimental][DocumentStatus]

This metric is required.

This metric SHOULD be specified with
[`ExplicitBucketBoundaries`](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/metrics/api.md#instrument-advice)
of `[ 0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10 ]`.

<!-- semconv metric.openai.chat_completions.duration(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `openai.chat_completions.duration` | Histogram | `s` | Duration of chat completion operation |
<!-- endsemconv -->

<!-- semconv metric.openai.chat_completions.duration(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `openai.model` | string | Model name | `text-davinci-003` | Required |
| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error |
| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required |

**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality.
Instrumentations SHOULD document the list of errors they report.

The cardinality of `error.type` within one instrumentation library SHOULD be low, but
telemetry consumers that aggregate data from multiple instrumentation libraries and applications
should be prepared for `error.type` to have high cardinality at query time, when no
additional filters are applied.

If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`.

**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent
the server address behind any intermediaries (e.g. proxies) if it's available.

`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

| Value | Description |
|---|---|
| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. |
<!-- endsemconv -->

## Embeddings

### Metric: `openai.embeddings.tokens`

**Status**: [Experimental][DocumentStatus]

This metric is required.

<!-- semconv metric.openai.embeddings.tokens(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `openai.embeddings.tokens` | Counter | `{token}` | Number of tokens used in prompt. |
<!-- endsemconv -->

<!-- semconv metric.openai.embeddings.tokens(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `openai.model` | string | Model name | `text-davinci-003` | Required |
| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error |
| `openai.usage.type` | string | Describes if tokens were used in prompt or completion | `prompt` | Recommended |
| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required |

**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality.
Instrumentations SHOULD document the list of errors they report.

The cardinality of `error.type` within one instrumentation library SHOULD be low, but
telemetry consumers that aggregate data from multiple instrumentation libraries and applications
should be prepared for `error.type` to have high cardinality at query time, when no
additional filters are applied.

If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`.

**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent
the server address behind any intermediaries (e.g. proxies) if it's available.

`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

| Value | Description |
|---|---|
| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. |
<!-- endsemconv -->

### Metric: `openai.embeddings.vector_size`

**Status**: [Experimental][DocumentStatus]

This metric is required.

<!-- semconv metric.openai.embeddings.vector_size(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `openai.embeddings.vector_size` | Counter | `{element}` | The size of returned vector. |
<!-- endsemconv -->

<!-- semconv metric.openai.embeddings.vector_size(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `openai.model` | string | Model name | `text-davinci-003` | Required |
| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error |
| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required |

**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality.
Instrumentations SHOULD document the list of errors they report.

The cardinality of `error.type` within one instrumentation library SHOULD be low, but
telemetry consumers that aggregate data from multiple instrumentation libraries and applications
should be prepared for `error.type` to have high cardinality at query time, when no
additional filters are applied.

If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`.

**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent
the server address behind any intermediaries (e.g. proxies) if it's available.

`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

| Value | Description |
|---|---|
| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. |
<!-- endsemconv -->

### Metric: `openai.embeddings.duration`

**Status**: [Experimental][DocumentStatus]

This metric is required.

This metric SHOULD be specified with
[`ExplicitBucketBoundaries`](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/metrics/api.md#instrument-advice)
of `[ 0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10 ]`.

<!-- semconv metric.openai.embeddings.duration(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `openai.embeddings.duration` | Histogram | `s` | Duration of embeddings operation |
<!-- endsemconv -->

<!-- semconv metric.openai.embeddings.duration(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `openai.model` | string | Model name | `text-davinci-003` | Required |
| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error |
| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required |

**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality.
Instrumentations SHOULD document the list of errors they report.

The cardinality of `error.type` within one instrumentation library SHOULD be low, but
telemetry consumers that aggregate data from multiple instrumentation libraries and applications
should be prepared for `error.type` to have high cardinality at query time, when no
additional filters are applied.

If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`.

**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent
the server address behind any intermediaries (e.g. proxies) if it's available.

`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

| Value | Description |
|---|---|
| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. |
<!-- endsemconv -->

[DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/document-status.md

## Image generation

### Metric: `openai.image_generations.duration`

**Status**: [Experimental][DocumentStatus]

This metric is required.

This metric SHOULD be specified with
[`ExplicitBucketBoundaries`](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/metrics/api.md#instrument-advice)
of `[ 0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10 ]`.

<!-- semconv metric.openai.image_generations.duration(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `openai.image_generations.duration` | Histogram | `s` | Duration of image generations operation |
<!-- endsemconv -->

<!-- semconv metric.openai.image_generations.duration(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `openai.model` | string | Model name | `text-davinci-003` | Required |
| `error.type` | string | Describes a class of error the operation ended with. [1] | `timeout`; `java.net.UnknownHostException`; `server_certificate_invalid`; `500` | Conditionally Required: if and only if operation has ended with an error |
| [`server.address`](../general/attributes.md) | string | Server address - domain name if available without reverse DNS lookup, otherwise IP address or Unix domain socket name. [2] | `example.com` | Required |

**[1]:** The `error.type` SHOULD be predictable and SHOULD have low cardinality.
Instrumentations SHOULD document the list of errors they report.

The cardinality of `error.type` within one instrumentation library SHOULD be low, but
telemetry consumers that aggregate data from multiple instrumentation libraries and applications
should be prepared for `error.type` to have high cardinality at query time, when no
additional filters are applied.

If the operation has completed successfully, instrumentations SHOULD NOT set `error.type`.

**[2]:** When observed from the client side, and when communicating through an intermediary, `server.address` SHOULD represent
the server address behind any intermediaries (e.g. proxies) if it's available.

`error.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

| Value | Description |
|---|---|
| `_OTHER` | A fallback error value to be used when the instrumentation does not define a custom value for it. |
<!-- endsemconv -->

[DocumentStatus]: https://github.com/open-telemetry/opentelemetry-specification/tree/v1.22.0/specification/document-status.md
Loading
Loading