From f6910f6e487330fce635408d90a99237ed1ba3b2 Mon Sep 17 00:00:00 2001 From: Mayo Date: Tue, 26 Jul 2022 10:30:44 +0000 Subject: [PATCH 1/7] add tracing --- src/gateway/observability/tracing/api.md | 206 ++++++++++++++++++ src/gateway/observability/tracing/index.md | 42 ++++ .../tracing/write-custom-trace-exporter.md | 39 ++++ src/gateway/plugin-development/telemetry.md | 21 ++ 4 files changed, 308 insertions(+) create mode 100644 src/gateway/observability/tracing/api.md create mode 100644 src/gateway/observability/tracing/index.md create mode 100644 src/gateway/observability/tracing/write-custom-trace-exporter.md create mode 100644 src/gateway/plugin-development/telemetry.md diff --git a/src/gateway/observability/tracing/api.md b/src/gateway/observability/tracing/api.md new file mode 100644 index 000000000000..af13adc73ac8 --- /dev/null +++ b/src/gateway/observability/tracing/api.md @@ -0,0 +1,206 @@ +--- +title: Tracing API +--- + +## Before you start + +Kong bundled the tracing framework since version 3.0.0, the tracing API is a part of the Kong core, +and the API is in the `kong.tracing` namespace. + +To make the tracing API standardized and available to all plugins, +we follow the [OpenTelemetry API specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md). +This specification defines the API that should be used to instrument your module. +If you already have the experience of using the OpenTelemetry API, it should be much easier to start. + +With the tracing API, you can instrument your module with the following: +- [Span](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#span) +- [Attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md) + +## Create a tracer + +Kong internally uses a global tracer instance to instrument the core modules and plugins. + +By default, the tracer is a [NoopTracer](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#get-a-tracer) which does nothing to avoid any overhead. The tracer is first initialized when `opentelemetry_tracing` configuration enabled. + +You can either create a new tracer manually, or use the global tracer instance. + +```lua +local tracer + +-- Create a new tracer +tracer = kong.tracing.new("custom-tracer") + +-- Use the global tracer +tracer = kong.tracing +``` + +### Sampling traces + +The tracer can be configured to sample traces. + +```lua +local tracer = kong.tracing.new("custom-tracer", { + -- Set the sampling rate to 0.1 + sampling_rate = 0.1, +}) +``` + +By default, the sampling rate is 1.0, meaning that all traces are sampled. + +## Create a span + +A Span represents a single operation within a trace. Spans can be nested to form a trace tree. Each trace contains a root span, which typically describes the entire operation and, optionally, one or more sub-spans for its sub-operations. + +```lua +local tracer = kong.tracing + +local span = tracer:start_span("my-span") +``` + +The span properties can be set by passing a table to the `start_span` method. + +```lua +local span = tracer:start_span("my-span", { + start_time_ns = ngx.now() * 1e9, -- override the start time + span_kind = 2, -- SPAN_KIND + -- UNSPECIFIED: 0 + -- INTERNAL: 1 + -- SERVER: 2 + -- CLIENT: 3 + -- PRODUCER: 4 + -- CONSUMER: 5 + should_sample = true, -- by setting it to `true` to ignore the sampling decision +}) +``` + +Please make sure to ends the span when you are done. + +```lua +span:finish() -- ends the span +``` + +**Note:** The span table will be cleared and put into the table pool after the span is finished, +do not use it after the span is finished. + +## Get/Set the active span + +The active span is the span that is currently being executed. + +To avoid overheads, the active span is manually set by calling the `set_active_span` method. +However, When you finish a span, the active span becomes the parent of the finished span. + + +To set/get the active span, you can use the following: + +```lua +local tracer = kong.tracing +local span = tracer:start_span("my-span") +tracer.set_active_span(span) + +local active_span = tracer.active_span() -- returns the active span +``` + +### Scope + +The tracers are scoped to a specific context by a namespace key. + +To get the active span for a specific namespace, you can use the following: + +```lua +-- get global tracer's active span, and set it as the parent of new created span +local global_tracer = kong.tracing +local tracer = kong.tracing.new("custom-tracer") + +local root_span = global_tracer.active_span() +local span = tracer.start_span("my-span", { + parent = root_span +}) +``` + +## Set the span attributes + +The attributes of a span are a map of key-value pairs +and can be set by passing a table to the `set_attributes` method. + +```lua +local span = tracer:start_span("my-span") +``` + +The OpenTelemetry specification defines the general semantic attributes, it can be used to describe the span. +It could also be meaningful to visualize the span in a UI. + +```lua +span:set_attribute("key", "value") +``` + +The following semantic conventions for spans are defined: + +* [General](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md): General semantic attributes that may be used in describing different kinds of operations. +* [HTTP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md): For HTTP client and server spans. +* [Database](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/database.md): For SQL and NoSQL client call spans. +* [RPC/RMI](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/rpc.md): For remote procedure call (e.g., gRPC) spans. +* [Messaging](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/messaging.md): For messaging systems (queues, publish/subscribe, etc.) spans. +* [FaaS](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/faas.md): For [Function as a Service](https://en.wikipedia.org/wiki/Function_as_a_service) (e.g., AWS Lambda) spans. +* [Exceptions](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/exceptions.md): For recording exceptions associated with a span. +* [Compatibility](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/compatibility.md): For spans generated by compatibility components, e.g. OpenTracing Shim layer. + +## Set the span events + +The events of a span are time-series events that can be set by passing a table to the `add_event` method. + +```lua +local span = kong.tracing:start_span("my-span") +span:add_event("my-event", { + -- attributes + ["key"] = "value", +}) +``` + +### Record error message + +The event could also be used to record error messages. + +```lua +local span = kong.tracing:start_span("my-span") +span:record_error("my-error-message") + +-- or (same as above) +span:add_event("exception", { + ["exception.message"] = "my-error-message", +}) +``` + +## Set the span status + +The status of a span is a status code and can be set by passing a table to the `set_status` method. + +```lua +local span = kong.tracing:start_span("my-span") +-- Status codes: +-- - `0` unset +-- - `1` ok +-- - `2` error +``` + +## Release the span (optional) + +The spans are stored in a pool, and can be released by calling the `release` method. + +```lua +local span = kong.tracing:start_span("my-span") +span:release() +``` + +By default, the span will be released after the Nginx request ends. + +## Visualize the trace + +Because of the compatibility with OpenTelemetry, the traces can be natively visualized through any OpenTelemetry UI. + +Please refer to the [OpenTelemetry plugin](/hub/kong-inc/opentelemetry) to see how to visualize the traces. + +## References + +- [Tracing PDK](/pdk/kong.tracing) +- [Measuring your plugin](/plugin-development/telemetry) +- [OpenTelemetry plugin](/hub/kong-inc/opentelemetry) diff --git a/src/gateway/observability/tracing/index.md b/src/gateway/observability/tracing/index.md new file mode 100644 index 000000000000..c2c24dd2b99d --- /dev/null +++ b/src/gateway/observability/tracing/index.md @@ -0,0 +1,42 @@ +--- +title: Tracing +--- + +## Core instrumentations + +| OT Category | Kong Components | Name | Span data | Notes | +| ------------------------- | --------------------------- | ------------------------------------ | ----------------------------------- | ------------------------------------------------------- | +| Database | Storage | DB query | SQL | database spec | +| Redis | Redis call | database spec | | | +| Redis cluster | Redis call | extra support for resty.rediscluster | | | +| LMDB | Transactions | database spec | | | +| HTTP | Core | HTTP Server Request / Consumer | Nginx server request | http spec | +| Balancer Retry / Upstream | Nginx balancer retries data | http spec | | | +| Core / Plugins | HTTP Client Request | HTTP client request | http spec, resty.http, e.g. plugins | | +| RPC | Core | gRPC | gRPC call | rpc spec | +| Messaging | Stream | Pub | OpenResty pub/sub | messaging spec | +| Sub | OpenResty pub/sub | | | | +| Kafka | | messaging spec | | | +| FAAS | Plugins | AWS Lambda | remote call | faas spec | +| Azure functions | | | | | +| Exceptions | Errors | OpenResty errors | | Capture runtime errors (logs can be linked by trace_id) | +| Networking | Core | DNS query | host, success | general spec | +| Plugins | LDAP | LDAP operations | ldap spec | | +| LDAP Enhanced | LDAP operations | ldap spec | | | +| OpenID Connect | | | | | +| GraphQL | GraphQL query | | | | +| Vault | Vault query | | | | +| Kong Internal | Core | Router | Execution time | | +| Plugin execution | Plugin name, execution time | start/end time only | | | + +## Propagation + +The tracing API support to propagate the following headers: +- `w3c` - [W3C trace context](https://www.w3.org/TR/trace-context/) +- `b3`, `b3-single` - [Zipkin headers](https://github.com/openzipkin/b3-propagation) +- `jaeger` - [Jaeger headers](https://www.jaegertracing.io/docs/client-libraries/#propagation-format) +- `ot` - [OpenTracing headers](https://github.com/opentracing/specification/blob/master/rfc/trace_identifiers.md) +- `datadog` - [Datadog headers](https://docs.datadoghq.com/tracing/agent/propagation/) + +The tracing API will detect the propagation format from the headers, and will use the appropriate format to propagate the span context. +If no appropriate format is found, then will fallback to the default format, which can be specified. diff --git a/src/gateway/observability/tracing/write-custom-trace-exporter.md b/src/gateway/observability/tracing/write-custom-trace-exporter.md new file mode 100644 index 000000000000..a26158ebf674 --- /dev/null +++ b/src/gateway/observability/tracing/write-custom-trace-exporter.md @@ -0,0 +1,39 @@ +--- +title: Write a custom trace exporter +--- + +Kong bundled OpenTelemetry plugin in core with a implementation of [OTLP/HTTP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md#otlphttp), but you can still write your own exporter at scale. + +The following sections shows how to write a custom trace exporter. + +## Before you start + +Please make sure you have read the following sections: + +- [Documentation of tracing framework](/gateway/{{page.kong_version}}/observability/tracing-framework) +- [Plugin Development](/gateway/{{page.kong_version}}/plugin-development) + +## Gathering the spans + +The spans are stored in the tracer's buffer. +The buffer is a queue of spans that are awaiting to be sent to the backend. + +You can access the buffer and process the span using the `process_span` function. + +```lua +-- Use the global tracer +local tracer = kong.tracing + +-- Process the span +local span_processor = function(span) + -- clone the span so it can be processed after the original one is cleared + local span_dup = table.clone(span) + -- you can transform the span, add tags, etc. to other specific data structures +end +``` + +The `span_processor` function should be called in the `log` phase of the plugin. + +## Full example + +Refer to [Github](https://github.com/Kong/kong/tree/master/spec/fixtures/custom_plugins/kong/plugins/tcp-trace-exporter) to see the example of a custom trace exporter. diff --git a/src/gateway/plugin-development/telemetry.md b/src/gateway/plugin-development/telemetry.md new file mode 100644 index 000000000000..927e5d952bf7 --- /dev/null +++ b/src/gateway/plugin-development/telemetry.md @@ -0,0 +1,21 @@ +--- +title: Plugin Development - Measuring your plugin +book: plugin_dev +chapter: 11 +--- + +This guide will provide you with step-by-step instructions +that will make it possible to measure your plugin at the runtime. + +These steps should be applied to each node in your Kong cluster, to ensure the +custom plugin(s) are available on each one of them. + +## Tracing + +Kong bundled the tracing framework since version 3.0.0, the tracing API is a part of the Kong core, +and the API is in the `kong.tracing` namespace. + +### Reference + +- [Tracing PDK](/pdk/kong.tracing) +- [Tracing Framework](/observability/tracing-framework) From 477b06b8b16ef061ff3781bee32baa4756078853 Mon Sep 17 00:00:00 2001 From: Mayo Date: Tue, 26 Jul 2022 11:07:39 +0000 Subject: [PATCH 2/7] fix links --- src/gateway/observability/tracing/index.md | 45 ++++++++++------------ 1 file changed, 21 insertions(+), 24 deletions(-) diff --git a/src/gateway/observability/tracing/index.md b/src/gateway/observability/tracing/index.md index c2c24dd2b99d..faf4c7ddd427 100644 --- a/src/gateway/observability/tracing/index.md +++ b/src/gateway/observability/tracing/index.md @@ -2,32 +2,27 @@ title: Tracing --- +In this section, we will describe the tracing capabilities of Kong. + ## Core instrumentations -| OT Category | Kong Components | Name | Span data | Notes | -| ------------------------- | --------------------------- | ------------------------------------ | ----------------------------------- | ------------------------------------------------------- | -| Database | Storage | DB query | SQL | database spec | -| Redis | Redis call | database spec | | | -| Redis cluster | Redis call | extra support for resty.rediscluster | | | -| LMDB | Transactions | database spec | | | -| HTTP | Core | HTTP Server Request / Consumer | Nginx server request | http spec | -| Balancer Retry / Upstream | Nginx balancer retries data | http spec | | | -| Core / Plugins | HTTP Client Request | HTTP client request | http spec, resty.http, e.g. plugins | | -| RPC | Core | gRPC | gRPC call | rpc spec | -| Messaging | Stream | Pub | OpenResty pub/sub | messaging spec | -| Sub | OpenResty pub/sub | | | | -| Kafka | | messaging spec | | | -| FAAS | Plugins | AWS Lambda | remote call | faas spec | -| Azure functions | | | | | -| Exceptions | Errors | OpenResty errors | | Capture runtime errors (logs can be linked by trace_id) | -| Networking | Core | DNS query | host, success | general spec | -| Plugins | LDAP | LDAP operations | ldap spec | | -| LDAP Enhanced | LDAP operations | ldap spec | | | -| OpenID Connect | | | | | -| GraphQL | GraphQL query | | | | -| Vault | Vault query | | | | -| Kong Internal | Core | Router | Execution time | | -| Plugin execution | Plugin name, execution time | start/end time only | | | +**Note** +Only works for the plugins that are built on top of Kong's tracing API. +e.g. OpenTelemetry plugin. + +Kong provides a set of core instrumentations for tracing, these can be configured in the `opentelemetry_tracing` configuration. + +- `off`: do not enable instrumentations. +- `request`: only enable request-level instrumentations. +- `all`: enable all the following instrumentations. +- `db_query`: trace database query, including Postgres and Cassandra. +- `dns_query`: trace DNS query. +- `router`: trace router execution, including router rebuilding. +- `http_client`: trace OpenResty HTTP client requests. +- `balancer`: trace balancer retries. +- `plugin_rewrite`: trace plugins iterator execution with rewrite phase. +- `plugin_access`: trace plugins iterator execution with access phase. +- `plugin_header_filter`: trace plugins iterator execution with header_filter phase. ## Propagation @@ -40,3 +35,5 @@ The tracing API support to propagate the following headers: The tracing API will detect the propagation format from the headers, and will use the appropriate format to propagate the span context. If no appropriate format is found, then will fallback to the default format, which can be specified. + +The propagation api works for both the OpenTelemetry plugin and the Zipkin plugin. From aca5ccfafaba23a76ccf33f00259fafb0db49e85 Mon Sep 17 00:00:00 2001 From: Mayo Date: Wed, 27 Jul 2022 06:27:38 +0000 Subject: [PATCH 3/7] fix broken links --- src/gateway/observability/tracing/api.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/gateway/observability/tracing/api.md b/src/gateway/observability/tracing/api.md index af13adc73ac8..257c238a29e3 100644 --- a/src/gateway/observability/tracing/api.md +++ b/src/gateway/observability/tracing/api.md @@ -201,6 +201,6 @@ Please refer to the [OpenTelemetry plugin](/hub/kong-inc/opentelemetry) to see h ## References -- [Tracing PDK](/pdk/kong.tracing) -- [Measuring your plugin](/plugin-development/telemetry) +- [Tracing PDK](/gateway/{{page.kong_version}}/pdk/kong.tracing) +- [Measuring your plugin](/gateway/{{page.kong_version}}/plugin-development/telemetry) - [OpenTelemetry plugin](/hub/kong-inc/opentelemetry) From d4c7ca146d076e80c58dfd4a6c63e5ebdfee1792 Mon Sep 17 00:00:00 2001 From: angel Date: Wed, 3 Aug 2022 12:01:14 -0400 Subject: [PATCH 4/7] placeholder --- app/_data/docs_nav_gateway_3.0.x.yml | 10 +++++ .../observability/tracing/api.md | 39 ++++++++++--------- .../observability/tracing/index.md | 0 .../tracing/write-custom-trace-exporter.md | 0 4 files changed, 30 insertions(+), 19 deletions(-) rename src/gateway/{ => kong-plugins}/observability/tracing/api.md (78%) rename src/gateway/{ => kong-plugins}/observability/tracing/index.md (100%) rename src/gateway/{ => kong-plugins}/observability/tracing/write-custom-trace-exporter.md (100%) diff --git a/app/_data/docs_nav_gateway_3.0.x.yml b/app/_data/docs_nav_gateway_3.0.x.yml index 963c3d3d81d0..0e012bbcb8da 100644 --- a/app/_data/docs_nav_gateway_3.0.x.yml +++ b/app/_data/docs_nav_gateway_3.0.x.yml @@ -443,6 +443,16 @@ items: url: /kong-plugins/rate-limiting/algorithms/rate-limiting - text: Configuring Fixed bucket url: /kong-plugins/rate-limiting/algorithms/fixed-bucket + - text: Observability + items: + - text: Tracing + items: + - text: Overview + url: /kong-plugins/observability/tracing/ + - text: How to Write a Custom Trace Exporter + url: /kong-plugins/observability/tracing/write-custom-trace-exporter + - text: Tracing API reference + url: /kong-plugins/observability/tracing/api - text: GraphQL url: /kong-plugins/graphql-quickstart - text: gRPC diff --git a/src/gateway/observability/tracing/api.md b/src/gateway/kong-plugins/observability/tracing/api.md similarity index 78% rename from src/gateway/observability/tracing/api.md rename to src/gateway/kong-plugins/observability/tracing/api.md index 257c238a29e3..ea36ca277019 100644 --- a/src/gateway/observability/tracing/api.md +++ b/src/gateway/kong-plugins/observability/tracing/api.md @@ -1,28 +1,28 @@ --- -title: Tracing API +title: How to Write a Custom Trace Exporter +content-type: how-to --- ## Before you start -Kong bundled the tracing framework since version 3.0.0, the tracing API is a part of the Kong core, -and the API is in the `kong.tracing` namespace. +In Gateway version 3.0.0, the tracing API became part of the Kong core application. +The API is in the `kong.tracing` namespace. -To make the tracing API standardized and available to all plugins, -we follow the [OpenTelemetry API specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md). -This specification defines the API that should be used to instrument your module. -If you already have the experience of using the OpenTelemetry API, it should be much easier to start. +The tracing API follows the [OpenTelemetry API specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md). +This specification defines how to use the API as an instrument to your module. +If you are familiar with the OpenTelemetry API, the tracing API will be familiar. -With the tracing API, you can instrument your module with the following: +With the tracing API, you can set the instrumentation of your module with the following operations: - [Span](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#span) - [Attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md) ## Create a tracer -Kong internally uses a global tracer instance to instrument the core modules and plugins. +Kong uses a global tracer internally to instrument the core modules and plugins. -By default, the tracer is a [NoopTracer](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#get-a-tracer) which does nothing to avoid any overhead. The tracer is first initialized when `opentelemetry_tracing` configuration enabled. +By default, the tracer is a [NoopTracer](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#get-a-tracer). The tracer is first initialized when `opentelemetry_tracing` configuration is enabled. -You can either create a new tracer manually, or use the global tracer instance. +You can create a new tracer manually, or use the global tracer instance: ```lua local tracer @@ -36,7 +36,7 @@ tracer = kong.tracing ### Sampling traces -The tracer can be configured to sample traces. +The sampling rate of a tracer can be configured: ```lua local tracer = kong.tracing.new("custom-tracer", { @@ -45,11 +45,11 @@ local tracer = kong.tracing.new("custom-tracer", { }) ``` -By default, the sampling rate is 1.0, meaning that all traces are sampled. +The default sampling rate is `1.0`, which samples all traces. ## Create a span -A Span represents a single operation within a trace. Spans can be nested to form a trace tree. Each trace contains a root span, which typically describes the entire operation and, optionally, one or more sub-spans for its sub-operations. +A span represents a single operation within a trace. Spans can be nested to form trace trees. Each trace contains a root span, which typically describes the entire operation and, optionally, one or more sub-spans for its sub-operations. ```lua local tracer = kong.tracing @@ -73,24 +73,25 @@ local span = tracer:start_span("my-span", { }) ``` -Please make sure to ends the span when you are done. +Make sure to ends the span when you are done: ```lua span:finish() -- ends the span ``` -**Note:** The span table will be cleared and put into the table pool after the span is finished, +{:.Note} +>**Note:** The span table will be cleared and put into the table pool after the span is finished, do not use it after the span is finished. -## Get/Set the active span +## Get or set the active span The active span is the span that is currently being executed. To avoid overheads, the active span is manually set by calling the `set_active_span` method. -However, When you finish a span, the active span becomes the parent of the finished span. +When you finish a span, the active span becomes the parent of the finished span. -To set/get the active span, you can use the following: +To setor get the active span, you can use the following example code: ```lua local tracer = kong.tracing diff --git a/src/gateway/observability/tracing/index.md b/src/gateway/kong-plugins/observability/tracing/index.md similarity index 100% rename from src/gateway/observability/tracing/index.md rename to src/gateway/kong-plugins/observability/tracing/index.md diff --git a/src/gateway/observability/tracing/write-custom-trace-exporter.md b/src/gateway/kong-plugins/observability/tracing/write-custom-trace-exporter.md similarity index 100% rename from src/gateway/observability/tracing/write-custom-trace-exporter.md rename to src/gateway/kong-plugins/observability/tracing/write-custom-trace-exporter.md From 1698f99ab0821c8efcf7e2d0e3a20f8382321df3 Mon Sep 17 00:00:00 2001 From: angel Date: Wed, 3 Aug 2022 15:14:35 -0400 Subject: [PATCH 5/7] copy edit --- app/_data/docs_nav_gateway_3.0.x.yml | 23 ++++++++++--------- .../kong-plugins/observability/tracing/api.md | 8 +++---- .../observability/tracing/index.md | 7 +++--- .../tracing/write-custom-trace-exporter.md | 22 ++++++++++-------- src/gateway/plugin-development/telemetry.md | 2 +- 5 files changed, 33 insertions(+), 29 deletions(-) diff --git a/app/_data/docs_nav_gateway_3.0.x.yml b/app/_data/docs_nav_gateway_3.0.x.yml index 0e012bbcb8da..27be6feede89 100644 --- a/app/_data/docs_nav_gateway_3.0.x.yml +++ b/app/_data/docs_nav_gateway_3.0.x.yml @@ -360,7 +360,17 @@ items: url: /plugin-development/tests - text: (un)Installing your plugin url: /plugin-development/distribution - - text: Plugin Developmen Kit + - text: Observability + items: + - text: Tracing + items: + - text: Overview + url: /kong-plugins/observability/tracing/ + - text: How to Write a Custom Trace Exporter + url: /kong-plugins/observability/tracing/write-custom-trace-exporter + - text: Tracing API reference + url: /kong-plugins/observability/tracing/api + - text: Plugin Development Kit items: - text: Overview url: /plugin-development/pdk/ @@ -443,16 +453,7 @@ items: url: /kong-plugins/rate-limiting/algorithms/rate-limiting - text: Configuring Fixed bucket url: /kong-plugins/rate-limiting/algorithms/fixed-bucket - - text: Observability - items: - - text: Tracing - items: - - text: Overview - url: /kong-plugins/observability/tracing/ - - text: How to Write a Custom Trace Exporter - url: /kong-plugins/observability/tracing/write-custom-trace-exporter - - text: Tracing API reference - url: /kong-plugins/observability/tracing/api + - text: GraphQL url: /kong-plugins/graphql-quickstart - text: gRPC diff --git a/src/gateway/kong-plugins/observability/tracing/api.md b/src/gateway/kong-plugins/observability/tracing/api.md index ea36ca277019..f80780c8f26d 100644 --- a/src/gateway/kong-plugins/observability/tracing/api.md +++ b/src/gateway/kong-plugins/observability/tracing/api.md @@ -1,6 +1,6 @@ --- -title: How to Write a Custom Trace Exporter -content-type: how-to +title: Tracing API Referenece +content-type: Reference --- ## Before you start @@ -91,7 +91,7 @@ To avoid overheads, the active span is manually set by calling the `set_active_s When you finish a span, the active span becomes the parent of the finished span. -To setor get the active span, you can use the following example code: +To set or get the active span, you can use the following example code: ```lua local tracer = kong.tracing @@ -203,5 +203,5 @@ Please refer to the [OpenTelemetry plugin](/hub/kong-inc/opentelemetry) to see h ## References - [Tracing PDK](/gateway/{{page.kong_version}}/pdk/kong.tracing) -- [Measuring your plugin](/gateway/{{page.kong_version}}/plugin-development/telemetry) +- [Measuring your plugin](/gateway/plugin-development/telemetry) - [OpenTelemetry plugin](/hub/kong-inc/opentelemetry) diff --git a/src/gateway/kong-plugins/observability/tracing/index.md b/src/gateway/kong-plugins/observability/tracing/index.md index faf4c7ddd427..632de425cadf 100644 --- a/src/gateway/kong-plugins/observability/tracing/index.md +++ b/src/gateway/kong-plugins/observability/tracing/index.md @@ -1,5 +1,6 @@ --- -title: Tracing +title: Tracing Reference +content-type: reference --- In this section, we will describe the tracing capabilities of Kong. @@ -15,14 +16,14 @@ Kong provides a set of core instrumentations for tracing, these can be configure - `off`: do not enable instrumentations. - `request`: only enable request-level instrumentations. - `all`: enable all the following instrumentations. -- `db_query`: trace database query, including Postgres and Cassandra. +- `db_query`: trace database query, including PostgresSQL and Cassandra. - `dns_query`: trace DNS query. - `router`: trace router execution, including router rebuilding. - `http_client`: trace OpenResty HTTP client requests. - `balancer`: trace balancer retries. - `plugin_rewrite`: trace plugins iterator execution with rewrite phase. - `plugin_access`: trace plugins iterator execution with access phase. -- `plugin_header_filter`: trace plugins iterator execution with header_filter phase. +- `plugin_header_filter`: trace plugins iterator execution with `header_filter` phase. ## Propagation diff --git a/src/gateway/kong-plugins/observability/tracing/write-custom-trace-exporter.md b/src/gateway/kong-plugins/observability/tracing/write-custom-trace-exporter.md index a26158ebf674..7c94b58e637f 100644 --- a/src/gateway/kong-plugins/observability/tracing/write-custom-trace-exporter.md +++ b/src/gateway/kong-plugins/observability/tracing/write-custom-trace-exporter.md @@ -1,24 +1,17 @@ --- -title: Write a custom trace exporter +title: How to write a custom trace exporter +content-type: how-to --- Kong bundled OpenTelemetry plugin in core with a implementation of [OTLP/HTTP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md#otlphttp), but you can still write your own exporter at scale. -The following sections shows how to write a custom trace exporter. - -## Before you start - -Please make sure you have read the following sections: - -- [Documentation of tracing framework](/gateway/{{page.kong_version}}/observability/tracing-framework) -- [Plugin Development](/gateway/{{page.kong_version}}/plugin-development) ## Gathering the spans The spans are stored in the tracer's buffer. The buffer is a queue of spans that are awaiting to be sent to the backend. -You can access the buffer and process the span using the `process_span` function. +You can access the buffer and process the span using the `span_processor` function. ```lua -- Use the global tracer @@ -37,3 +30,12 @@ The `span_processor` function should be called in the `log` phase of the plugin. ## Full example Refer to [Github](https://github.com/Kong/kong/tree/master/spec/fixtures/custom_plugins/kong/plugins/tcp-trace-exporter) to see the example of a custom trace exporter. + + + +## More Information + +For more information about plugin development and the tracing framework, read the following docs: + +- [Documentation of tracing framework](/gateway/plugin-development/observability/tracing-framework) +- [Plugin Development](/gateway/plugin-development/plugin-development) \ No newline at end of file diff --git a/src/gateway/plugin-development/telemetry.md b/src/gateway/plugin-development/telemetry.md index 927e5d952bf7..7ff512e4f948 100644 --- a/src/gateway/plugin-development/telemetry.md +++ b/src/gateway/plugin-development/telemetry.md @@ -18,4 +18,4 @@ and the API is in the `kong.tracing` namespace. ### Reference - [Tracing PDK](/pdk/kong.tracing) -- [Tracing Framework](/observability/tracing-framework) +- [Tracing Framework](/gateway/plugin-development/observability/tracing-framework) From 19e2e7470102fb5d8fef71184a4ec7cdfcaef05a Mon Sep 17 00:00:00 2001 From: Mayo Date: Mon, 8 Aug 2022 13:23:39 +0800 Subject: [PATCH 6/7] remove plugin development doc --- src/gateway/plugin-development/telemetry.md | 21 --------------------- 1 file changed, 21 deletions(-) delete mode 100644 src/gateway/plugin-development/telemetry.md diff --git a/src/gateway/plugin-development/telemetry.md b/src/gateway/plugin-development/telemetry.md deleted file mode 100644 index 7ff512e4f948..000000000000 --- a/src/gateway/plugin-development/telemetry.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Plugin Development - Measuring your plugin -book: plugin_dev -chapter: 11 ---- - -This guide will provide you with step-by-step instructions -that will make it possible to measure your plugin at the runtime. - -These steps should be applied to each node in your Kong cluster, to ensure the -custom plugin(s) are available on each one of them. - -## Tracing - -Kong bundled the tracing framework since version 3.0.0, the tracing API is a part of the Kong core, -and the API is in the `kong.tracing` namespace. - -### Reference - -- [Tracing PDK](/pdk/kong.tracing) -- [Tracing Framework](/gateway/plugin-development/observability/tracing-framework) From 410301c0391ed1f415ed0461bb8496426cf71d70 Mon Sep 17 00:00:00 2001 From: Mayo Date: Mon, 8 Aug 2022 13:47:36 +0800 Subject: [PATCH 7/7] move tracing to kong production --- app/_data/docs_nav_gateway_3.0.x.yml | 18 ++++++++---------- .../tracing/api.md | 1 - .../tracing/index.md | 0 .../tracing/write-custom-trace-exporter.md | 9 --------- 4 files changed, 8 insertions(+), 20 deletions(-) rename src/gateway/{kong-plugins/observability => kong-production}/tracing/api.md (99%) rename src/gateway/{kong-plugins/observability => kong-production}/tracing/index.md (100%) rename src/gateway/{kong-plugins/observability => kong-production}/tracing/write-custom-trace-exporter.md (79%) diff --git a/app/_data/docs_nav_gateway_3.0.x.yml b/app/_data/docs_nav_gateway_3.0.x.yml index 27be6feede89..8c6c069cf3d2 100644 --- a/app/_data/docs_nav_gateway_3.0.x.yml +++ b/app/_data/docs_nav_gateway_3.0.x.yml @@ -143,6 +143,14 @@ items: url: /kong-production/monitoring/prometheus - text: How to monitor with Statsd url: /kong-production/monitoring/statsd + - text: Tracing + items: + - text: Overview + url: /kong-production/tracing/ + - text: How to Write a Custom Trace Exporter + url: /kong-production/tracing/write-custom-trace-exporter + - text: Tracing API reference + url: /kong-production/tracing/api - text: Resource Sizing Guidelines url: /kong-production/sizing-guidelines - text: Kong security update process @@ -360,16 +368,6 @@ items: url: /plugin-development/tests - text: (un)Installing your plugin url: /plugin-development/distribution - - text: Observability - items: - - text: Tracing - items: - - text: Overview - url: /kong-plugins/observability/tracing/ - - text: How to Write a Custom Trace Exporter - url: /kong-plugins/observability/tracing/write-custom-trace-exporter - - text: Tracing API reference - url: /kong-plugins/observability/tracing/api - text: Plugin Development Kit items: - text: Overview diff --git a/src/gateway/kong-plugins/observability/tracing/api.md b/src/gateway/kong-production/tracing/api.md similarity index 99% rename from src/gateway/kong-plugins/observability/tracing/api.md rename to src/gateway/kong-production/tracing/api.md index f80780c8f26d..20d246c8b8e1 100644 --- a/src/gateway/kong-plugins/observability/tracing/api.md +++ b/src/gateway/kong-production/tracing/api.md @@ -203,5 +203,4 @@ Please refer to the [OpenTelemetry plugin](/hub/kong-inc/opentelemetry) to see h ## References - [Tracing PDK](/gateway/{{page.kong_version}}/pdk/kong.tracing) -- [Measuring your plugin](/gateway/plugin-development/telemetry) - [OpenTelemetry plugin](/hub/kong-inc/opentelemetry) diff --git a/src/gateway/kong-plugins/observability/tracing/index.md b/src/gateway/kong-production/tracing/index.md similarity index 100% rename from src/gateway/kong-plugins/observability/tracing/index.md rename to src/gateway/kong-production/tracing/index.md diff --git a/src/gateway/kong-plugins/observability/tracing/write-custom-trace-exporter.md b/src/gateway/kong-production/tracing/write-custom-trace-exporter.md similarity index 79% rename from src/gateway/kong-plugins/observability/tracing/write-custom-trace-exporter.md rename to src/gateway/kong-production/tracing/write-custom-trace-exporter.md index 7c94b58e637f..8f5481692671 100644 --- a/src/gateway/kong-plugins/observability/tracing/write-custom-trace-exporter.md +++ b/src/gateway/kong-production/tracing/write-custom-trace-exporter.md @@ -30,12 +30,3 @@ The `span_processor` function should be called in the `log` phase of the plugin. ## Full example Refer to [Github](https://github.com/Kong/kong/tree/master/spec/fixtures/custom_plugins/kong/plugins/tcp-trace-exporter) to see the example of a custom trace exporter. - - - -## More Information - -For more information about plugin development and the tracing framework, read the following docs: - -- [Documentation of tracing framework](/gateway/plugin-development/observability/tracing-framework) -- [Plugin Development](/gateway/plugin-development/plugin-development) \ No newline at end of file