Add tracing sections for gateway 3.0 (#4014)

* add tracing

* fix links

* fix broken links

* placeholder

* copy edit

* remove plugin development doc

* move tracing to kong production

Co-authored-by: angel <angel.guarisma@konghq.com>

app/_data/docs_nav_gateway_3.0.x.yml

@@ -152,6 +152,14 @@ items:
             url: /kong-production/monitoring/prometheus
           - text: How to monitor with Statsd
             url: /kong-production/monitoring/statsd
+      - text: Tracing
+        items: 
+          - text: Overview
+            url: /kong-production/tracing/
+          - text: How to Write a Custom Trace Exporter
+            url: /kong-production/tracing/write-custom-trace-exporter
+          - text: Tracing API reference
+            url: /kong-production/tracing/api
       - text: Resource Sizing Guidelines
         url: /kong-production/sizing-guidelines
       - text: Kong security update process
@@ -369,7 +377,7 @@ items:
         url: /plugin-development/tests
       - text: (un)Installing your plugin
         url: /plugin-development/distribution
-      - text: Plugin Developmen Kit
+      - text: Plugin Development Kit
         items:
         - text: Overview
           url: /plugin-development/pdk/
@@ -452,6 +460,7 @@ items:
                 url:  /kong-plugins/rate-limiting/algorithms/rate-limiting
               - text: Configuring Fixed bucket
                 url: /kong-plugins/rate-limiting/algorithms/fixed-bucket
+
       - text: GraphQL
         url: /kong-plugins/graphql-quickstart
       - text: gRPC

src/gateway/kong-production/tracing/api.md

@@ -0,0 +1,206 @@
+---
+title: Tracing API Referenece
+content-type: Reference
+---
+
+## Before you start
+
+In Gateway version 3.0.0, the tracing API became part of the Kong core application.
+The API is in the `kong.tracing` namespace.
+
+The tracing API follows the [OpenTelemetry API specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md).
+This specification defines how to use the API as an instrument to your module. 
+If you are familiar with the OpenTelemetry API, the tracing API will be familiar. 
+
+With the tracing API, you can set the instrumentation of your module with the following operations:
+- [Span](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#span)
+- [Attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md)
+
+## Create a tracer
+
+Kong uses a global tracer internally to instrument the core modules and plugins.
+
+By default, the tracer is a [NoopTracer](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#get-a-tracer). The tracer is first initialized when `opentelemetry_tracing` configuration is enabled.
+
+You can create a new tracer manually, or use the global tracer instance: 
+
+```lua
+local tracer
+
+-- Create a new tracer
+tracer = kong.tracing.new("custom-tracer")
+
+-- Use the global tracer
+tracer = kong.tracing
+```
+
+### Sampling traces
+
+The sampling rate of a tracer can be configured: 
+
+```lua
+local tracer = kong.tracing.new("custom-tracer", {
+  -- Set the sampling rate to 0.1
+  sampling_rate = 0.1,
+})
+```
+
+The default sampling rate is `1.0`, which samples all traces.
+
+## Create a span
+
+A span represents a single operation within a trace. Spans can be nested to form trace trees. Each trace contains a root span, which typically describes the entire operation and, optionally, one or more sub-spans for its sub-operations.
+
+```lua
+local tracer = kong.tracing
+
+local span = tracer:start_span("my-span")
+```
+
+The span properties can be set by passing a table to the `start_span` method.
+
+```lua
+local span = tracer:start_span("my-span", {
+  start_time_ns = ngx.now() * 1e9, -- override the start time
+  span_kind = 2, -- SPAN_KIND
+                  -- UNSPECIFIED: 0
+                  -- INTERNAL: 1
+                  -- SERVER: 2
+                  -- CLIENT: 3
+                  -- PRODUCER: 4
+                  -- CONSUMER: 5
+  should_sample = true, -- by setting it to `true` to ignore the sampling decision
+})
+```
+
+Make sure to ends the span when you are done:
+
+```lua
+span:finish() -- ends the span
+```
+
+{:.Note}
+>**Note:** The span table will be cleared and put into the table pool after the span is finished,
+do not use it after the span is finished.
+
+## Get or set the active span
+
+The active span is the span that is currently being executed.
+
+To avoid overheads, the active span is manually set by calling the `set_active_span` method.
+When you finish a span, the active span becomes the parent of the finished span.
+
+
+To set or get the active span, you can use the following example code:
+
+```lua
+local tracer = kong.tracing
+local span = tracer:start_span("my-span")
+tracer.set_active_span(span)
+
+local active_span = tracer.active_span() -- returns the active span
+```
+
+### Scope
+
+The tracers are scoped to a specific context by a namespace key.
+
+To get the active span for a specific namespace, you can use the following:
+
+```lua
+-- get global tracer's active span, and set it as the parent of new created span
+local global_tracer = kong.tracing
+local tracer = kong.tracing.new("custom-tracer")
+
+local root_span = global_tracer.active_span()
+local span = tracer.start_span("my-span", {
+  parent = root_span
+})
+```
+
+## Set the span attributes
+
+The attributes of a span are a map of key-value pairs
+and can be set by passing a table to the `set_attributes` method.
+
+```lua
+local span = tracer:start_span("my-span")
+```
+
+The OpenTelemetry specification defines the general semantic attributes, it can be used to describe the span.
+It could also be meaningful to visualize the span in a UI.
+
+```lua
+span:set_attribute("key", "value")
+```
+
+The following semantic conventions for spans are defined:
+
+* [General](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md): General semantic attributes that may be used in describing different kinds of operations.
+* [HTTP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md): For HTTP client and server spans.
+* [Database](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/database.md): For SQL and NoSQL client call spans.
+* [RPC/RMI](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/rpc.md): For remote procedure call (e.g., gRPC) spans.
+* [Messaging](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/messaging.md): For messaging systems (queues, publish/subscribe, etc.) spans.
+* [FaaS](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/faas.md): For [Function as a Service](https://en.wikipedia.org/wiki/Function_as_a_service) (e.g., AWS Lambda) spans.
+* [Exceptions](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/exceptions.md): For recording exceptions associated with a span.
+* [Compatibility](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/compatibility.md): For spans generated by compatibility components, e.g. OpenTracing Shim layer.
+
+## Set the span events
+
+The events of a span are time-series events that can be set by passing a table to the `add_event` method.
+
+```lua
+local span = kong.tracing:start_span("my-span")
+span:add_event("my-event", {
+  -- attributes
+  ["key"] = "value",
+})
+```
+
+### Record error message
+
+The event could also be used to record error messages.
+
+```lua
+local span = kong.tracing:start_span("my-span")
+span:record_error("my-error-message")
+
+-- or (same as above)
+span:add_event("exception", {
+  ["exception.message"] = "my-error-message",
+})
+```
+
+## Set the span status
+
+The status of a span is a status code and can be set by passing a table to the `set_status` method.
+
+```lua
+local span = kong.tracing:start_span("my-span")
+-- Status codes:
+-- - `0` unset
+-- - `1` ok
+-- - `2` error
+```
+
+## Release the span (optional)
+
+The spans are stored in a pool, and can be released by calling the `release` method.
+
+```lua
+local span = kong.tracing:start_span("my-span")
+span:release()
+```
+
+By default, the span will be released after the Nginx request ends.
+
+## Visualize the trace
+
+Because of the compatibility with OpenTelemetry, the traces can be natively visualized through any OpenTelemetry UI.
+
+Please refer to the [OpenTelemetry plugin](/hub/kong-inc/opentelemetry) to see how to visualize the traces.
+
+## References
+
+- [Tracing PDK](/gateway/{{page.kong_version}}/pdk/kong.tracing)
+- [OpenTelemetry plugin](/hub/kong-inc/opentelemetry)

src/gateway/kong-production/tracing/index.md

@@ -0,0 +1,40 @@
+---
+title: Tracing Reference
+content-type: reference
+---
+
+In this section, we will describe the tracing capabilities of Kong.
+
+## Core instrumentations
+
+**Note**
+Only works for the plugins that are built on top of Kong's tracing API.
+e.g. OpenTelemetry plugin.
+
+Kong provides a set of core instrumentations for tracing, these can be configured in the `opentelemetry_tracing` configuration.
+
+- `off`: do not enable instrumentations.
+- `request`: only enable request-level instrumentations.
+- `all`: enable all the following instrumentations.
+- `db_query`: trace database query, including PostgresSQL and Cassandra.
+- `dns_query`: trace DNS query.
+- `router`: trace router execution, including router rebuilding.
+- `http_client`: trace OpenResty HTTP client requests.
+- `balancer`: trace balancer retries.
+- `plugin_rewrite`: trace plugins iterator execution with rewrite phase.
+- `plugin_access`: trace plugins iterator execution with access phase.
+- `plugin_header_filter`: trace plugins iterator execution with `header_filter` phase.
+
+## Propagation
+
+The tracing API support to propagate the following headers:
+- `w3c` - [W3C trace context](https://www.w3.org/TR/trace-context/)
+- `b3`, `b3-single` - [Zipkin headers](https://github.com/openzipkin/b3-propagation)
+- `jaeger` - [Jaeger headers](https://www.jaegertracing.io/docs/client-libraries/#propagation-format)
+- `ot` - [OpenTracing headers](https://github.com/opentracing/specification/blob/master/rfc/trace_identifiers.md)
+- `datadog` - [Datadog headers](https://docs.datadoghq.com/tracing/agent/propagation/)
+
+The tracing API will detect the propagation format from the headers, and will use the appropriate format to propagate the span context.
+If no appropriate format is found, then will fallback to the default format, which can be specified.
+
+The propagation api works for both the OpenTelemetry plugin and the Zipkin plugin.

src/gateway/kong-production/tracing/write-custom-trace-exporter.md

@@ -0,0 +1,32 @@
+---
+title: How to write a custom trace exporter
+content-type: how-to
+---
+
+Kong bundled OpenTelemetry plugin in core with a implementation of [OTLP/HTTP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md#otlphttp), but you can still write your own exporter at scale.
+
+
+## Gathering the spans
+
+The spans are stored in the tracer's buffer.
+The buffer is a queue of spans that are awaiting to be sent to the backend.
+
+You can access the buffer and process the span using the `span_processor` function.
+
+```lua
+-- Use the global tracer
+local tracer = kong.tracing
+
+-- Process the span
+local span_processor = function(span)
+    -- clone the span so it can be processed after the original one is cleared
+    local span_dup = table.clone(span)
+    -- you can transform the span, add tags, etc. to other specific data structures
+end
+```
+
+The `span_processor` function should be called in the `log` phase of the plugin.
+
+## Full example
+
+Refer to [Github](https://github.com/Kong/kong/tree/master/spec/fixtures/custom_plugins/kong/plugins/tcp-trace-exporter) to see the example of a custom trace exporter.