Skip to content

Commit

Permalink
add tracing
Browse files Browse the repository at this point in the history
  • Loading branch information
@mayocream
mayocream committed Jul 26, 2022
1 parent 5f26984 commit c1c1716
Show file tree
Hide file tree
Showing 5 changed files with 356 additions and 6 deletions.
54 changes: 48 additions & 6 deletions app/_data/docs_nav_gateway_3.0.x.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,7 @@ items:
- text: Helpers CLI
url: /developer-portal/helpers/cli

- title: Monitor
- title: Observability
icon: /assets/images/icons/documentation/icn-vitals.svg
items:
- text: Kong Vitals
Expand All @@ -310,11 +310,50 @@ items:
url: /vitals/vitals-prometheus-strategy
- text: Estimate Vitals Storage in PostgreSQL
url: /vitals/vitals-estimates
- text: Prometheus plugin
url: /hub/kong-inc/prometheus
absolute_url: true
- text: Zipkin plugin
url: /hub/kong-inc/zipkin
- text: Logs
items:
- text: File Log plugin
url: /hub/kong-inc/file-log
absolute_url: true
- text: HTTP Log plugin
url: /hub/kong-inc/http-log
absolute_url: true
- text: Kafka Log plugin
url: /hub/kong-inc/kafka-log
absolute_url: true
- text: TCP Log plugin
url: /hub/kong-inc/tcp-log
absolute_url: true
- text: UDP Log plugin
url: /hub/kong-inc/udp-log
absolute_url: true
- text: Syslog plugin
url: /hub/kong-inc/syslog
absolute_url: true
- text: Metrics
items:
- text: Prometheus plugin
url: /hub/kong-inc/prometheus
absolute_url: true
- text: StatsD plugin
url: /hub/kong-inc/statsd
absolute_url: true
- text: Tracing
items:
- text: OpenTelemetry tracing
url: /observability/tracing
absolute_url: true
- text: Write custom trace exporter
url: /observability/tracing/write-custom-trace-exporter
absolute_url: true
- text: Zipkin plugin
url: /hub/kong-inc/zipkin
absolute_url: true
- text: OpenTelemetry Plugin
url: /hub/kong-inc/opentelemetry
absolute_url: true
- text: Tracing Framework
url: /observability/tracing/api
absolute_url: true

- title: Reference
Expand Down Expand Up @@ -507,5 +546,8 @@ items:
- text: (un)Installing your plugin
url: /plugin-development/distribution

- text: Measuring your plugin
url: /plugin-development/telemetry

- text: Plugins in Other Languages
url: /reference/external-plugins
206 changes: 206 additions & 0 deletions src/gateway/observability/tracing/api.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
---
title: Tracing API
---

## Before you start

Kong bundled the tracing framework since version 3.0.0, the tracing API is a part of the Kong core,
and the API is in the `kong.tracing` namespace.

To make the tracing API standardized and available to all plugins,
we follow the [OpenTelemetry API specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md).
This specification defines the API that should be used to instrument your module.
If you already have the experience of using the OpenTelemetry API, it should be much easier to start.

With the tracing API, you can instrument your module with the following:
- [Span](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#span)
- [Attributes](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md)

## Create a tracer

Kong internally uses a global tracer instance to instrument the core modules and plugins.

By default, the tracer is a [NoopTracer](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#get-a-tracer) which does nothing to avoid any overhead. The tracer is first initialized when `opentelemetry_tracing` configuration enabled.

You can either create a new tracer manually, or use the global tracer instance.

```lua
local tracer

-- Create a new tracer
tracer = kong.tracing.new("custom-tracer")

-- Use the global tracer
tracer = kong.tracing
```

### Sampling traces

The tracer can be configured to sample traces.

```lua
local tracer = kong.tracing.new("custom-tracer", {
-- Set the sampling rate to 0.1
sampling_rate = 0.1,
})
```

By default, the sampling rate is 1.0, meaning that all traces are sampled.

## Create a span

A Span represents a single operation within a trace. Spans can be nested to form a trace tree. Each trace contains a root span, which typically describes the entire operation and, optionally, one or more sub-spans for its sub-operations.

```lua
local tracer = kong.tracing

local span = tracer:start_span("my-span")
```

The span properties can be set by passing a table to the `start_span` method.

```lua
local span = tracer:start_span("my-span", {
start_time_ns = ngx.now() * 1e9, -- override the start time
span_kind = 2, -- SPAN_KIND
-- UNSPECIFIED: 0
-- INTERNAL: 1
-- SERVER: 2
-- CLIENT: 3
-- PRODUCER: 4
-- CONSUMER: 5
should_sample = true, -- by setting it to `true` to ignore the sampling decision
})
```

Please make sure to ends the span when you are done.

```lua
span:finish() -- ends the span
```

**Note:** The span table will be cleared and put into the table pool after the span is finished,
do not use it after the span is finished.

## Get/Set the active span

The active span is the span that is currently being executed.

To avoid overheads, the active span is manually set by calling the `set_active_span` method.
However, When you finish a span, the active span becomes the parent of the finished span.


To set/get the active span, you can use the following:

```lua
local tracer = kong.tracing
local span = tracer:start_span("my-span")
tracer.set_active_span(span)

local active_span = tracer.active_span() -- returns the active span
```

### Scope

The tracers are scoped to a specific context by a namespace key.

To get the active span for a specific namespace, you can use the following:

```lua
-- get global tracer's active span, and set it as the parent of new created span
local global_tracer = kong.tracing
local tracer = kong.tracing.new("custom-tracer")

local root_span = global_tracer.active_span()
local span = tracer.start_span("my-span", {
parent = root_span
})
```

## Set the span attributes

The attributes of a span are a map of key-value pairs
and can be set by passing a table to the `set_attributes` method.

```lua
local span = tracer:start_span("my-span")
```

The OpenTelemetry specification defines the general semantic attributes, it can be used to describe the span.
It could also be meaningful to visualize the span in a UI.

```lua
span:set_attribute("key", "value")
```

The following semantic conventions for spans are defined:

* [General](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md): General semantic attributes that may be used in describing different kinds of operations.
* [HTTP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md): For HTTP client and server spans.
* [Database](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/database.md): For SQL and NoSQL client call spans.
* [RPC/RMI](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/rpc.md): For remote procedure call (e.g., gRPC) spans.
* [Messaging](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/messaging.md): For messaging systems (queues, publish/subscribe, etc.) spans.
* [FaaS](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/faas.md): For [Function as a Service](https://en.wikipedia.org/wiki/Function_as_a_service) (e.g., AWS Lambda) spans.
* [Exceptions](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/exceptions.md): For recording exceptions associated with a span.
* [Compatibility](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/compatibility.md): For spans generated by compatibility components, e.g. OpenTracing Shim layer.

## Set the span events

The events of a span are time-series events that can be set by passing a table to the `add_event` method.

```lua
local span = kong.tracing:start_span("my-span")
span:add_event("my-event", {
-- attributes
["key"] = "value",
})
```

### Record error message

The event could also be used to record error messages.

```lua
local span = kong.tracing:start_span("my-span")
span:record_error("my-error-message")

-- or (same as above)
span:add_event("exception", {
["exception.message"] = "my-error-message",
})
```

## Set the span status

The status of a span is a status code and can be set by passing a table to the `set_status` method.

```lua
local span = kong.tracing:start_span("my-span")
-- Status codes:
-- - `0` unset
-- - `1` ok
-- - `2` error
```

## Release the span (optional)

The spans are stored in a pool, and can be released by calling the `release` method.

```lua
local span = kong.tracing:start_span("my-span")
span:release()
```

By default, the span will be released after the Nginx request ends.

## Visualize the trace

Because of the compatibility with OpenTelemetry, the traces can be natively visualized through any OpenTelemetry UI.

Please refer to the [OpenTelemetry plugin](/hub/kong-inc/opentelemetry) to see how to visualize the traces.

## References

- [Tracing PDK](/pdk/kong.tracing)
- [Measuring your plugin](/plugin-development/telemetry)
- [OpenTelemetry plugin](/hub/kong-inc/opentelemetry)
42 changes: 42 additions & 0 deletions src/gateway/observability/tracing/index.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
title: Tracing
---

## Core instrumentations

| OT Category | Kong Components | Name | Span data | Notes |
| ------------------------- | --------------------------- | ------------------------------------ | ----------------------------------- | ------------------------------------------------------- |
| Database | Storage | DB query | SQL | database spec |
| Redis | Redis call | database spec | | |
| Redis cluster | Redis call | extra support for resty.rediscluster | | |
| LMDB | Transactions | database spec | | |
| HTTP | Core | HTTP Server Request / Consumer | Nginx server request | http spec |
| Balancer Retry / Upstream | Nginx balancer retries data | http spec | | |
| Core / Plugins | HTTP Client Request | HTTP client request | http spec, resty.http, e.g. plugins | |
| RPC | Core | gRPC | gRPC call | rpc spec |
| Messaging | Stream | Pub | OpenResty pub/sub | messaging spec |
| Sub | OpenResty pub/sub | | | |
| Kafka | | messaging spec | | |
| FAAS | Plugins | AWS Lambda | remote call | faas spec |
| Azure functions | | | | |
| Exceptions | Errors | OpenResty errors | | Capture runtime errors (logs can be linked by trace_id) |
| Networking | Core | DNS query | host, success | general spec |
| Plugins | LDAP | LDAP operations | ldap spec | |
| LDAP Enhanced | LDAP operations | ldap spec | | |
| OpenID Connect | | | | |
| GraphQL | GraphQL query | | | |
| Vault | Vault query | | | |
| Kong Internal | Core | Router | Execution time | |
| Plugin execution | Plugin name, execution time | start/end time only | | |

## Propagation

The tracing API support to propagate the following headers:
- `w3c` - [W3C trace context](https://www.w3.org/TR/trace-context/)
- `b3`, `b3-single` - [Zipkin headers](https://github.com/openzipkin/b3-propagation)
- `jaeger` - [Jaeger headers](https://www.jaegertracing.io/docs/client-libraries/#propagation-format)
- `ot` - [OpenTracing headers](https://github.com/opentracing/specification/blob/master/rfc/trace_identifiers.md)
- `datadog` - [Datadog headers](https://docs.datadoghq.com/tracing/agent/propagation/)

The tracing API will detect the propagation format from the headers, and will use the appropriate format to propagate the span context.
If no appropriate format is found, then will fallback to the default format, which can be specified.
39 changes: 39 additions & 0 deletions src/gateway/observability/tracing/write-custom-trace-exporter.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
title: Write a custom trace exporter
---

Kong bundled OpenTelemetry plugin in core with a implementation of [OTLP/HTTP](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md#otlphttp), but you can still write your own exporter at scale.

The following sections shows how to write a custom trace exporter.

## Before you start

Please make sure you have read the following sections:

- [Documentation of tracing framework](/gateway/{{page.kong_version}}/observability/tracing-framework)
- [Plugin Development](/gateway/{{page.kong_version}}/plugin-development)

## Gathering the spans

The spans are stored in the tracer's buffer.
The buffer is a queue of spans that are awaiting to be sent to the backend.

You can access the buffer and process the span using the `process_span` function.

```lua
-- Use the global tracer
local tracer = kong.tracing

-- Process the span
local span_processor = function(span)
-- clone the span so it can be processed after the original one is cleared
local span_dup = table.clone(span)
-- you can transform the span, add tags, etc. to other specific data structures
end
```

The `span_processor` function should be called in the `log` phase of the plugin.

## Full example

Refer to [Github](https://github.com/Kong/kong/tree/master/spec/fixtures/custom_plugins/kong/plugins/tcp-trace-exporter) to see the example of a custom trace exporter.
21 changes: 21 additions & 0 deletions src/gateway/plugin-development/telemetry.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
title: Plugin Development - Measuring your plugin
book: plugin_dev
chapter: 11
---

This guide will provide you with step-by-step instructions
that will make it possible to measure your plugin at the runtime.

These steps should be applied to each node in your Kong cluster, to ensure the
custom plugin(s) are available on each one of them.

## Tracing

Kong bundled the tracing framework since version 3.0.0, the tracing API is a part of the Kong core,
and the API is in the `kong.tracing` namespace.

### Reference

- [Tracing PDK](/pdk/kong.tracing)
- [Tracing Framework](/observability/tracing-framework)

0 comments on commit c1c1716

Please sign in to comment.