Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP feat: support customizing additional labels of metrics #6450

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 24 additions & 9 deletions apisix/plugins/prometheus/exporter.lua
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ local metrics = {}

local inner_tab_arr = {}

local additional_label_names = {}

local function gen_arr(...)
clear_tab(inner_tab_arr)
for i = 1, select('#', ...) do
Expand Down Expand Up @@ -81,6 +83,7 @@ function _M.init()
if attr and attr.metric_prefix then
metric_prefix = attr.metric_prefix
end
additional_label_names = attr.labels or {}

prometheus = base_prometheus.init("prometheus-metrics", metric_prefix)
metrics.connections = prometheus:gauge("nginx_http_current_connections",
Expand Down Expand Up @@ -109,15 +112,17 @@ function _M.init()
-- no consumer in request.
metrics.status = prometheus:counter("http_status",
"HTTP status codes per service in APISIX",
{"code", "route", "matched_uri", "matched_host", "service", "consumer", "node"})
{"code", "route", "matched_uri", "matched_host", "service", "consumer", "node",
table.unpack(additional_label_names)})

metrics.latency = prometheus:histogram("http_latency",
"HTTP request latency in milliseconds per service in APISIX",
{"type", "route", "service", "consumer", "node"}, DEFAULT_BUCKETS)
{"type", "route", "service", "consumer", "node", table.unpack(additional_label_names)}
, DEFAULT_BUCKETS)

metrics.bandwidth = prometheus:counter("bandwidth",
"Total bandwidth in bytes consumed per service in APISIX",
{"type", "route", "service", "consumer", "node"})
{"type", "route", "service", "consumer", "node", table.unpack(additional_label_names)})

end

Expand Down Expand Up @@ -149,28 +154,38 @@ function _M.log(conf, ctx)
matched_uri = ctx.curr_req_matched._path or ""
matched_host = ctx.curr_req_matched._host or ""
end

local additional_label_values = {}
for i, name in pairs(additional_label_names) do
table.insert(additional_label_values, vars[name])
end

metrics.status:inc(1,
gen_arr(vars.status, route_id, matched_uri, matched_host,
service_id, consumer_name, balancer_ip))
service_id, consumer_name, balancer_ip, table.unpack(additional_label_values)))

local latency, upstream_latency, apisix_latency = latency_details(ctx)
metrics.latency:observe(latency,
gen_arr("request", route_id, service_id, consumer_name, balancer_ip))
gen_arr("request", route_id, service_id, consumer_name, balancer_ip,
table.unpack(additional_label_values)))

if upstream_latency then
metrics.latency:observe(upstream_latency,
gen_arr("upstream", route_id, service_id, consumer_name, balancer_ip))
gen_arr("upstream", route_id, service_id, consumer_name, balancer_ip,
table.unpack(additional_label_values)))
end

metrics.latency:observe(apisix_latency,
gen_arr("apisix", route_id, service_id, consumer_name, balancer_ip))
gen_arr("apisix", route_id, service_id, consumer_name, balancer_ip,
table.unpack(additional_label_values)))

metrics.bandwidth:inc(vars.request_length,
gen_arr("ingress", route_id, service_id, consumer_name, balancer_ip))
gen_arr("ingress", route_id, service_id, consumer_name, balancer_ip,
table.unpack(additional_label_values)))

metrics.bandwidth:inc(vars.bytes_sent,
gen_arr("egress", route_id, service_id, consumer_name, balancer_ip))
gen_arr("egress", route_id, service_id, consumer_name, balancer_ip,
table.unpack(additional_label_values)))
end


Expand Down
3 changes: 3 additions & 0 deletions conf/config-default.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,9 @@ plugin_attr:
export_addr:
ip: 127.0.0.1
port: 9091
# labels:
#remote_addr: $remote_addr
#request_method: $request_method
Copy link
Contributor

@zhendongcmss zhendongcmss Feb 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is useful for us. we add metrics too on our code as hard code way.
Can you list lables or give a link which labels are support on doc ?

server-info:
report_interval: 60 # server info report interval (unit: second)
report_ttl: 3600 # live time for server info in etcd (unit: second)
Expand Down
20 changes: 19 additions & 1 deletion docs/en/latest/plugins/prometheus.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,10 +129,28 @@ Here is an example:

```yaml
plugin_attr:
prometheus:
prometheus:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
prometheus:
prometheus:

There seems to be a mistake here.

export_uri: /apisix/metrics
```

## How to custom additional labels

The plugin is supported to custom labels for metrics, `Status codes` and `Latency`, in the `plugin_attr` section of `conf/config.yaml`

| Name | Type | Default | Description |
| ---------- | ------ | ------- | --------------------------------- |
| labels | array | | custom labels from [Nginx variables](http://nginx.org/en/docs/varindex.html) |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better include the references to APISIX build-in variables as well


Here is an example:

```yaml:
plugin_attr:
prometheus:
labels:
- request_method
- server_addr
```

### Grafana dashboard

Metrics exported by the plugin can be graphed in Grafana using a drop in dashboard.
Expand Down
18 changes: 18 additions & 0 deletions docs/zh/latest/plugins/prometheus.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,24 @@ plugin_attr:
export_uri: /apisix/metrics
```

## 如何自定义附加标签

该插件支持在`conf/config`的`plugin_attr`部分自定义度量标签`Status codes`和`Latency`。亚马尔`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

亚马尔 ??
Maybe `yaml" ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
该插件支持在`conf/config``plugin_attr`部分自定义度量标签`Status codes``Latency`亚马尔`
该插件支持在 `conf/config``plugin_attr` 部分自定义度量标签 `Status codes``Latency`


| 名称 | 类型 | 默认值 | 描述
| --------- | ----- | -------- | ------------------------- |
| labels | array | | 使用 [Nginx 变量](http://nginx.org/en/docs/varindex.html)作为自定义标签 |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto


下面是一个例子:

```yaml:
plugin_attr:
prometheus:
labels:
- request_method
- server_addr
```

### Grafana 面板

插件导出的指标可以在 Grafana 进行图形化绘制显示。
Expand Down