Skip to content

Commit

Permalink
[klogs] Use maps instead of columns for custom fields (#437)
Browse files Browse the repository at this point in the history 
This is a follow up for kobsio/klogs#34 were we
changed the database schema for klogs to use maps instead of arrays for
custom fields.

To be able to query the data ingested by the klogs plugin we had to
adjust the queries to also use maps instead of columns for these fields.
  • Loading branch information
@ricoberger
ricoberger authored Oct 17, 2022
1 parent cdfaeb8 commit c6edeb6
Show file tree
Hide file tree
Showing 19 changed files with 690 additions and 912 deletions.
58 changes: 31 additions & 27 deletions docs/plugins/klogs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,10 @@ To use the klogs plugin the following configuration is needed in the satellites
| options.database | string | The name of the database. | Yes |
| options.username | string | Username to access a ClickHouse instance. | No |
| options.password | string | Password to access a ClickHouse instance. | No |
| options.readTimeout | string | The read timeout for operations. | No |
| options.writeTimeout | string | The write timeout for operations. | No |
| options.dialTimeout | string | ClickHouse dial timeout. The default value is `10s`. | No |
| options.connMaxLifetime | string | ClickHouse maximum connection lifetime. The default value is `1h`. | No |
| options.maxIdleConns | number | ClickHouse maximum number of idle connections. The default value is `5`. | No |
| options.maxOpenConns | number | ClickHouse maximum number of open connections. The default value is `10`. | No |
| options.materializedColumns | []string | A list of materialized columns. See [kobsio/klogs](https://github.com/kobsio/klogs#configuration) for more information. | No |

```yaml
Expand All @@ -27,10 +29,12 @@ plugins:
options:
address:
database:
writeTimeout:
readTimeout:
username:
password:
dialTimeout:
connMaxLifetime:
maxIdleConns:
maxOpenConns:
materializedColumns:
```
Expand Down Expand Up @@ -107,16 +111,16 @@ kobs supports multiple operators which can be used in a query to retrieve logs f
| `_not_` | Exclude the term from the query. | `cluster='kobs-demo' _and_ _not_ namespace='bookinfo'` |
| `_and_` | Both terms must be included in the results. | `namespace='bookinfo' _and_ app='bookinfo'` |
| `_or_` | The result can contain one of the given terms. | `namespace='bookinfo' _or_ namespace='istio-system'` |
| `_exists_` | The field can not be `null` | `container_name='istio-proxy' _and_ _exists_ content.request_id` |
| `_exists_` | The field can not be `null` | `container_name='istio-proxy' _and_ _exists_ content_request_id` |
| `=` | The field must have this value. | `namespace='bookinfo'` |
| `!=` | The field should not have this value. | `namespace!='bookinfo'` |
| `>` | The value of the field must be greater than the specified value. | `content.response_code>499` |
| `>=` | The value of the field must be greater than or equal to the specified value. | `content.response_code>=500` |
| `<` | The value of the field must be lower than the specified value. | `content.response_code<500` |
| `<=` | The value of the field must be lower than or equal to the specified value. | `content.response_code<=499` |
| `=~` | The value of the field is compared using `ILIKE`. | `content.upstream_cluster=~'inbound%'` |
| `!~` | The value of the field is compared using `NOT ILIKE`. | `content.upstream_cluster!~'inbound%'` |
| `~` | The value of the field must match the regular expression. The syntax of the `re2` regular expressions can be found [here](https://github.com/google/re2/wiki/Syntax). | `content.upstream_cluster~'inbound.*'` |
| `>` | The value of the field must be greater than the specified value. | `content_response_code>499` |
| `>=` | The value of the field must be greater than or equal to the specified value. | `content_response_code>=500` |
| `<` | The value of the field must be lower than the specified value. | `content_response_code<500` |
| `<=` | The value of the field must be lower than or equal to the specified value. | `content_response_code<=499` |
| `=~` | The value of the field is compared using `ILIKE`. | `content_upstream_cluster=~'inbound%'` |
| `!~` | The value of the field is compared using `NOT ILIKE`. | `content_upstream_cluster!~'inbound%'` |
| `~` | The value of the field must match the regular expression. The syntax of the `re2` regular expressions can be found [here](https://github.com/google/re2/wiki/Syntax). | `content_upstream_cluster~'inbound.*'` |

#### Default Fields

Expand All @@ -133,7 +137,7 @@ In the following you can find a list of fields which are available for each log

#### Examples

- `namespace='bookinfo' _and_ app='bookinfo' _and_ container_name='istio-proxy' _and_ content.upstream_cluster~'inbound.*'`: Select all inbound Istio logs from the bookinfo app in the bookinfo namespace.
- `namespace='bookinfo' _and_ app='bookinfo' _and_ container_name='istio-proxy' _and_ content_upstream_cluster~'inbound.*'`: Select all inbound Istio logs from the bookinfo app in the bookinfo namespace.

### Logs Dashboard

Expand All @@ -154,18 +158,18 @@ spec:
type: logs
queries:
- name: Istio Logs
query: "namespace='bookinfo' _and_ app='bookinfo' _and_ container_name='istio-proxy' _and_ content.upstream_cluster~'inbound.*'"
query: "namespace='bookinfo' _and_ app='bookinfo' _and_ container_name='istio-proxy' _and_ content_upstream_cluster~'inbound.*'"
fields:
- "pod_name"
- "content.authority"
- "content.route_name"
- "content.protocol"
- "content.method"
- "content.path"
- "content.response_code"
- "content.upstream_service_time"
- "content.bytes_received"
- "content.bytes_sent"
- "content_authority"
- "content_route_name"
- "content_protocol"
- "content_method"
- "content_path"
- "content_response_code"
- "content_upstream_service_time"
- "content_bytes_received"
- "content_bytes_sent"
```

### Aggregation Dashboard
Expand Down Expand Up @@ -210,7 +214,7 @@ spec:
query: "cluster='kobs-demo' _and_ app='myapplication' _and_ container_name='myapplication'"
chart: pie
options:
sliceBy: content.level
sliceBy: content_level
sizeByOperation: count
- size: 3
panels:
Expand All @@ -222,14 +226,14 @@ spec:
options:
type: aggregation
aggregation:
query: "cluster='kobs-demo' _and_ app='myapplication' _and_ container_name='istio-proxy' _and_ content.response_code>0"
query: "cluster='kobs-demo' _and_ app='myapplication' _and_ container_name='istio-proxy' _and_ content_response_code>0"
chart: line
options:
horizontalAxisOperation: time
verticalAxisOperation: avg
verticalAxisField: content.duration
verticalAxisField: content_duration
breakDownByFields:
- content.response_code
- content_response_code
```

![Aggregation Example](assets/klogs-aggregation.png)
32 changes: 16 additions & 16 deletions docs/plugins/sql.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,8 @@ The following options can be used for a panel with the SQL plugin:
query: |
SELECT
toStartOfInterval(timestamp, INTERVAL 60 second) AS time,
avg(fields_number.value[indexOf(fields_number.key, 'content.duration')]) as avg_duration,
avg(fields_number.value[indexOf(fields_number.key, 'content.upstream_service_time')]) as avg_ust,
avg(fields_number['content_duration']) as avg_duration,
avg(fields_number['content_upstream_service_time']) as avg_ust,
avg_duration - avg_ust as avg_diff
FROM
logs.logs
Expand All @@ -122,7 +122,7 @@ The following options can be used for a panel with the SQL plugin:
AND namespace='myservice'
AND app='myservice'
AND container_name='istio-proxy'
AND match(fields_string.value[indexOf(fields_string.key, 'content.upstream_cluster')], '^inbound.*')
AND match(fields_string['content_upstream_cluster'], '^inbound.*')
GROUP BY
time
ORDER BY
Expand Down Expand Up @@ -153,7 +153,7 @@ The following options can be used for a panel with the SQL plugin:
query: |
SELECT
toStartOfInterval(timestamp, INTERVAL 60 second) AS time,
avg(fields_number.value[indexOf(fields_number.key, 'content.duration')]) - avg(fields_number.value[indexOf(fields_number.key, 'content.upstream_service_time')]) as avg_diff
avg(fields_number['content_duration']) - avg(fields_number['content_upstream_service_time']) as avg_diff
FROM
logs.logs
WHERE
Expand All @@ -162,7 +162,7 @@ The following options can be used for a panel with the SQL plugin:
AND namespace='myservice'
AND app='myservice'
AND container_name='istio-proxy'
AND match(fields_string.value[indexOf(fields_string.key, 'content.upstream_cluster')], '^inbound.*')
AND match(fields_string['content_upstream_cluster'], '^inbound.*')
GROUP BY
time
ORDER BY
Expand All @@ -188,8 +188,8 @@ The following options can be used for a panel with the SQL plugin:
query: |
SELECT
toStartOfInterval(timestamp, INTERVAL 60 second) AS time,
avg(fields_number.value[indexOf(fields_number.key, 'content.duration')]) as avg_duration,
avg(fields_number.value[indexOf(fields_number.key, 'content.upstream_service_time')]) as avg_ust
avg(fields_number['content_duration']) as avg_duration,
avg(fields_number['content_upstream_service_time']) as avg_ust
FROM
logs.logs
WHERE
Expand All @@ -198,7 +198,7 @@ The following options can be used for a panel with the SQL plugin:
AND namespace='myservice'
AND app='myservice'
AND container_name='istio-proxy'
AND match(fields_string.value[indexOf(fields_string.key, 'content.upstream_cluster')], '^inbound.*')
AND match(fields_string['content_upstream_cluster'], '^inbound.*')
GROUP BY
time
ORDER BY
Expand Down Expand Up @@ -230,8 +230,8 @@ The following options can be used for a panel with the SQL plugin:
- name: Log Levels
query: |
SELECT
content.level,
count(content.level) as count_data
content_level,
count(content_level) as count_data
FROM
logs.logs
WHERE
Expand All @@ -241,9 +241,9 @@ The following options can be used for a panel with the SQL plugin:
AND app='myservice'
AND container_name='myservice'
GROUP BY
content.level
content_level
columns:
content.level:
content_level:
title: Level
count_data:
title: Count
Expand All @@ -258,8 +258,8 @@ The following options can be used for a panel with the SQL plugin:
type: pie
query: |
SELECT
content.level,
count(content.level) as count_data
content_level,
count(content_level) as count_data
FROM
logs.logs
WHERE
Expand All @@ -269,8 +269,8 @@ The following options can be used for a panel with the SQL plugin:
AND app='myservice'
AND container_name='myservice'
GROUP BY
content.level
pieLabelColumn: content.level
content_level
pieLabelColumn: content_level
pieValueColumn: count_data
```

Expand Down
18 changes: 12 additions & 6 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ require (
github.com/Azure/go-autorest/autorest v0.11.28
github.com/Azure/go-autorest/autorest/azure/auth v0.5.11
github.com/Azure/go-autorest/autorest/date v0.3.0
github.com/ClickHouse/clickhouse-go v1.5.4
github.com/ClickHouse/clickhouse-go/v2 v2.3.0
github.com/DataDog/datadog-api-client-go/v2 v2.3.1
github.com/andygrunwald/go-jira v1.16.0
github.com/coreos/go-oidc/v3 v3.3.0
Expand Down Expand Up @@ -72,16 +72,17 @@ require (
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v0.5.1 // indirect
github.com/ClickHouse/ch-go v0.47.3 // indirect
github.com/DataDog/zstd v1.5.0 // indirect
github.com/Microsoft/go-winio v0.5.2 // indirect
github.com/PuerkitoBio/goquery v1.5.1 // indirect
github.com/PuerkitoBio/purell v1.1.1 // indirect
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect
github.com/ajg/form v1.5.1 // indirect
github.com/andybalholm/brotli v1.0.4 // indirect
github.com/andybalholm/cascadia v1.1.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/cloudflare/golz4 v0.0.0-20150217214814-ef862a3cdc58 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dimchansky/utfbom v1.1.1 // indirect
github.com/docker/distribution v2.8.0+incompatible // indirect
Expand All @@ -94,6 +95,8 @@ require (
github.com/fatih/structs v1.1.0 // indirect
github.com/felixge/httpsnoop v1.0.3 // indirect
github.com/fluxcd/pkg/apis/kustomize v0.5.0 // indirect
github.com/go-faster/city v1.0.1 // indirect
github.com/go-faster/errors v0.6.1 // indirect
github.com/go-logr/logr v1.2.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-openapi/jsonpointer v0.19.5 // indirect
Expand All @@ -111,7 +114,7 @@ require (
github.com/gorilla/mux v1.8.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.1 // indirect
github.com/hashicorp/go-retryablehttp v0.5.1 // indirect
github.com/hashicorp/go-version v1.4.0 // indirect
github.com/hashicorp/go-version v1.6.0 // indirect
github.com/imdario/mergo v0.3.12 // indirect
github.com/inconshreveable/mousetrap v1.0.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
Expand All @@ -135,13 +138,16 @@ require (
github.com/opencontainers/image-spec v1.0.2 // indirect
github.com/openshift/api v0.0.0-20200221181648-8ce0047d664f // indirect
github.com/openzipkin/zipkin-go v0.4.0 // indirect
github.com/paulmach/orb v0.7.1 // indirect
github.com/pierrec/lz4/v4 v4.1.15 // indirect
github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_model v0.2.0 // indirect
github.com/prometheus/procfs v0.8.0 // indirect
github.com/rs/xid v1.4.0 // indirect
github.com/rs/zerolog v1.20.0 // indirect
github.com/segmentio/asm v1.2.0 // indirect
github.com/shopspring/decimal v1.3.1 // indirect
github.com/sirupsen/logrus v1.9.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
Expand All @@ -153,11 +159,11 @@ require (
github.com/xdg-go/stringprep v1.0.3 // indirect
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect
go.opentelemetry.io/otel/metric v0.31.0 // indirect
go.uber.org/atomic v1.9.0 // indirect
go.uber.org/multierr v1.7.0 // indirect
go.uber.org/atomic v1.10.0 // indirect
go.uber.org/multierr v1.8.0 // indirect
golang.org/x/net v0.0.0-20220826154423-83b083e8dc8b // indirect
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4 // indirect
golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10 // indirect
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab // indirect
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 // indirect
golang.org/x/text v0.3.7 // indirect
golang.org/x/time v0.0.0-20220411224347-583f2d630306 // indirect
Expand Down
Loading

0 comments on commit c6edeb6

Please sign in to comment.