Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch partition metricset from client to broker #3029

Merged
merged 6 commits into from
Dec 1, 2016
Merged

Switch partition metricset from client to broker #3029

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
03cb921
Switch paritition metricset from client to broker
ruflin Nov 17, 2016
9559407
Add additional close
ruflin Nov 30, 2016
2490e31
Add retry to metadata queries
Nov 30, 2016
bd46985
kafka broker connection settings
Nov 30, 2016
cae454b
kafka partitions metricset: list topics to be queried
Nov 30, 2016
fb3ef8c
rename error to error.code
Nov 30, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion glide.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ import:
- package: github.com/miekg/dns
version: 5d001d020961ae1c184f9f8152fdc73810481677
- package: github.com/Shopify/sarama
version: fix/sasl-handshake
version: enh/offset-replica-id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking, does this branch contain the sasl fix?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, the branch is directly branched off from sasl-handshake.

repo: https://github.com/urso/sarama
- package: github.com/rcrowley/go-metrics
version: ab2277b1c5d15c3cba104e9cbddbdfc622df5ad8
Expand Down
2 changes: 2 additions & 0 deletions libbeat/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ services:
expose:
- 9092
- 2181
environment:
- ADVERTISED_HOST=kafka

# Overloading kibana with a simple image as it is not needed here
kibana:
Expand Down
21 changes: 21 additions & 0 deletions metricbeat/_meta/beat.full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,27 @@ metricbeat.modules:
#period: 10s
#hosts: ["localhost:9092"]

#client_id: metricbeat

#metadata.retries: 3
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it we should use metadata namespace here, as in the end from a user perspective it is just retrieds. Metadata is like an internal "logic" thing.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense.

#metadata.backoff: 250ms

# List of Topics to query metadata for. If empty, all topics will be queried.
#topics: []

# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

# SASL authentication
#username: ""
#password: ""

#------------------------------- MongoDB Module ------------------------------
#- module: mongodb
Expand Down
59 changes: 57 additions & 2 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1959,15 +1959,70 @@ Oldest offset of the partition.


[float]
=== kafka.partition.partition
== partition Fields

Partition data.



[float]
=== kafka.partition.partition.id

type: long

Partition id.


[float]
=== kafka.partition.topic
=== kafka.partition.partition.leader

type: long

Leader id (broker).


[float]
=== kafka.partition.partition.isr

type: list

List of isr ids.


[float]
=== kafka.partition.partition.replica

type: long

Replica id (broker).


[float]
=== kafka.partition.partition.insync_replica

type: boolean

Indicates if replica is included in the in-sync replicate set (ISR).


[float]
=== kafka.partition.partition.error.code

type: long

Error code from fetching partition.


[float]
=== kafka.partition.topic.error.code

type: long

topic error code.


[float]
=== kafka.partition.topic.name

type: keyword

Expand Down
21 changes: 21 additions & 0 deletions metricbeat/docs/modules/kafka.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,27 @@ metricbeat.modules:
#period: 10s
#hosts: ["localhost:9092"]

#client_id: metricbeat

#metadata.retries: 3
#metadata.backoff: 250ms

# List of Topics to query metadata for. If empty, all topics will be queried.
#topics: []

# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

# SASL authentication
#username: ""
#password: ""
----

[float]
Expand Down
21 changes: 21 additions & 0 deletions metricbeat/metricbeat.full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,27 @@ metricbeat.modules:
#period: 10s
#hosts: ["localhost:9092"]

#client_id: metricbeat

#metadata.retries: 3
#metadata.backoff: 250ms

# List of Topics to query metadata for. If empty, all topics will be queried.
#topics: []

# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

# SASL authentication
#username: ""
#password: ""

#------------------------------- MongoDB Module ------------------------------
#- module: mongodb
Expand Down
39 changes: 35 additions & 4 deletions metricbeat/metricbeat.template-es2x.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -950,12 +950,43 @@
}
},
"partition": {
"type": "long"
"properties": {
"error": {
"properties": {
"code": {
"type": "long"
}
}
},
"id": {
"type": "long"
},
"insync_replica": {
"type": "boolean"
},
"leader": {
"type": "long"
},
"replica": {
"type": "long"
}
}
},
"topic": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
"properties": {
"error": {
"properties": {
"code": {
"type": "long"
}
}
},
"name": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
}
}
}
}
}
Expand Down
37 changes: 34 additions & 3 deletions metricbeat/metricbeat.template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -957,11 +957,42 @@
}
},
"partition": {
"type": "long"
"properties": {
"error": {
"properties": {
"code": {
"type": "long"
}
}
},
"id": {
"type": "long"
},
"insync_replica": {
"type": "boolean"
},
"leader": {
"type": "long"
},
"replica": {
"type": "long"
}
}
},
"topic": {
"ignore_above": 1024,
"type": "keyword"
"properties": {
"error": {
"properties": {
"code": {
"type": "long"
}
}
},
"name": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
}
Expand Down
21 changes: 21 additions & 0 deletions metricbeat/module/kafka/_meta/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,24 @@
#period: 10s
#hosts: ["localhost:9092"]

#client_id: metricbeat
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would remove all tehse config options from the short config.


#metadata.retries: 3
#metadata.backoff: 250ms

# List of Topics to query metadata for. If empty, all topics will be queried.
#topics: []

# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

# SASL authentication
#username: ""
#password: ""
17 changes: 11 additions & 6 deletions metricbeat/module/kafka/partition/_meta/data.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,19 @@
"id": 0
},
"offset": {
"newest": 13,
"newest": 11,
"oldest": 0
},
"partition": 0,
"replicas": [
0
],
"topic": "testtopic"
"partition": {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs update. I will do a PR as soon as this is merged.

"error": 0,
"id": 0,
"insync_replica": true,
"leader": 0,
"replica": 0
},
"topic": {
"name": "test-metricbeat-8760238589576171408"
}
}
},
"metricset": {
Expand Down
41 changes: 39 additions & 2 deletions metricbeat/module/kafka/partition/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,49 @@
type: long
description: >
Oldest offset of the partition.

- name: partition
type: group
description: >
Partition data.
fields:
- name: id
type: long
description: >
Partition id.

- name: leader
type: long
description: >
Leader id (broker).
- name: isr
type: list
description: >
List of isr ids.
- name: replica
type: long
description: >
Replica id (broker).

- name: insync_replica
type: boolean
description: >
Indicates if replica is included in the in-sync replicate set (ISR).

- name: error.code
type: long
description: >
Error code from fetching partition.

- name: topic.error.code
type: long
description: >
Partition id.
- name: topic
topic error code.
- name: topic.name
type: keyword
description: >
Topic name

- name: broker.id
type: long
description: >
Expand All @@ -32,3 +67,5 @@
type: keyword
description: >
Broker address


Loading