Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka monitoring & connection issues #70

Open
monimail opened this issue May 14, 2018 · 1 comment
Open

Kafka monitoring & connection issues #70

monimail opened this issue May 14, 2018 · 1 comment

Comments

@monimail
Copy link

monimail commented May 14, 2018

Hello,

I am trying to monitor how Kafka behaves in terms of performance.
I tried it with jolokia in the openbmp_aio container as described here : https://github.com/paksu/kafka-jolokia-telegraf-collector but jolokia never sends anything.

As a result, I installed kafka on my host server in order to try to install jolokia more easily, but now I'm getting those log lines :

2018-05-14T11:57:38.613781 | ERROR    | send_bmp_raw       | rtr=xx.xx.xx.xx: Failed to produce bmp raw message: Local: Queue full
2018-05-14T11:57:38.714158 | ERROR    | produce            | rtr=xx.xx.xx.xx: Failed to produce message: Local: Queue full
2018-05-14T11:57:38.814367 | ERROR    | produce            | rtr=xx.xx.xx.xx: Failed to produce message: Local: Queue full
2018-05-14T11:57:38.914610 | ERROR    | send_bmp_raw       | rtr=xx.xx.xx.xx: Failed to produce bmp raw message: Local: Queue full

I am also having issues with the openbmp_kafka container that was not accepting TCP connections from the other containers (as described in SNAS/website#85).

Any help would be appreciated.

Thanks in advance.

@3fr61n
Copy link

3fr61n commented May 30, 2018

For monitoring kafka, I used logstash (with jmx plugin)

input {
    jmx {
        path => "/usr/share/logstash/resources/jmx"
        polling_frequency => 60
        type => "kafka-jmx"
        nb_thread => 4
    }

In the jmx directory I have the folowing json with all beans to be monitored

.... cat kafka.json 
{
  "host" : "kafka",
  "port" : 9999,
  "alias" : "ns",
  "queries" : [
  {
   "object_name" : "kafka.server:type=*,name=*"
  },
  {
   "object_name" : "kafka.server:type=*,name=*,topic=*"
  },
  {
   "object_name" : "kafka.cluster:name=*,partition=*,topic=*,type=*"
  },
  {
   "object_name" : "kafka.log:name=*,partition=*,topic=*,type=*"
  } ]
}

Then you can insert that date in whatever database you like (and logstash support).

Hope this would be useful

Regards

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants