Some slack notifications are not coming through

[ananthaa-advisory]

Hi ,

Only some of the Slack notifications are coming through. But NOT all of them. COuld you please help.

I checked that my SLACK is even capable of receiving 100 messages in a second. However, hosts which belong to Passive checks notifications are not coming through,

Under templates.conf, we have as many templates like generic-host,generic-service,passive-checks and so on.

I have added the line - "vars.notify_slack=true" in all the templates in templates.conf file.
I checked the logs under /var/log/icinga2/icinga2.log and notifications checks are notified only for emails and not going for SLACK Channel.

template Service "passive-checks" {
  max_check_attempts = 3
  check_command = "dummy"
  check_interval = 60m
  retry_interval = 30s
  enable_active_checks = true
  vars.dummy_text = "No Passive Check."
  zone = "Yellow"
  vars.notify_slack=true
}

My passive service check configuration for the service:

apply Service "MCS_ALARMS" {
import "passive-checks"
check_command = "dummy"
max_check_attempts = 2
vars.dummy_state = 2
vars.dummy_text = "No Passive Check Result Received from host."
assign where host.name == "gold.sample.com"
assign where host.name == "gold1.sample.com"
assign where host.name == "gold2-sample.com"
}

Note -  this file is declared as a separate file and not in services.conf file. However, they can still be imported as long as they are placed in the ICINGA2 main directory.

Thanks
Aravind

[ananthaa-advisory]

HI @jjethwa ,

By the way, do you what is the difference between this two:

vars.notify_slack=true
and
vars.notify_slack = true ( Here i have space in between two) --> I mentioned like this in my TEST box environment and it is working fine.

And , also in your GITHUB , i dont see a template.conf file present.

[ananthaa-advisory]

Please help , thanks :)

[jjethwa]

Hi @ananthaa-advisory

There should be no functional difference between the two variable assignment statements you posted, but it's better to have the spaces in between the variable name, =, and value.

I have not included the template.conf file as you can add the contents of the slack service/host notification files from this repo to that file or keep them separate. It just depends on your setup. I like to split the configs out as it's easier to manage and maintain in source control.

[ananthaa-advisory]

Thanks.

Also, can we make changes to your slack-service-notification script to a for loop, so that it will write messages one by one to slack channel.and by the way we can avoid rate policy concepts from slack .

I tried today with a while loop and didn't execute anything.:(

[jjethwa]

Hi @ananthaa-advisory

Unfortunately, that's not how notifications work. They are triggered per event as far as I know 😢

[ananthaa-advisory]

HI @jjethwa ,

Need some help.
I need a notification template which would send notification for all the configured service check for a specific slack channel.
I posted a couple of question from Icinga forum and not had a good reply from them.

THey said "Assign where true" parameter would send all the service notification for a specific email id/ slack group. Still not working.

Do you have any idea.

Notifications are coming from Icinga to Slack. however certain of them are missing and there is a mismatch between email notification and the slack notifications .

Here is my template for Slack service

 apply Notification "mail-service-notification-slack" to Service {
   import "slack-service-notification"
   users = [ "slack" ]
   user_groups = [ "slack1" ]
   interval = 6h
  assign where service.name == "true"
   assign where host.vars.notification.mail
  assign where host.vars.notification.mail
  assign where host.vars.notification.pager
  assign where "P1" in service.groups
  assign where host.vars.cluster
  assign where host.vars.os
  assign where host.name
  assign where true
  assign where service.vars.notify_slack == true
 
   period = "24x7"
 }

[jjethwa]

Hi @ananthaa-advisory

That's a really config you have. Is mail-service-notification-slack correct? Then these assignments don't make sense to me:

assign where service.name == "true"
assign where true

Why are these repeated?
assign where host.vars.notification.mail
assign where host.vars.notification.mail

Reduce the number of assign statements

[ananthaa-advisory]

Hi @jjethwa

Just returned from out of town.
Ok, I have changed the template like below. still its not calling the slack template for some of the service checks.

 apply Notification "mail-service-notification-slack" to Service {
   import "slack-service-notification"
   users = [ "slack" ]
   user_groups = [ "slack1" ]
   interval = 6h
   assign where host.vars.notification.mail
  assign where host.name
  assign where service.vars.notify_slack == true
    period = "24x7"
 }

Do I share some Icinga logs where some service checks are calling "mail-service-notification-slack" and some of them are not.

thanks

[jjethwa]

Hi @ananthaa-advisory

Do you need

assign where host.vars.notification.mail
assign where host.name

[ananthaa-advisory]

Hi , Even if I comment these lines. Only some notification are coming through. Thanks, Aravind

…

On Mar 21, 2018 12:54 AM, "Jordan Jethwa" ***@***.***> wrote: Hi @ananthaa-advisory <https://github.com/ananthaa-advisory> Do you need assign where host.vars.notification.mail assign where host.name — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANIRA2dFvnA5G4q2dd4cH4RKIuImPrR9ks5tgVd9gaJpZM4PZ3Ym> .

[jjethwa]

Hi @ananthaa-advisory

Just wanted to simplify the config as much as possible. Can you go through the logs and check the triggered notifications?

[ananthaa-advisory]

HI @jjethwa ,

Thanks for your reply, You are only hope though. Since, In Icinga forum, they are not giving a proper reply.

Here are my logs:

Service where Slack template is being Called out:

[2018-03-22 10:55:55 -0700] information/Checkable: Checking for configured notifications for object 'serverA!Salt_min'
[2018-03-22 10:55:55 -0700] information/Notification: Sending 'Problem' notification 'serverA!Salt_min!mail-service-notification-slack' for user 'slack'
[2018-03-22 10:55:55 -0700] information/Notification: Sending 'Problem' notification 'serverA!Salt_min!groupGroup-service-notification' for user 'group'
[2018-03-22 10:55:55 -0700] information/Notification: Completed sending 'Problem' notification 'serverA!Salt_min!mail-service-notification-slack' for checkable 'serverA!Salt_min' and user 'slack'.
[2018-03-22 10:55:55 -0700] information/Notification: Completed sending 'Problem' notification 'serverA!Salt_min!groupGroup-service-notification' for checkable 'serverA!Salt_min' and user 'group'.

Service where slack template is not being called Out:

[2018-03-22 11:09:49 -0700] information/Notification: Sending 'Recovery' notification 'serverB!Kafka_Brokers!mail-service-notification' for user 'Xaviour'
[2018-03-22 11:09:49 -0700] information/Notification: Sending 'Recovery' notification 'serverB!Kafka_Brokers!mail-service-notification' for user 'Anil'
[2018-03-22 11:09:49 -0700] information/Notification: Sending 'Recovery' notification 'serverB!Kafka_Brokers!mail-service-notification' for user 'Hello123'

[2018-03-22 11:09:49 -0700] information/Notification: Completed sending 'Recovery' notification 'serverB!Kafka_Brokers!mail-service-notification' for checkable 'serverB!Kafka_Brokers' and user 'Xaviour'.
[2018-03-22 11:09:49 -0700] information/Notification: Completed sending 'Recovery' notification 'serverB!Kafka_Brokers!mail-service-notification' for checkable 'serverB!Kafka_Brokers' and user 'slack'.
[2018-03-22 11:09:49 -0700] information/Notification: Completed sending 'Recovery' notification 'serverB!Kafka_Brokers!mail-service-notification' for checkable 'serverB!Kafka_Brokers' and user 'Hello123'.

[jjethwa]

Hi @ananthaa-advisory

I'll try my best! These issues are hard to troubleshoot as configurations are so different from install to install 😛

So the first thing to do is check the host configs between serverA and serverB then check out the service configs for Salt_min and Kafka_Brokers. Make sure to check out what the host and service configs inherit as well.

[ananthaa-advisory]

Hi @jjethwa ,

Please see my service configs below 👍

For kafka_Brokers:

apply Service "Kafka_Brokers" {
  import "generic-service"
  check_command = "Kafka_Brokers"
  command_endpoint = host.name
  assign where host.address == "ServerB"
  assign where host.address == "serverC"
 
}

For Salt_min:

apply Service "Salt_min" {
  import "generic-service"
  check_command = "salt_min"
  command_endpoint = host.name
  assign where host.vars.os == "Linux" && host.vars.appliance != "Big"
}

[jjethwa]

Hi @ananthaa-advisory

For both of those services, you should have notify_slack = true unless it's inherited from something else?

[ananthaa-advisory]

Hi @jjethwa,

Yeah I m calling from generic-service globally. As generic-service is being called by all service checks. Right.

[jjethwa]

Hi @ananthaa-advisory

That should work then 😞 Just as a test, can you try adding notify_slack = true to Kafka brokers?

[ananthaa-advisory]

Sure will do, will share the results .
My doubt is - if I invoke the vars.notify_slack=true, it still calls generic-service template , so it should notify the slack channel.

My strong guess is . There should be bug within Icinga. Have you faced any challenges like this in your enviorment.

[jjethwa]

Yeah, it probably won't work, but it's worth a shot hehe. I have not had a similar issue with mail or slack notifications. You're probably hitting a weird corner case somewhere 😛

[ananthaa-advisory]

Hi Jethwa, Just one more help after a year almost. Hope you are doing good by the way. Have you tried to receive messages from Slack to a Linux server using outgoing Web hook. Thanks, Aravind

…

On Thu, 29 Mar 2018, 12:34 am Jordan Jethwa, ***@***.***> wrote: Yeah, it probably won't work, but it's worth a shot hehe. I have not had a similar issue with mail or slack notifications. You're probably hitting a weird corner case somewhere 😛 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANIRAwVj3JMK4gxzc90FzdoVMBZ4SUL2ks5ti96igaJpZM4PZ3Ym> .

[jjethwa]

Hi @ananthaa-advisory

Thanks, I'm doing good. Keeping busy hehe. How are you? 😄 I have not used outgoing web hooks from Slack, sorry. I've only played with bots.

[ananthaa-advisory]

Oh that's OK Jethwa. Thank you so much for your reply. Just Curious, which country you from.

…

On Wed, 27 Feb 2019, 12:19 am Jordan Jethwa, ***@***.***> wrote: Hi @ananthaa-advisory <https://github.com/ananthaa-advisory> Thanks, I'm doing good. Keeping busy hehe. How are you? 😄 I have not used outgoing web hooks from Slack, sorry. I've only played with bots. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANIRA3aIdueUuwJVxBBCattxVsh9bvURks5vRYG4gaJpZM4PZ3Ym> .

[jjethwa]

I'm from San Francisco, CA, USA 😄