[Openshift] Pod keeps restarting as the registered watcher says "Config file /config/config.yaml is updated"

[iamvvra]

1st, this project is good. I came here by chance before I wanted to write my own cluster event listener (kind of) to publish events directly to Mattermost.

Though I have deployed this bot in my Openshift cluster running on-premise, I have trouble running it.

The bug description
The deployed pod keeps restarting as the registerd watcher of the config/config.yaml says the file was modified. There is no sign of change from external source or manual.

To Reproduce
I use helm to create botkube in the botkube namespace. My Mattermost too is deployed in Openshift, but in a different namespace, mattermost.
helm install --name botkube --namespace botkube \ --set config.communications.mattermost.enabled=true \ --set config.communications.mattermost.url=http://mattermost.mattermost.svc:8065 \ --set config.communications.mattermost.token=generatedtoken \ --set config.communications.mattermost.team=my-team \ --set config.communications.mattermost.channel=cluster-stat \ --set config.settings.clustername=development-cluster \ --set config.settings.allowkubectl=true \ helm/botkube
(I have also used OCP specific route instead of svc).

Expected behavior
The bot should start with the provided configuration loaded in the /config/config.yaml.

Screenshots
Find a screenshot attached.

Let me know if you need any other detail. Also, tell me, please, if I am doing it right.

[PrasadG193]

Hi @iamvvra ,
Are you using correct Mattermost URL? Because I see Mattermost is unreachable through the logs.

[iamvvra]

Yes, this log has that issue, but I fixed that by providing OCP route url - however, the initial messages reach Mattermost. See the screenshot

Every pod restart send the first message to Mattermost successfully.

[PrasadG193]

Are you still getting config update logs after setting OCP route url?

[iamvvra]

Yes, I get the same error even after I change it to a route, most recent deployment log for your ref. But, was that the cause of this issue?

[PrasadG193]

That's strange. I hope you are using the latest release for deployment. Which cluster you are using? Also is there some process/job which might be updating the configmap? Could you please watch and tell me the status of configmap botkube-configmap?
Could you please post output of @BotKube version?

[iamvvra]

I am using the latest one v0.8.0. However, I tried to run this in my AWS EKS cluster with no restart (though the bot couldnt reach my Mattermost). There is no other jobs or process in this namespace to modify this configmap. As the pod keeps restarting I am attaching the event log.

[PrasadG193]

Could you please send logs of the BotKube pod? Also please make sure that you are using the latest helm chart from v0.8.0 branch while deploying

[iamvvra]

Log attached. I configured from your release branch source.

botkube.log

[PrasadG193]

ERRO[2019-08-06T16:03:31Z] Error in connecting to Mattermost team ngap
Error: : Invalid or expired session, please login again., 

Are you using correct token? Can you please try regenerating token?

[codenio]

Hi,
I guess there are some issues with mattermost latest version.

• Got this same error, when tried to setup mattermost and botkube in my local (few weeks before).
  Checked by generating mutiple tokens(user and bot tokens), but session was expiring instantly.
• got CORS error as well due to client4 model in mattermost-server go pkg. Try adding CORS settings in mattermost config file

So it is good to check with this prespective too, by trying with privous stable version of mattermost.
@iamvvra can you check the Debug logs by setting evn LOG_LEVEL=Debug in the botkube pod to debug this error further.

[iamvvra]

@aananthraj I am running the bot in log level set to debug, also, I have provided one of the log in this thread. Moreover, I got the same error having tried with previous v0.7.0 version.

Having tried with new token today, I get the same error Error connecting to Mattermost, additionally with Segmentation violation - error below.

INFO[2019-08-07T05:07:18Z] Config file /config/config.yaml is updated. Hence restarting the Pod
INFO[2019-08-07T05:07:18Z] Config file /config/config.yaml is updated. Hence restarting the Pod

ERRO[2019-08-07T05:07:18Z] Error in connecting to Mattermost team devs
Error: : Invalid or expired session, please login again.,
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1268edc]

goroutine 96 [running]:
github.com/infracloudio/botkube/pkg/bot.(*mmBot).Start.func1(0xc0006ea080, 0xc0002d9920)
/go/src/github.com/infracloudio/botkube/pkg/bot/mattermost.go:104 +0x5c
created by github.com/infracloudio/botkube/pkg/bot.(*mmBot).Start
/go/src/github.com/infracloudio/botkube/pkg/bot/mattermost.go:101 +0x5c4

[iamvvra]

An update, I find the connectivity to Mattermost is now established, however the pod continues to restart assuming the /config/config.yaml has changed. I will see in my side if any monitoring agents or tools provokes this false change alert.

[PrasadG193]

@iamvvra please don't use the older version of BotKube with newer helm charts. Which version of Mattermost you are using? We have tested it with Mattermost v5.11.1. We are trying to understand if the issue is with the latest version of Mattermost or from client side.

[iamvvra]

[PrasadG193]

Can you please try with v5.11.1+ version?

[iamvvra]

Sure, I'll try that.

[iamvvra]

I have deployed the said Mattermost 5.11.1 app separately. The BotKube reacts the same assuming the config.yaml was modified, keeps restarting the pod. However, the initial notification hits the Mattermost. I am not attaching logs as it was similar to one provided earlier.

Version & the channel the BotKube sends notifications. You can see multiple notifications everytime the pod restarts.

[iamvvra]

Is this have anything to do with Websockets?

[PrasadG193]

@iamvvra I don't think it is a problem related to communication. Could you please verify that any other service is not updating botkube-configmap?

[cyberox]

I'm having the same issue trying to connect Botkube 0.8.0 to slack on OpenShift.
I receive the same messages in slack, so the communication with slack is working. The pod keeps restarting because of the config.yaml change.

[PrasadG193]

I suspect if this is due to Kubernetes release. What version of K8s you are using? @cyberox @iamvvra

[cyberox]

I'm running OpenShift 3.11, based on Kubernetes 1.11.

[iamvvra]

[PrasadG193]

I tried reproducing the issue on K8s v1.11 - minikube, GKE and EKS, couldn't reproduce it. I think it has to do with how Openshift manages resources. I think it updates mounted files for some reason. For now, I am opening another issue to have an ability to disable config file watcher and disable automatic restart on config changed
Edit:
Opened issue: #150

[iamvvra]

Yes, a flag passed (helm too) on to the watcher to disable the auto reboot, default being true.

[iamvvra]

I have tested this fix, it works!