Multiple instances of hubot-slack responding to messages #382
Labels
auto-triage-stale
needs info
An issue that is claimed to be a bug and hasn't been reproduced, or otherwise needs more info
Comments
|
@jivany are you using the hubot reload script? I've seen issues with the hubot reload script and slack that I couldn't explain; I ended up removing the script from my installation. |
|
@iancward not using the reload script. Had this happen again at some point last night and all I see in the logs is that the RTM connection was established: Could this be more like #210 where it's node-slack or even hubot reconnecting because they think the previous connection is "dead"? Anyway to prevent that or at least explicitly close the first connection before attempting to reconnect? |
|
@jivany We have been having the same issue with no particular pattern/timing that we could identify. Just curious, what was the node, hubot and hubot-slack packages versions when you encounter this issue. |
|
@serkanh node 0.10.35; hubot 2.12.0 (?) and hubot-slack 4.2.2 Not 100% certain about the hubot version as I thought my package.json was set to have it pickup the latest version but my logs show 2.12.0. |
|
Same issue here. Hubot is in a docker container,and as soon as I start 2 instances for high availability, he starts answering messages twice. here are the relevant npm versions : |
|
I'm having this same issue when working on my script to have hubot deploy my latest updates to my hubot scripts then reloading the scripts using https://github.com/vinta/hubot-reload-scripts Basically what happens is - I run It sends a POST request to laravel forge which deploys my latest code from master to the server hubot runs on. Which then sends a success message to the channel in slack that hubot is in, when hubot hears
|
|
@Treyone in your case, if you are running two hubot processes this is the expected behavior. i'm not sure if there are any hubot scripts in the ecosystem to tackle this issue, but essentially you need to put all messages in some unified queue and each process needs to pull from that queue to "claim" it. @jivany is this issue still occurring for you? i'm wondering if the reconnection improvements in v3.10.0 of @Kcharle could you also set the |
aoberoi
added
the
needs info
|
@aoberoi I'm on hubot-slack version 4.4.0 and This is problematic for us because we run deploys using hubot, so it spawns multiple deploys at the exact same time when responding. It looks like it does happen due to a reconnect. FYI I have updated my upstart config to include |
|
@aoberoi Update - it happened again and I now have |
|
@aoberoi I'd like to let you know that we're still encountering this from time to time, notably over weekends it seems but it can happen on weekdays as well. Is it safe to say that you have the feedback that you needed? Can someone look at this? |
|
@schuylr thank you so much for reporting all these logs! i'm so sorry that i overlooked these messages for as long as i did. i can see the reconnection occurring in both cases, so that's a helpful piece of evidence. i have a suspicion that the underlying |
aoberoi
removed
the
needs info
|
working on reproducing this one today! |
|
@schuylr @jivany @Kcharle @serkanh i've tried all the ways i know how to reproduce this issue, including forcing network disruptions and using the i'm going to mark this issue as needs feedback until i can find a concrete reproduction. one piece of information that would be helpful is a list of other external scripts and the specific script that was triggered more than once. |
aoberoi
added
the
needs info
|
@aoberoi I'm not sure what to tell you other than it happened again overnight today. I run no other external scripts that I know of other than my custom deploy script, and it's just simple listeners. The service runs on Ubuntu 14.04.5 LTS The service file looks like this My Is there any other info that I can possibly provide? Maybe this is related to some sort of connection reset from Slack's side? |
|
@aoberoi Here's a new log with DEBUG: Notice that the message is received once, but somehow matched twice, as if the script is loaded on top of the existing script. |
|
thanks for the data @schuylr! i have some follow up questions:
just to illustrate my last point, here's a very verbose log taken from my machine while i try to simulate this by closing my network connection and the reconnecting it later. (note: i'm using v4.5.2, and i've turned on logging at the |
|
Regarding the package-lock.json, I'm not sure why hubot is missing from there, but I guess my I will bump to 4.5.2 and see if we still encounter issues (we had another today).
I don't have any reference to something like
I also didn't write any code that would cause a reload as far as I would know. I'm not even sure of the method calls to do this.
Correct, the bot is doing nothing when we're not in the office, and usually we see this happen overnight on the following day. |
|
@schuylr 4.5.3 is the latest now, but thank you! |
|
Has anyone found this issue. I am seeing it 100% of the time. Here is the log: Here is my packags.jason |
|
👋 It looks like this issue has been open for 30 days with no activity. We'll mark this as stale for now, and wait 10 days for an update or for further comment before closing this issue out. If you think this issue needs to be prioritized, please comment to get the thread going again! Maintainers also review issues marked as stale on a regular basis and comment or adjust status if the issue needs to be reprioritized. |
|
As this issue has been inactive for more than one month, we will be closing it. Thank you to all the participants! If you would like to raise a related issue, please create a new issue which includes your specific details and references this issue number. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
I what appears to be a random occurrence, hubot on Slack will start sending multiple responses. Sometimes it is only duplicate responses but worst case seen has resulted in >20 responses generated when a single was expected. The amount of time before this starts happening is variable - sometimes only a few minutes, other times many days.
Similar to #153 but this happens on an already running process with no indications in the logs of re-connections occurring and there is only one node process (hubot) running on the openshift gear.
Is there any way to ensure there is only one RTM connection established?
Reproducible in:
{project_name} version: 4.2.2
OS version(s): openshift.com
Device(s):
Steps to reproduce:
Expected result:
Expect hubot to only reply once.
Actual result:
Sometimes get duplicate responses until hubot is restarted. Worst case so far appears to be >20 responses when only a single response is expected.
Attachments:
Nothing relevant found in logs so far.
The text was updated successfully, but these errors were encountered: