Clearing credentials on disconnect could cause issues

[arthurnn]

On a failover retry, we could lose the credentials as they get cleared after a disconnect.
see

moped/lib/moped/failover/retry.rb

Line 27 in 62af32c

node.disconnect

moped/lib/moped/connection.rb

Line 77 in 62af32c

credentials.clear

cc @jonhyman
[related #286]

[jonhyman]

Thanks. I worked on this a bit on my train today, I'll finish it up when I'm back at a computer in the morning and submit the pulls sometime tomorrow.

[jonhyman]

@edejong can you point your app at https://github.com/jonhyman/moped, branch /feature/fix-credential-clear and see if you still have issues?

[edejong]

Hey sorry for the late response, will try this weekend!

[jonhyman]

Sounds good, thanks!

[edejong]

So far it seems I can't even reproduce the bug with the main repo so I can't say anything sensible yet. I'll leave it overnight and see if it disconnects. It only happened after a long time of inaction so please bear with me ;)

[edejong]

Ok I've been able to reproduce the bug with the old version, it just takes a bit longer for it to occur than it used to which is good to know. Now testing with your branch, let's give it a full day at least.

[edejong]

Unfortunately it still seems to happen using your branch:

failed with error 16540: "not authorized for insert on ..."

[jonhyman]

@edejong any way you can log some more info? I'm a bit confused as to how it would not have logged in. Before, you had said that apply_credentials as credentials == logins now, so it did not ever log in. But I removed all that at https://github.com/mongoid/moped/pull/291/files#diff-804e34c24aec99cb6acc5b0563656272L21 and it should always attempt a login unless it thinks it's already logged in.

[edejong]

@jonhyman will add more info asap.

Strangely I seem to be having a similar problem with mongoid 3+moped 1.5. I'm hosting on ObjectRocket and using rails and unicorn. With mongoid 4 I get the not authorized error after leaving the server running for a long time without traffic. But with mongoid 3 under the same circumstances I get documents not persisting to the database, without errors, in safe mode. If I refresh the page that creates the document a couple of times after a while they slowly start persisting, it seems as if every unicorn process silently fails once before working properly again. In some ways this is even worse as I have no way of knowing an insert succeeded besides manually checking after creation. No idea if it is mongoid or moped yet, will have to test further; it is a pain to test as I have to wait a long time for disconnects. Was the reconnect code in mongoid 2 rewritten? Almost seems to imply something else is going wrong for me...

[jonhyman]

Do you have firewall rules which might be killing persistent connections
and not sending a reset or something like that? Maybe your mongos needs
restarting? What do the ObjectRocket guys think?

Sent from my mobile device
On Jul 15, 2014 4:43 AM, "edejong" notifications@github.com wrote:

@jonhyman https://github.com/jonhyman will add more info asap.Strangely
I seem to be having a similar problem with mongoid 3+moped 1.5. I'm hosting
on ObjectRocket and using rails and unicorn. With mongoid 4 I get the not
authorized error after leaving the server running for a long time without
traffic. But with mongoid 3 I get documents not persisting to the database,
without errors, in safe mode. If I refresh the page that creates the
document a couple of times after a while they slowly start persisting, it
seems as if every unicorn process silently fails once before working
properly again. In some ways this is even worse as I have no way of knowing
an insert succeeded besides manually checking after creation. No

Reply to this email directly or view it on GitHub
#288 (comment).

[arthurnn]

closing this as #324 was merged and it should solve this issue