Handles a wait_for_connection call right after a task caused a shutdown

[s1113950]

Fixes #655

wait_for_connection in Ansible calls both a connection.reset() as well as an _execute_module.

In the case of issue 655, a shutdown call right before the wait_for_connection caused Mitogen to disconnect from the host and then be stuck in a bad state when _execute_module was called.

wait_for_connection is set up to retry continually to connect, which was masking errors that Mitogen was throwing when it couldn't set up a connection.

To fix I forced Mitogen to reconnect if the ping module is used with wait_for_connection and made sure that if we're trying to create a Pool that we have a router.

[s1113950]

/aside to travis: oh travis! Please report results!

[s1113950]

Travis does not seem to wanna launch a test 🤦

[s1113950]

New plan:
don't use docker-machine since it seems to not work well with add_host (no docker sock file). Wonder if it's possible to install docker for mac via terminal only. Could also figure out how to connect docker-machine with an add_host-ed docker container setup.

[s1113950]

@cognifloyd can you review this one as well? :) 🙏

[s1113950]

ooh I'll squash commits into 1 actually since there's no current chat history

[cognifloyd]

I don't understand all of the code involved (I haven't been able to grok all of the innards of mitogen so far), but from what I can tell, this looks reasonable.
Just one question about the test playbook.

[cognifloyd]

What prevents this from running on non-mac hosts? I don't see any safety checks to bail if the host is not a mac, instead it will just go up in smoke when the dmg can't be unpacked, or the /Volumes directory doesn't exist.

[s1113950]

The target host only runs on our Mac setup because Mac is the only test that runs localhost_ansible tests (see azure-pipelines.yml). Technically someone could run it themselves on a non-mac and it might not work but the tests are really set up for running through CI mostly so I didn't worry about that. I left a note so that if we end up running localhost_ansible on another host then we'd know to refactor this file a bit. It's not super clear though; if you'd like I could edit the comment to reflect something different!

[cognifloyd]

Nah. The comment as-is is enough. :)

[s1113950]

Sounds good!

[luto]

fixes the reboot module as well. Thank you!