Implement WaitForResult in tests to eliminate race conditions

[tiwilliam]

This is an alternative solution to using sleeps in test. WaitForResult will retry the test call every 10ms and timeout after 5s. A sample test call look like this:

testutil.WaitForResult(func() (bool, error) {
    value, err := SomethingToTest()
    return value == "correct", err
}, func(err error) {
    t.Fatalf("Something went wrong: %v", err)
})

[armon]

I think this is a great way to do it. Great work!

[tiwilliam]

Thanks, yeah it seem to work quite fine on Travis, just some files left to patch.

Ping #131.

[tiwilliam]

I'm gathering strength to deal with the last failing tests, just 8 more I'm aware of.

[armon]

@tiwilliam Awesome work. Thanks so much, it'll be great to have the tests be more reliable.

[armon]

@tiwilliam Actually, I'm going to merge this in now! We can port the tests in master!

[tiwilliam]

Cool!