Improve expectation API to allow for testing of more intricate scenarios

[wvanbergen]

This makes the API for mock broker expectations a bit more powerful.

• Allow setting a latency for the response (this replaces the previous broker-wide latency setting)
• Allow setting a Before and After callback around the response.

Use these capabilities to add some tests:

• The first client tests asserts that if one broker times out, it will continue to use the second broker to discover metadata.
• The second client test will make sure that eventually NewClient will return an error if all brokers have high latency.
• Test the producer with multiple batches in flight at different retry levels.

@Shopify/kafka

[wvanbergen]

This can now be set separately for every expectation/response.

[wvanbergen]

This now returns itself so the methods below can be chained.

Technically this is a bit shitty, because we immediately add it to the expectations channel, but allow it to be altered later. I guess this could cause concurrency issues, but because of the synchronous way our tests are set up I don't think this will cause issues.

[wvanbergen]

Blegh, the race detector is on to me. I guess this needs fixing after all. API suggestions?

[wvanbergen]

CI is failing due to kisielk/errcheck#62

[wvanbergen]

This is failing on Travis, even though this works fine locally:

--- FAIL: TestSlowCluster (0.12 seconds)
    allocs.go:1: write tcp 127.0.0.1:46446: broken pipe
    allocs.go:1: write tcp 127.0.0.1:47507: broken pipe
    allocs.go:1: write tcp 127.0.0.1:55751: broken pipe

any ideas?

[wvanbergen]

I assume this is caused by the mock broker trying to write a response during/after the reader timeout of the client.

[wvanbergen]

Miraculously, it works by adding a parameter on whether to expect errors to the expectation.

[wvanbergen]

Hmmm, in some cases is the read timeout appears to be triggered before the mock broker has figured out a request is incoming, and so it won't resolve the expectation. Any ideas of making this more robust, compared to simply raising the read timeout value?

[eapache]

why not just a slice?

[wvanbergen]

To make sure the index matches the BrokerID. Broker IDs are 1-indexed, while slice indices start at 0.

[eapache]

This will not be merged in its current form - #312 should be fixed first.