Consume CQ results on request timeouts

[dim]

As a follow-up to #3517, I have been able to isolate the actual issue. When timeouts occur during the execution of a CQ, the result channel is never consumed, which causes a deadlock in the query_executor. Because we are stuck in the executor the mapper is never closed and the read-only transaction is never released, locking all subsequent writes.

[corylanou]

Can you write a test that simulates this easily to show that this fixes it? Also, could we just close the channel instead of draining it? Not sure if that will cause a panic if it is still trying to be written to on the other side or not (I didn't read all the code).

[dim]

I tried to write a test but there are quite a few dependencies, which makes it difficult. I have demonstrated the problem in the other ticket. Yes, just closing the channel will cause a panic.

[corylanou]

This service was set up using interfaces to purposefully make it testable. What dependencies specifically are causing the problem for the test? Let me know if I can help. Thanks!

[jwilder]

@corylanou I think this makes sense. If the PointsWriter times out, the in-flight writes are not currently cancelable and will still hold onto the write locks. PointsWriter kicks off each write in a goroutine but we can't interrupt it to release the lock because it's block down in the bolt shard. The timeout causes the PointsWriter to return and then exit early from the CQ query loop but the the writing goroutine holding the lock will still live on. If we drain the results channel, the writing goroutine will be able to acquire the write and complete.

@dim If you could sign the CLA we can get this and #3517 merged.

[dim]

@jwilder done, signed.