Fix race when stopping chunks memcached client (cortexproject#4511)

* Fix race when stopping chunks memcached client

We need to check again if we have been asked to quit before writing to
the results chan.

Also, the reader of that chan should not close it, because this can
trigger a panic in a writer. The chan will be cleaned up by garbage-
collection once all the readers and writers have exited on quit.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>

* simplify select

Don't open another race where quit is closed just after we check it.

Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
Signed-off-by: Manish Kumar Gupta <manishkg@microsoft.com>

pkg/chunk/cache/memcached.go

@@ -89,7 +89,13 @@ func NewMemcached(cfg MemcachedConfig, client MemcachedClient, name string, reg
 						batchID: input.batchID,
 					}
 					res.found, res.bufs, res.missed = c.fetch(input.ctx, input.keys)
-					input.resultCh <- res
+					// No-one will be reading from resultCh if we were asked to quit
+					// during the fetch, so check again before writing to it.
+					select {
+					case <-c.quit:
+						return
+					case input.resultCh <- res:
+					}
 				}
 			}
 		}()
@@ -214,7 +220,6 @@ loopResults:
 			results[result.batchID] = result
 		}
 	}
-	close(resultsCh)
 
 	for _, result := range results {
 		if result == nil {