Data race in subscriber Service.Close and PointsWriter.WritePoints

[jwilder]

==================
WARNING: DATA RACE
Write by goroutine 63:
  runtime.closechan()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/runtime/chan.go:295 +0x0
  github.com/influxdb/influxdb/services/subscriber.(*Service).Close()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/subscriber/service.go:88 +0xb6
  github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Close()
      /Users/jason/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:457 +0x2c9
  github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Close()
      /Users/jason/go/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:137 +0xe6
  main.(*Main).Run.func1()
      /Users/jason/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:99 +0x2e

Previous read by goroutine 47:
  runtime.chansend()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/runtime/chan.go:107 +0x0
  github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints()
      /Users/jason/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:257 +0x66f
  github.com/influxdb/influxdb/services/httpd.(*Handler).serveWriteLine()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:530 +0x1189
  github.com/influxdb/influxdb/services/httpd.(*Handler).serveWrite()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:376 +0x8eb
  github.com/influxdb/influxdb/services/httpd.(*Handler).(github.com/influxdb/influxdb/services/httpd.serveWrite)-fm()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:106 +0x55
  github.com/influxdb/influxdb/services/httpd.authenticate.func1()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:681 +0x80
  net/http.HandlerFunc.ServeHTTP()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/net/http/server.go:1422 +0x47
  github.com/influxdb/influxdb/services/httpd.gzipFilter.func1()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:742 +0x2da
  net/http.HandlerFunc.ServeHTTP()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/net/http/server.go:1422 +0x47
  github.com/influxdb/influxdb/services/httpd.versionHeader.func1()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:751 +0xea
  net/http.HandlerFunc.ServeHTTP()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/net/http/server.go:1422 +0x47
[write] 2015/10/28 13:00:35 write failed for shard 3 on node 1: store is closed
  github.com/influxdb/influxdb/services/httpd.cors.func1()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:784 +0x394
  net/http.HandlerFunc.ServeHTTP()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/net/http/server.go:1422 +0x47
  github.com/influxdb/influxdb/services/httpd.requestID.func1()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:794 +0x1c7
[write] 2015/10/28 13:00:35 write failed for shard 3 on node 1: store is closed
  net/http.HandlerFunc.ServeHTTP()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/net/http/server.go:1422 +0x47
  github.com/influxdb/influxdb/services/httpd.logging.func1()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:802 +0x145
  net/http.HandlerFunc.ServeHTTP()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/net/http/server.go:1422 +0x47
  github.com/influxdb/influxdb/services/httpd.recovery.func1()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:821 +0x1a2
  net/http.HandlerFunc.ServeHTTP()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/net/http/server.go:1422 +0x47
  github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP()
      /Users/jason/go/src/github.com/bmizerany/pat/mux.go:109 +0x372
  github.com/influxdb/influxdb/services/httpd.(*Handler).ServeHTTP()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/handler.go:172 +0x428
  net/http.serverHandler.ServeHTTP()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/net/http/server.go:1862 +0x206
  net/http.(*conn).serve()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/net/http/server.go:1361 +0x117c

Goroutine 63 (running) created at:
  main.(*Main).Run()
      /Users/jason/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:100 +0xed7
  main.main()
      /Users/jason/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:46 +0x3ff

Goroutine 47 (running) created at:
  net/http.(*Server).Serve()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/net/http/server.go:1910 +0x464
  net/http.Serve()
      /private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/net/http/server.go:1765 +0xc5
  github.com/influxdb/influxdb/services/httpd.(*Service).serve()
      /Users/jason/go/src/github.com/influxdb/influxdb/services/httpd/service.go:133 +0xc9

[otoolep]

@nathanielc

[otoolep]

Taking the liberty of assigning to @nathanielc

[jwilder]

To repro:

go install -race ./... && influxd

Then run:

influx_stress -test ./cmd/influx_stress/examples/100M_STD.toml

And CTRL-C the server/restart a few times. Should happens when stopping the server.