Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[0.9.3rc1] Invalid time stamp in graphite metric causes panic #3785

Closed
pkittenis opened this issue Aug 21, 2015 · 2 comments
Closed

[0.9.3rc1] Invalid time stamp in graphite metric causes panic #3785

pkittenis opened this issue Aug 21, 2015 · 2 comments

Comments

@pkittenis
Copy link

Got the following stack trace from 0.9.3rc1.

Cause is invalid time stamp in graphite metric. Confirmed to happen on latest version from master and 0.9.2 as well.

To replicate:

echo "test 1 `date +%s`1" |nc localhost 2003

Where localhost:2003 is graphite plugin tcp listen port.

Stack trace:

unexpected fault address 0x0
fatal error: fault
[signal 0xb code=0x80 addr=0x0 pc=0x41085b]

goroutine 73310 [running]:
runtime.gothrow(0xb046d0, 0x5)
    /root/.gvm/gos/go1.4.2/src/runtime/panic.go:503 +0x8e fp=0xc2433ca880 sp=0xc2433ca868
runtime.sigpanic()
    /root/.gvm/gos/go1.4.2/src/runtime/sigpanic_unix.go:29 +0x265 fp=0xc2433ca8d0 sp=0xc2433ca880
runtime.mallocgc(0xb0, 0xa35560, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/runtime/malloc.go:183 +0x8ab fp=0xc2433ca980 sp=0xc2433ca8d0
runtime.newobject(0xa35560, 0x410a49)
    /root/.gvm/gos/go1.4.2/src/runtime/malloc.go:353 +0x49 fp=0xc2433ca9a8 sp=0xc2433ca980
encoding/json.Marshal(0x8e25a0, 0xc21552dfe0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/encoding/json/encode.go:134 +0x57 fp=0xc2433ca9e0 sp=0xc2433ca9a8
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).writeSeriesAndFields(0xc22e0f56c0, 0xc21ae886c0, 0xc23bcf6da8, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:410 +0x12f fp=0xc2433caad0 sp=0xc2433ca9e0
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc22e0f56c0, 0xc255257c00, 0x11, 0x20, 0xc21ae886c0, 0xc23bcf6da8, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:231 +0xc4 fp=0xc2433cac20 sp=0xc2433caad0
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc233133880, 0xc255257c00, 0x11, 0x20, 0xc21ae886c0, 0xc23bcf6da8, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:207 +0xc3 fp=0xc2433cacd0 sp=0xc2433cac20
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc24c84c500, 0xc255257c00, 0x11, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:199 +0x3ba fp=0xc2433cae90 sp=0xc2433cacd0
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x48, 0xc255257c00, 0x11, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7 fp=0xc2433caed8 sp=0xc2433cae90
github.com/influxdb/influxdb/cluster.func·002(0x48, 0x1, 0xc255257c00, 0x11, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc fp=0xc2433cafb8 sp=0xc2433caed8
runtime.goexit()
    /root/.gvm/gos/go1.4.2/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2433cafc0 sp=0xc2433cafb8
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 1 [chan receive, 778 minutes]:
main.(*Main).Run(0xc20808fa80, 0xc20800a010, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:91 +0x777
main.main()
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/main.go:42 +0xdc

goroutine 6 [syscall, 778 minutes]:
os/signal.loop()
    /root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:27 +0x35

goroutine 17 [syscall, 778 minutes, locked to thread]:
runtime.goexit()
    /root/.gvm/gos/go1.4.2/src/runtime/asm_amd64.s:2232 +0x1

goroutine 8 [IO wait, 778 minutes]:
net.(*pollDesc).Wait(0xc20800e450, 0x72, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20800e450, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc20800e3f0, 0x0, 0x2b798fb7ce30, 0xc2080fe000)
    /root/.gvm/gos/go1.4.2/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc208036df0, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc208036df0, 0x0, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:244 +0x4c
github.com/influxdb/influxdb/tcp.(*Mux).Serve(0xc2080e2960, 0x2b798fb7f3b0, 0xc208036df0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tcp/mux.go:45 +0x57
created by github.com/influxdb/influxdb/cmd/influxd/run.func·003
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:313 +0x8f2

goroutine 9 [chan receive, 778 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc208036df8, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/meta.(*raftLayer).Accept(0xc2080e2ab0, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/meta/store.go:2058 +0x64
github.com/hashicorp/raft.(*NetworkTransport).listen(0xc20800e4d0)
    /root/go/src/github.com/hashicorp/raft/net_transport.go:346 +0x50
created by github.com/hashicorp/raft.NewNetworkTransport
    /root/go/src/github.com/hashicorp/raft/net_transport.go:138 +0x2c4

goroutine 10 [select]:
github.com/hashicorp/raft.(*Raft).leaderLoop(0xc2080c21c0)
    /root/go/src/github.com/hashicorp/raft/raft.go:815 +0xc7c
github.com/hashicorp/raft.(*Raft).runLeader(0xc2080c21c0)
    /root/go/src/github.com/hashicorp/raft/raft.go:787 +0x74e
github.com/hashicorp/raft.(*Raft).run(0xc2080c21c0)
    /root/go/src/github.com/hashicorp/raft/raft.go:584 +0xc9
github.com/hashicorp/raft.*Raft.(github.com/hashicorp/raft.run)·fm()
    /root/go/src/github.com/hashicorp/raft/raft.go:241 +0x27
github.com/hashicorp/raft.func·011()
    /root/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
    /root/go/src/github.com/hashicorp/raft/state.go:153 +0xe3

goroutine 11 [select, 657 minutes]:
github.com/hashicorp/raft.(*Raft).runFSM(0xc2080c21c0)
    /root/go/src/github.com/hashicorp/raft/raft.go:498 +0xf74
github.com/hashicorp/raft.*Raft.(github.com/hashicorp/raft.runFSM)·fm()
    /root/go/src/github.com/hashicorp/raft/raft.go:242 +0x27
github.com/hashicorp/raft.func·011()
    /root/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
    /root/go/src/github.com/hashicorp/raft/state.go:153 +0xe3

goroutine 12 [select]:
github.com/hashicorp/raft.(*Raft).runSnapshots(0xc2080c21c0)
    /root/go/src/github.com/hashicorp/raft/raft.go:1602 +0x466
github.com/hashicorp/raft.*Raft.(github.com/hashicorp/raft.runSnapshots)·fm()
    /root/go/src/github.com/hashicorp/raft/raft.go:243 +0x27
github.com/hashicorp/raft.func·011()
    /root/go/src/github.com/hashicorp/raft/state.go:152 +0x51
created by github.com/hashicorp/raft.(*raftState).goFunc
    /root/go/src/github.com/hashicorp/raft/state.go:153 +0xe3

goroutine 13 [select, 778 minutes]:
github.com/influxdb/influxdb/meta.(*localRaft).logLeaderChanges(0xc20809cde0)
    /root/go/src/github.com/influxdb/influxdb/meta/state.go:187 +0x46e
created by github.com/influxdb/influxdb/meta.(*localRaft).open
    /root/go/src/github.com/influxdb/influxdb/meta/state.go:177 +0x130c

goroutine 15 [chan receive, 778 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc208036e00, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/meta.(*Store).serveExecListener(0xc20806e000)
    /root/go/src/github.com/influxdb/influxdb/meta/store.go:619 +0x8f
created by github.com/influxdb/influxdb/meta.(*Store).Open
    /root/go/src/github.com/influxdb/influxdb/meta/store.go:239 +0x391

goroutine 16 [chan receive, 778 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc208036e08, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/meta.(*Store).serveRPCListener(0xc20806e000)
    /root/go/src/github.com/influxdb/influxdb/meta/store.go:723 +0xc9
created by github.com/influxdb/influxdb/meta.(*Store).Open
    /root/go/src/github.com/influxdb/influxdb/meta/store.go:242 +0x3db

goroutine 20 [select]:
github.com/influxdb/influxdb/services/hh.(*Service).retryWrites(0xc208055360)
    /root/go/src/github.com/influxdb/influxdb/services/hh/service.go:103 +0x2e7
created by github.com/influxdb/influxdb/services/hh.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/hh/service.go:67 +0x20e

goroutine 19 [select, 778 minutes]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).monitorErrorChan(0xc2080b20e0, 0xc20809c4e0)
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:442 +0x14a
created by github.com/influxdb/influxdb/cmd/influxd/run.func·003
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:319 +0xa63

goroutine 21 [select, 56 minutes]:
github.com/influxdb/influxdb/services/hh.(*Service).expireWrites(0xc208055360)
    /root/go/src/github.com/influxdb/influxdb/services/hh/service.go:121 +0x2a4
created by github.com/influxdb/influxdb/services/hh.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/hh/service.go:68 +0x228

goroutine 22 [chan receive, 778 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc208036e10, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/cluster.(*Service).serve(0xc20800e2a0)
    /root/go/src/github.com/influxdb/influxdb/cluster/service.go:83 +0x102
created by github.com/influxdb/influxdb/cluster.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/cluster/service.go:60 +0x138

goroutine 23 [select, 8 minutes]:
github.com/influxdb/influxdb/services/precreator.(*Service).runPrecreation(0xc20800adc0)
    /root/go/src/github.com/influxdb/influxdb/services/precreator/service.go:75 +0x407
created by github.com/influxdb/influxdb/services/precreator.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/precreator/service.go:53 +0x20a

goroutine 24 [chan receive, 778 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc208036e18, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x80
github.com/influxdb/influxdb/services/snapshotter.(*Service).serve(0xc20800aeb0)
    /root/go/src/github.com/influxdb/influxdb/services/snapshotter/service.go:75 +0xad
created by github.com/influxdb/influxdb/services/snapshotter.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/snapshotter/service.go:48 +0x133

goroutine 25 [IO wait, 778 minutes]:
net.(*pollDesc).Wait(0xc20815e300, 0x72, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20815e300, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc20815e2a0, 0x0, 0x2b798fb7ce30, 0xc2080dc2f0)
    /root/.gvm/gos/go1.4.2/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc208154030, 0xf02120, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc208154030, 0x0, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:244 +0x4c
net/http.(*Server).Serve(0xc20809c540, 0x2b798fb7f3b0, 0xc208154030, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1728 +0x92
net/http.Serve(0x2b798fb7f3b0, 0xc208154030, 0x2b798fb83158, 0xc2080dc2e0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1606 +0xa1
github.com/influxdb/influxdb/services/admin.(*Service).serve(0xc20800af50)
    /root/go/src/github.com/influxdb/influxdb/services/admin/service.go:107 +0x147
created by github.com/influxdb/influxdb/services/admin.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/admin/service.go:70 +0x472

goroutine 26 [select]:
github.com/influxdb/influxdb/services/continuous_querier.(*Service).backgroundLoop(0xc20800e310)
    /root/go/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:157 +0x3e9
created by github.com/influxdb/influxdb/services/continuous_querier.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:93 +0x2df

goroutine 27 [IO wait, 530 minutes]:
net.(*pollDesc).Wait(0xc20815e370, 0x72, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20815e370, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc20815e310, 0x0, 0x2b798fb7ce30, 0xc2559982d8)
    /root/.gvm/gos/go1.4.2/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc208154038, 0xc208180dc8, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc208154038, 0x0, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/tcpsock_posix.go:244 +0x4c
net/http.(*Server).Serve(0xc2080e0000, 0x2b798fb7f3b0, 0xc208154038, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1728 +0x92
net/http.Serve(0x2b798fb7f3b0, 0xc208154038, 0x2b798fb83108, 0xc20800e380, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/net/http/server.go:1606 +0xa1
github.com/influxdb/influxdb/services/httpd.(*Service).serve(0xc20800b090)
    /root/go/src/github.com/influxdb/influxdb/services/httpd/service.go:108 +0x79
created by github.com/influxdb/influxdb/services/httpd.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/httpd/service.go:76 +0x68b

goroutine 28 [select, 8 minutes]:
github.com/influxdb/influxdb/services/retention.(*Service).deleteShardGroups(0xc20809ca20)
    /root/go/src/github.com/influxdb/influxdb/services/retention/service.go:69 +0x342
created by github.com/influxdb/influxdb/services/retention.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/retention/service.go:45 +0x138

goroutine 29 [select, 8 minutes]:
github.com/influxdb/influxdb/services/retention.(*Service).deleteShards(0xc20809ca20)
    /root/go/src/github.com/influxdb/influxdb/services/retention/service.go:102 +0x71b
created by github.com/influxdb/influxdb/services/retention.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/retention/service.go:46 +0x152

goroutine 34 [runnable]:
time.NewTimer(0x3b9aca00, 0xc271654000)
    /root/.gvm/gos/go1.4.2/src/time/sleep.go:66
github.com/influxdb/influxdb/tsdb.func·002()
    /root/go/src/github.com/influxdb/influxdb/tsdb/batcher.go:81 +0x278
created by github.com/influxdb/influxdb/tsdb.(*PointBatcher).Start
    /root/go/src/github.com/influxdb/influxdb/tsdb/batcher.go:104 +0x29a

goroutine 35 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints(0xc20800e230, 0xc255ee3200, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:210 +0x634
github.com/influxdb/influxdb/services/graphite.(*Service).processBatches(0xc2080720c0, 0xc208156600)
    /root/go/src/github.com/influxdb/influxdb/services/graphite/service.go:260 +0x239
created by github.com/influxdb/influxdb/services/graphite.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/graphite/service.go:105 +0x708

goroutine 36 [chan send]:
github.com/influxdb/influxdb/services/graphite.(*Service).handleLine(0xc2080720c0, 0xc255ee3e40, 0x35)
    /root/go/src/github.com/influxdb/influxdb/services/graphite/service.go:246 +0x2a7
github.com/influxdb/influxdb/services/graphite.func·003()
    /root/go/src/github.com/influxdb/influxdb/services/graphite/service.go:219 +0x1f7
created by github.com/influxdb/influxdb/services/graphite.(*Service).openUDPServer
    /root/go/src/github.com/influxdb/influxdb/services/graphite/service.go:222 +0x2fa

goroutine 37 [select, 778 minutes]:
github.com/influxdb/influxdb/cmd/influxd/run.(*Command).monitorServerErrors(0xc2080b4300)
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:140 +0x2da
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run
    /root/go/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:122 +0xdef

goroutine 5834 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc20998f180, 0xc21eeef2c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 70 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc2080b21c0, 0xc2083dd320)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73249 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc230714d00, 0xc22e053f30, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x5b, 0xc22e053f30, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x5b, 0x1, 0xc22e053f30, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73251 [syscall]:
syscall.Syscall(0x4a, 0x68, 0x0, 0x0, 0xf0a888, 0xc26a0cf968, 0x2b798fb7ce30)
    /root/.gvm/gos/go1.4.2/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.Fsync(0x68, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/syscall/zsyscall_linux_amd64.go:503 +0x55
os.(*File).Sync(0xc2204786d8, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/os/file_posix.go:129 +0x6a
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).writeSeriesAndFields(0xc234be97a0, 0xc219d77a70, 0xc21ef5bc80, 0xf, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:427 +0x361
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc234be97a0, 0xc2159b8800, 0xf, 0x10, 0xc219d77a70, 0xc21ef5bc80, 0xf, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:231 +0xc4
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc249b48c40, 0xc2159b8800, 0xf, 0x10, 0xc219d77a70, 0xc21ef5bc80, 0xf, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:207 +0xc3
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc25544ab00, 0xc2159b8800, 0xf, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:199 +0x3ba
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x4a, 0xc2159b8800, 0xf, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x4a, 0x1, 0xc2159b8800, 0xf, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 43 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc2080b2460, 0xc2085a82a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73215 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc21c0bf7e0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2159b8900, 0xc, 0x10, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc21c0bf7e0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2159b8900, 0xc, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 848 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc2080b3500, 0xc20ff8d320)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73217 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1980, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc255ee3a00, 0x4, 0x4, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1980, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc255ee3a00, 0x4, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73278 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1b80, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e038070, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1b80, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e038070, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 347 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc20998e460, 0xc20c4f5f20)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73297 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc21c0be0a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc255ee3540, 0x4, 0x4, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc21c0be0a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc255ee3540, 0x4, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9892 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee627e0, 0xc23cd7e240)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 72707 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22d007340, 0xc24b3b0780)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9700 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be8fc0, 0xc244e12f60)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 71390 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee63500, 0xc24029dc80)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 10019 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62c40, 0xc236492de0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9706 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62540, 0xc244e13560)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9998 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9da40, 0xc237a83320)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73303 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc24c84ca00, 0xc22e053ee0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x3d, 0xc22e053ee0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x3d, 0x1, 0xc22e053ee0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73277 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b19a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e053fc0, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b19a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e053fc0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73329 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23a67b300, 0xc22e053f50, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x3e, 0xc22e053f50, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x3e, 0x1, 0xc22e053f50, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 10045 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee63340, 0xc21390f7a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 71446 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc2080b2d20, 0xc26229ff80)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73302 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc24c84c700, 0xc22fcb3800, 0x21, 0x40, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x1b, 0xc22fcb3800, 0x21, 0x40, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x1b, 0x1, 0xc22fcb3800, 0x21, 0x40)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9690 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22e0f5960, 0xc24f91d860)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73313 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc24c84d500, 0xc2249b2a00, 0x15, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x2e, 0xc2249b2a00, 0x15, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x2e, 0x1, 0xc2249b2a00, 0x15, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 3953 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc2080b2540, 0xc22e9e6780)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9898 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee628c0, 0xc23cd7e540)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73321 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1b60, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc255ee3a40, 0x3, 0x4, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1b60, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc255ee3a40, 0x3, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73309 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23b524b00, 0xc22e0529f0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x35, 0xc22e0529f0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x35, 0x1, 0xc22e0529f0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73283 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1640, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc255ee3940, 0x4, 0x4, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1640, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc255ee3940, 0x4, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73273 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc221ffbd40, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc255257a00, 0x15, 0x20, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc221ffbd40, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc255257a00, 0x15, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 71939 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee63960, 0xc2311598c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9708 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22e0f5b20, 0xc23d066e40)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73293 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1720, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e053ef0, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1720, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e053ef0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9499 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc2080b28c0, 0xc2421eafc0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 10031 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62e00, 0xc236493620)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73237 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23b524900, 0xc255ee3280, 0x3, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x46, 0xc255ee3280, 0x3, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x46, 0x1, 0xc255ee3280, 0x3, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9673 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9c2a0, 0xc24f3d1c80)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 10065 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee63260, 0xc230324840)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73242 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc214803b00, 0xc255257a00, 0x15, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x36, 0xc255257a00, 0x15, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x36, 0x1, 0xc255257a00, 0x15, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9904 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee629a0, 0xc23cd7f6e0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9959 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9960, 0xc2383e08a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 10109 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc2346a4000, 0xc24ebf22a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73301 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23a67ac00, 0xc2159b8b00, 0xd, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x26, 0xc2159b8b00, 0xd, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x26, 0x1, 0xc2159b8b00, 0xd, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73320 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1ac0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2221b1ae0, 0x2, 0x2, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1ac0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2221b1ae0, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73213 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc21c0bef20, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2159b8700, 0x10, 0x10, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc21c0bef20, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2159b8700, 0x10, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73296 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2156cb760, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc213801180, 0x6, 0x8, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2156cb760, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc213801180, 0x6, 0x8)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73247 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc2268bbe00, 0xc2249b3000, 0x1b, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x27, 0xc2249b3000, 0x1b, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x27, 0x1, 0xc2249b3000, 0x1b, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9930 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9d260, 0xc2195cede0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9965 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9a40, 0xc2383e1a40)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73288 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc214a92ba0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2249b2c00, 0x1a, 0x20, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc214a92ba0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2249b2c00, 0x1a, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73324 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b14a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc255ee3900, 0x4, 0x4, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b14a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc255ee3900, 0x4, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73333 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23b524d00, 0xc22fcb3400, 0x23, 0x40, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x4c, 0xc22fcb3400, 0x23, 0x40, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x4c, 0x1, 0xc22fcb3400, 0x23, 0x40)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73322 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2258d20a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22fcb3000, 0x22, 0x40, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2258d20a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22fcb3000, 0x22, 0x40)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 71428 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc2080b2e00, 0xc23e14de00)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9612 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9c0e0, 0xc24f3d11a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 8031 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc20998eee0, 0xc2390c4540)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73266 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2258d3a20, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc24b7c4000, 0x3a9, 0x400, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2258d3a20, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc24b7c4000, 0x3a9, 0x400)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 10025 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62d20, 0xc236493080)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73312 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc25544b800, 0xc255ee3a40, 0x3, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x61, 0xc255ee3a40, 0x3, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x61, 0x1, 0xc255ee3a40, 0x3, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73295 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2156ca240, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2249b2400, 0x1d, 0x20, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2156ca240, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2249b2400, 0x1d, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73233 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc21550e600, 0xc22e053e70, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x44, 0xc22e053e70, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x44, 0x1, 0xc22e053e70, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 10103 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9ea0, 0xc23391dec0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 71373 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee621c0, 0xc2508dff80)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9985 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62fc0, 0xc23b0b6540)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73284 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b16c0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2221b16e0, 0x2, 0x2, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b16c0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2221b16e0, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 71434 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9dea0, 0xc23acd40c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 71940 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be87e0, 0xc2311599e0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9641 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22e0f57a0, 0xc244dd3380)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73241 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc2080a5400, 0xc22e0380b0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x76, 0xc22e0380b0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x76, 0x1, 0xc22e0380b0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9697 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9c620, 0xc23d0a9020)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9792 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22c3aa0e0, 0xc23cf6c300)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9865 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9d0a0, 0xc23cf4d0e0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9691 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22e0f5a40, 0xc23d0a8c00)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73318 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc221ffb1c0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e0529f0, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc221ffb1c0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e0529f0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73276 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1520, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e053e40, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1520, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e053e40, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9811 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9cd20, 0xc23cf7df80)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9759 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9180, 0xc23d0a7500)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9764 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9ca80, 0xc23d054720)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9781 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9cb60, 0xc23d0a7860)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9721 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9c8c0, 0xc23d05c120)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9846 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be95e0, 0xc23cf5b920)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9770 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9cc40, 0xc23d054a80)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9852 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be96c0, 0xc23cf5bc80)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73306 [syscall]:
syscall.Syscall(0x4a, 0x80, 0x0, 0x0, 0xf0a888, 0xc28a503968, 0x2b798fb7ce30)
    /root/.gvm/gos/go1.4.2/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.Fsync(0x80, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/syscall/zsyscall_linux_amd64.go:503 +0x55
os.(*File).Sync(0xc220478868, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/os/file_posix.go:129 +0x6a
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).writeSeriesAndFields(0xc22c3aa2a0, 0xc20f96ea80, 0xc22a126fc0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:427 +0x361
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc22c3aa2a0, 0xc2249b2400, 0x1d, 0x20, 0xc20f96ea80, 0xc22a126fc0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:231 +0xc4
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc24d0d62c0, 0xc2249b2400, 0x1d, 0x20, 0xc20f96ea80, 0xc22a126fc0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:207 +0xc3
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc24c84dd00, 0xc2249b2400, 0x1d, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:199 +0x3ba
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x1f, 0xc2249b2400, 0x1d, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x1f, 0x1, 0xc2249b2400, 0x1d, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 71968 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be8e00, 0xc2394906c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 10032 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22c3aa460, 0xc23b0f4840)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73328 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc25544a600, 0xc2221b1ae0, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x15, 0xc2221b1ae0, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x15, 0x1, 0xc2221b1ae0, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9874 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9cee0, 0xc23cf4a000)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9910 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62a80, 0xc23cd7f9e0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9880 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9cfc0, 0xc23cf4a3c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9999 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9ddc0, 0xc237a834a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73286 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1a80, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e038010, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1a80, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e038010, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73272 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1c20, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e0380b0, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1c20, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e0380b0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9822 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9880, 0xc23cd97e00)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 10059 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee63180, 0xc2303243c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9984 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9d960, 0xc23b2bc5a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9987 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9b20, 0xc23630e2a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 10007 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22c3aa2a0, 0xc237d859e0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73230 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23a67a800, 0xc2221b1be0, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x72, 0xc2221b1be0, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x72, 0x1, 0xc2221b1be0, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 71232 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9dce0, 0xc26229fce0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9976 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9d6c0, 0xc22c3ca720)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9948 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9d500, 0xc22c3cb500)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9970 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9d5e0, 0xc22c3ca3c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9936 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9d340, 0xc2195cf0e0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9942 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9d420, 0xc2195cf560)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73334 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc24c84cb00, 0xc213801200, 0x6, 0x8, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x13, 0xc213801200, 0x6, 0x8, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x13, 0x1, 0xc213801200, 0x6, 0x8)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73314 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc21c0be4c0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2249b2800, 0x1c, 0x20, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc21c0be4c0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2249b2800, 0x1c, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73223 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23b524500, 0xc2159b8c00, 0xf, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x2f, 0xc2159b8c00, 0xf, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x2f, 0x1, 0xc2159b8c00, 0xf, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9798 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9340, 0xc23d055560)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73271 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1a00, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e053fe0, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1a00, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e053fe0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73290 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc214a937a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22fcb3800, 0x21, 0x40, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc214a937a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22fcb3800, 0x21, 0x40)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73319 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221a75a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc255257c00, 0x11, 0x20, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221a75a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc255257c00, 0x11, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 71933 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee63880, 0xc231159680)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 10131 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee63420, 0xc2214cf0e0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9886 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62b60, 0xc2195ce240)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73226 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23a67bb00, 0xc255ee39c0, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x50, 0xc255ee39c0, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x50, 0x1, 0xc255ee39c0, 0x4, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9667 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9c1c0, 0xc24f3d1860)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73222 [syscall]:
syscall.Syscall(0x4a, 0x1b, 0x0, 0x0, 0xf0a888, 0xc27bd0d968, 0x2b798fb7ce30)
    /root/.gvm/gos/go1.4.2/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.Fsync(0x1b, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/syscall/zsyscall_linux_amd64.go:503 +0x55
os.(*File).Sync(0xc208036158, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/os/file_posix.go:129 +0x6a
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).writeSeriesAndFields(0xc20998eee0, 0xc226fefa40, 0xc23bcf6d78, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:427 +0x361
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc20998eee0, 0xc2159b8900, 0xc, 0x10, 0xc226fefa40, 0xc23bcf6d78, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:231 +0xc4
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc21e1ad900, 0xc2159b8900, 0xc, 0x10, 0xc226fefa40, 0xc23bcf6d78, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:207 +0xc3
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23ef57b00, 0xc2159b8900, 0xc, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:199 +0x3ba
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x45, 0xc2159b8900, 0xc, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x45, 0x1, 0xc2159b8900, 0xc, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73325 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1840, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e053f50, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1840, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e053f50, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9747 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be90a0, 0xc23d0a6ba0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 10085 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9c00, 0xc226e420c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 71351 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be8460, 0xc247a85da0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 4703 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc20998efc0, 0xc21be751a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73304 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc214803500, 0xc2249b2800, 0x1c, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x25, 0xc2249b2800, 0x1c, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x25, 0x1, 0xc2249b2800, 0x1c, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73216 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc214a92280, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2159b8c00, 0xf, 0x10, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc214a92280, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2159b8c00, 0xf, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73316 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b14e0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2221b1500, 0x2, 0x2, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b14e0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2221b1500, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9741 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9c7e0, 0xc23d0a6600)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73267 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc221ffb2a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc255ee3280, 0x3, 0x4, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc221ffb2a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc255ee3280, 0x3, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73294 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2198e1900, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22fcb2c00, 0x32, 0x40, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2198e1900, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22fcb2c00, 0x32, 0x40)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73269 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1800, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc255ee3980, 0x3, 0x4, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1800, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc255ee3980, 0x3, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9339 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22d006620, 0xc24f917f80)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73224 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc216877200, 0xc2221b1580, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x10, 0xc2221b1580, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x10, 0x1, 0xc2221b1580, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9662 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62380, 0xc244e12a20)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9627 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22e0f4fc0, 0xc2226b83c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 71917 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee63a40, 0xc2416eb920)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 10097 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9dc0, 0xc226e42a20)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9859 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9ce00, 0xc23cf4cd20)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 71367 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9dc00, 0xc251aec420)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 8716 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22d0068c0, 0xc24e76c840)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9916 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9d880, 0xc2505f83c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73225 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc238c68100, 0xc22e053fe0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x58, 0xc22e053fe0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x58, 0x1, 0xc22e053fe0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73245 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23a67a300, 0xc2249b2c00, 0x1a, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x3b, 0xc2249b2c00, 0x1a, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x3b, 0x1, 0xc2249b2c00, 0x1a, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 71352 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be8540, 0xc251902a20)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9810 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9420, 0xc23d0674a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 71955 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be89a0, 0xc2416ebf80)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73317 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1820, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e053f30, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1820, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e053f30, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73238 [syscall]:
syscall.Syscall(0x4a, 0xf, 0x0, 0x0, 0xf0a888, 0xc2728cb830, 0x2b798fb7ce30)
    /root/.gvm/gos/go1.4.2/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.Fsync(0xf, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/syscall/zsyscall_linux_amd64.go:503 +0x55
os.(*File).Sync(0xc220478038, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/os/file_posix.go:129 +0x6a
github.com/influxdb/influxdb/tsdb/engine/wal.(*Partition).Write(0xc2080c49c0, 0xc218fa3000, 0x3e, 0x40, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:797 +0x8ca
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc2080b21c0, 0xc24b7c4000, 0x3a9, 0x400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:241 +0x379
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc2080fcb00, 0xc24b7c4000, 0x3a9, 0x400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:207 +0xc3
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc2080a4500, 0xc24b7c4000, 0x3a9, 0x400, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:199 +0x3ba
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x1, 0xc24b7c4000, 0x3a9, 0x400, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x1, 0x1, 0xc24b7c4000, 0x3a9, 0x400)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73274 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221a7800, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc213800e80, 0x7, 0x8, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221a7800, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc213800e80, 0x7, 0x8)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73240 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc20f9bbc00, 0xc213800e80, 0x7, 0x8, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x1a, 0xc213800e80, 0x7, 0x8, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x1a, 0x1, 0xc213800e80, 0x7, 0x8)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9753 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9c9a0, 0xc23d0a6fc0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73228 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc25544a100, 0xc22fcb3c00, 0x23, 0x40, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x3a, 0xc22fcb3c00, 0x23, 0x40, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x3a, 0x1, 0xc22fcb3c00, 0x23, 0x40)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73270 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b18e0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc255ee39c0, 0x4, 0x4, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b18e0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc255ee39c0, 0x4, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9635 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9c460, 0xc244dd3080)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73248 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc230714c00, 0xc22e053ef0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x56, 0xc22e053ef0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x56, 0x1, 0xc22e053ef0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73231 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc25544bb00, 0xc2159b8600, 0x10, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x49, 0xc2159b8600, 0x10, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x49, 0x1, 0xc2159b8600, 0x10, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 71520 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc2080b2ee0, 0xc2103eb8c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73323 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2258d3ca0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22fcb3400, 0x23, 0x40, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2258d3ca0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22fcb3400, 0x23, 0x40)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73246 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23a67a600, 0xc2249b2600, 0x1a, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x22, 0xc2249b2600, 0x1a, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x22, 0x1, 0xc2249b2600, 0x1a, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9613 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee622a0, 0xc24f3d1380)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9537 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62000, 0xc22d894e40)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73268 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1560, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2221b1580, 0x2, 0x2, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1560, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2221b1580, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73279 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1bc0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2221b1be0, 0x2, 0x2, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1bc0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2221b1be0, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9917 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22c3aa380, 0xc236492a20)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9656 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22e0f5880, 0xc244e124e0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73282 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b15a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e053e70, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b15a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e053e70, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73308 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc214803700, 0xc255ee3540, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x43, 0xc255ee3540, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x43, 0x1, 0xc255ee3540, 0x4, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73252 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23b525600, 0xc2159b8700, 0x10, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x29, 0xc2159b8700, 0x10, 0x10, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x29, 0x1, 0xc2159b8700, 0x10, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9171 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc20998e540, 0xc24e6c7560)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73298 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23ef56700, 0xc255ee3940, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x28, 0xc255ee3940, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x28, 0x1, 0xc255ee3940, 0x4, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9924 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9d180, 0xc2195ce4e0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 71409 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be8620, 0xc2550d1620)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9799 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62700, 0xc23cf4c540)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73243 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc2384e0400, 0xc22e038070, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x6c, 0xc22e038070, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x6c, 0x1, 0xc22e038070, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9447 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22d0070a0, 0xc223dd1b60)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73291 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc214a93f20, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2249b3000, 0x1b, 0x20, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc214a93f20, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2249b3000, 0x1b, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73280 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc21c0bed00, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2159b8600, 0x10, 0x10, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc21c0bed00, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2159b8600, 0x10, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73330 [syscall]:
syscall.Syscall(0x4a, 0x47, 0x0, 0x0, 0xf0a888, 0xc28e3cd968, 0x2b798fb7ce30)
    /root/.gvm/gos/go1.4.2/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.Fsync(0x47, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/syscall/zsyscall_linux_amd64.go:503 +0x55
os.(*File).Sync(0xc208036a80, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/os/file_posix.go:129 +0x6a
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).writeSeriesAndFields(0xc234be90a0, 0xc21a10d170, 0xc23bcf6d88, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:427 +0x361
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc234be90a0, 0xc2249b3600, 0x1b, 0x20, 0xc21a10d170, 0xc23bcf6d88, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:231 +0xc4
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc23600e880, 0xc2249b3600, 0x1b, 0x20, 0xc21a10d170, 0xc23bcf6d88, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:207 +0xc3
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc245615c00, 0xc2249b3600, 0x1b, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:199 +0x3ba
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x14, 0xc2249b3600, 0x1b, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x14, 0x1, 0xc2249b3600, 0x1b, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73307 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc24c84c900, 0xc213801180, 0x6, 0x8, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x33, 0xc213801180, 0x6, 0x8, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x33, 0x1, 0xc213801180, 0x6, 0x8)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 10091 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9ce0, 0xc226e425a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73326 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc21c0be000, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc213801200, 0x6, 0x8, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc21c0be000, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc213801200, 0x6, 0x8)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9835 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9d7a0, 0xc237d84540)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73292 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1700, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22e053ee0, 0x1, 0x1, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1700, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22e053ee0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73287 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2156cb140, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2249b2600, 0x1a, 0x20, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2156cb140, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2249b2600, 0x1a, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9735 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9c700, 0xc23d0a62a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73221 [runnable]:
syscall.Syscall(0x4a, 0x58, 0x0, 0x0, 0x0, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.Fsync(0x58, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/syscall/zsyscall_linux_amd64.go:503 +0x55
os.(*File).Sync(0xc220478670, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/os/file_posix.go:129 +0x6a
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).writeSeriesAndFields(0xc23ba9cd20, 0xc213040b10, 0xc215bc8100, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:427 +0x361
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc23ba9cd20, 0xc255ee3a00, 0x4, 0x4, 0xc213040b10, 0xc215bc8100, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:231 +0xc4
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc237a7db00, 0xc255ee3a00, 0x4, 0x4, 0xc213040b10, 0xc215bc8100, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:207 +0xc3
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23b525300, 0xc255ee3a00, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:199 +0x3ba
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x3f, 0xc255ee3a00, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x3f, 0x1, 0xc255ee3a00, 0x4, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9805 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9500, 0xc23cf4c7e0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73311 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23b524e00, 0xc22fcb3000, 0x22, 0x40, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x12, 0xc22fcb3000, 0x22, 0x40, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x12, 0x1, 0xc22fcb3000, 0x22, 0x40)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73332 [runnable]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc25544bf00, 0xc255ee3900, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x4d, 0xc255ee3900, 0x4, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x4d, 0x1, 0xc255ee3900, 0x4, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73285 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b1a40, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2221b1a60, 0x2, 0x2, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b1a40, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2221b1a60, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 9866 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be97a0, 0xc23cf4d2c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73315 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b13c0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2249b3600, 0x1b, 0x20, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b13c0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2249b3600, 0x1b, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73227 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc238c68200, 0xc22e053fc0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x55, 0xc22e053fc0, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x55, 0x1, 0xc22e053fc0, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 10073 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22c3aa540, 0xc2385b6de0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9586 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be88c0, 0xc23deba420)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9594 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62460, 0xc244ddbbc0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73244 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23b525f00, 0xc22e038010, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x57, 0xc22e038010, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x57, 0x1, 0xc22e038010, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9679 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc23ba9c380, 0xc24f91c060)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 7157 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc20998e000, 0xc24bb21b00)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73331 [syscall]:
syscall.Syscall(0x4a, 0x2c, 0x0, 0x0, 0xf0a888, 0xc289ffb968, 0x2b798fb7ce30)
    /root/.gvm/gos/go1.4.2/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.Fsync(0x2c, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/syscall/zsyscall_linux_amd64.go:503 +0x55
os.(*File).Sync(0xc220478598, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/os/file_posix.go:129 +0x6a
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).writeSeriesAndFields(0xc22ee622a0, 0xc21ae88360, 0xc23bcf6d98, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:427 +0x361
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc22ee622a0, 0xc2221b1500, 0x2, 0x2, 0xc21ae88360, 0xc23bcf6d98, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:231 +0xc4
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc233122900, 0xc2221b1500, 0x2, 0x2, 0xc21ae88360, 0xc23bcf6d98, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:207 +0xc3
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc254147f00, 0xc2221b1500, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:199 +0x3ba
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x20, 0xc2221b1500, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x20, 0x1, 0xc2221b1500, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73229 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc230714800, 0xc22e053e40, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x5a, 0xc22e053e40, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x5a, 0x1, 0xc22e053e40, 0x1, 0x1)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73281 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b04a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2249b3200, 0x1a, 0x20, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b04a0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2249b3200, 0x1a, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 71887 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee63b20, 0xc2896d8960)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 10034 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62ee0, 0xc23b0945a0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 71927 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee637a0, 0xc231158f00)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73299 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23a67be00, 0xc2221b16e0, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x4f, 0xc2221b16e0, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x4f, 0x1, 0xc2221b16e0, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73289 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc21c0bff00, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2159b8b00, 0xd, 0x10, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc21c0bff00, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2159b8b00, 0xd, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73300 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23a67b000, 0xc2221b1a60, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x51, 0xc2221b1a60, 0x2, 0x2, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x51, 0x1, 0xc2221b1a60, 0x2, 0x2)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73214 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc21c0bf160, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2159b8800, 0xf, 0x10, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc21c0bf160, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2159b8800, 0xf, 0x10)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 73305 [syscall]:
syscall.Syscall(0x4a, 0x51, 0x0, 0x0, 0xf0a888, 0xc26b4b1968, 0x2b798fb7ce30)
    /root/.gvm/gos/go1.4.2/src/syscall/asm_linux_amd64.s:21 +0x5
syscall.Fsync(0x51, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/syscall/zsyscall_linux_amd64.go:503 +0x55
os.(*File).Sync(0xc2277b8598, 0x0, 0x0)
    /root/.gvm/gos/go1.4.2/src/os/file_posix.go:129 +0x6a
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).writeSeriesAndFields(0xc23ba9cc40, 0xc21b54b410, 0xc23177e488, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:427 +0x361
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc23ba9cc40, 0xc22fcb2c00, 0x32, 0x40, 0xc21b54b410, 0xc23177e488, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:231 +0xc4
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc24bf8a880, 0xc22fcb2c00, 0x32, 0x40, 0xc21b54b410, 0xc23177e488, 0x1, 0x1, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:207 +0xc3
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc23b525100, 0xc22fcb2c00, 0x32, 0x40, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:199 +0x3ba
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x41, 0xc22fcb2c00, 0x32, 0x40, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x41, 0x1, 0xc22fcb2c00, 0x32, 0x40)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 71911 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be8700, 0xc24fb0c240)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9650 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22e0f56c0, 0xc2337d53e0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73275 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc2221b0a40, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc22fcb3c00, 0x23, 0x40, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc2221b0a40, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc22fcb3c00, 0x23, 0x40)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447

goroutine 71358 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee630a0, 0xc254ebc0c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9776 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be9260, 0xc23d054fc0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9541 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc234be8000, 0xc24cd72b40)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 9598 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22ee62620, 0xc23d06df80)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73239 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc25544af00, 0xc255ee3980, 0x3, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0x42, 0xc255ee3980, 0x3, 0x4, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0x42, 0x1, 0xc255ee3980, 0x3, 0x4)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 73232 [semacquire]:
sync.(*Mutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/mutex.go:66 +0xd3
sync.(*RWMutex).Lock(0xc2083f81e0)
    /root/.gvm/gos/go1.4.2/src/sync/rwmutex.go:82 +0x33
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc24c84ce00, 0xc2249b3200, 0x1a, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/shard.go:161 +0x125
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc2080321a0, 0xf, 0xc2249b3200, 0x1a, 0x20, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/store.go:289 +0xf7
github.com/influxdb/influxdb/cluster.func·002(0xf, 0x1, 0xc2249b3200, 0x1a, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xfc
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x30b

goroutine 9841 [select]:
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc22c3aa1c0, 0xc237d852c0)
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x458
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
    /root/go/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:215 +0x4bc

goroutine 73327 [select]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc20800e230, 0xc21c0bfbe0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0x1, 0xc2249b2a00, 0x15, 0x20, ...)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x913
github.com/influxdb/influxdb/cluster.func·001(0xc21c0bfbe0, 0xb33810, 0x8, 0xc208029da8, 0x7, 0xc2249b2a00, 0x15, 0x20)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0xad
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x447
@pkittenis pkittenis changed the title [0.9.3rc1] "Unexpected fault" crash [0.9.3rc1] Invalid time stamp in graphite metric causes panic Aug 21, 2015
@otoolep otoolep self-assigned this Aug 21, 2015
@otoolep otoolep added this to the 0.9.3 milestone Aug 21, 2015
@beckettsean
Copy link
Contributor

@pkittenis Thanks for the analysis, but I'm a little confused by your repro step.

echo "test 1 date +%s1" outputs a string like test 1 14401794881, which is not a valid point, regardless of the timestamp. Perhaps you mean echo "test value=1 date +%s1"?

An 11 digit timestamp in seconds, such as of 14,401,794,881, exceeds the maximum time range for InfluxDB: https://influxdb.com/docs/v0.9/guides/troubleshooting.html#time-ranges, but the write succeeds for me without a panic, using the HTTP endpoint. There are no points written, for which I've opened a new bug: #3789

@pkittenis
Copy link
Author

Is test 1 14401794881 being written to the graphite plugin listen port, per default port 2003?

test 1 1440179488 is a valid graphite metric line format. test 1 14401794881 has an 11 digital timestamp, which is invalid, yes. It's not expected that anything be written to the db.

A panic only occurs when writing to the graphite plugin port, not influxdb's HTTP endpoint. No panic occurs when attempting to write invalid 11 digit timestamps on the HTTP end point, default port 8086.

My config:

[[graphite]]
  enabled = true
  bind-address = ":2003"

Everything else default.

Client feeding invalid metric line with 11 digit timestamp into port 2003.

echo "test 1 14401794881" | nc localhost 2003

Influxdb output and stack trace. This is on a fresh db, nothing else done to it.

[metastore] 2015/08/21 22:06:58 [INFO] raft: Node at 127.0.0.1:8088 [Follower] entering Follower state
[metastore] 2015/08/21 22:06:59 [WARN] raft: Heartbeat timeout reached, starting election
[metastore] 2015/08/21 22:06:59 [INFO] raft: Node at 127.0.0.1:8088 [Candidate] entering Candidate state
[metastore] 2015/08/21 22:06:59 [DEBUG] raft: Votes needed: 1
[metastore] 2015/08/21 22:06:59 [DEBUG] raft: Vote granted. Tally: 1
[metastore] 2015/08/21 22:06:59 [INFO] raft: Election won. Tally: 1
[metastore] 2015/08/21 22:06:59 [INFO] raft: Node at 127.0.0.1:8088 [Leader] entering Leader state
[metastore] 2015/08/21 22:06:59 [INFO] raft: Disabling EnableSingleNode (bootstrap)
[metastore] 2015/08/21 22:06:59 [DEBUG] raft: Node 127.0.0.1:8088 updated peer set (2): [127.0.0.1:8088]
[metastore] 2015/08/21 22:06:59 created local node: id=1, host=127.0.0.1:8088
[admin] 2015/08/21 22:06:59 listening on HTTP: [::]:8083
[httpd] 2015/08/21 22:06:59 authentication enabled: false
[httpd] 2015/08/21 22:06:59 listening on HTTP: [::]:8086
[graphite] 2015/08/21 22:07:00 ensured target database graphite exists
[graphite] 2015/08/21 22:07:00 tcp Graphite input opened on [::]:2003
2015/08/21 22:07:00 InfluxDB starting, version 0.9.2, commit 6c0a91f775f9fc0e625d17ffa04a3fe86945ba09
2015/08/21 22:07:00 GOMAXPROCS set to 6
[run] 2015/08/21 22:07:00 listening for signals
2015/08/21 22:07:00 Sending anonymous usage statistics to m.influxdb.com
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x58 pc=0x5c9796]

goroutine 67 [running]:
github.com/influxdb/influxdb/cluster.(*PointsWriter).MapShards(0xc2080101c0, 0xc20819c1c0, 0x7, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:177 +0x696
github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints(0xc2080101c0, 0xc20819c1c0, 0x0, 0x0)
    /root/go/src/github.com/influxdb/influxdb/cluster/points_writer.go:195 +0x1c6
github.com/influxdb/influxdb/services/graphite.(*Service).processBatches(0xc2080860c0, 0xc2081106c0)
    /root/go/src/github.com/influxdb/influxdb/services/graphite/service.go:257 +0x239
created by github.com/influxdb/influxdb/services/graphite.(*Service).Open
    /root/go/src/github.com/influxdb/influxdb/services/graphite/service.go:104 +0x70b
<snip>

Hope that is clearer.

jwilder added a commit that referenced this issue Aug 24, 2015
If a timestamp was larger than the max epoch value was sent via
graphite it would cause the timestamp to overflow when it was
marshaled/unmarshaled back from the raft log.  The overflow cause
the shard group to get created with the wrong timestamp which cause
a panic when writing the point.  The panic was caused because the
timestamp that were supposed to exists in a map created by MapShards
did not actually exist so a nil ShardGroup was used.

The change prevents creating the point with an invalid timestamp.  Since
graphite using a timestamp in seconds, the maximum range is known and
can be prevented.  This also adds a check for the minimum range as well.

Fixes #3785
@jwilder jwilder assigned jwilder and unassigned otoolep Aug 24, 2015
@beckettsean beckettsean modified the milestones: 0.9.4, 0.9.3 Aug 27, 2015
jwilder added a commit that referenced this issue Sep 8, 2015
If a timestamp was larger than the max epoch value was sent via
graphite it would cause the timestamp to overflow when it was
marshaled/unmarshaled back from the raft log.  The overflow cause
the shard group to get created with the wrong timestamp which cause
a panic when writing the point.  The panic was caused because the
timestamp that were supposed to exists in a map created by MapShards
did not actually exist so a nil ShardGroup was used.

The change prevents creating the point with an invalid timestamp.  Since
graphite using a timestamp in seconds, the maximum range is known and
can be prevented.  This also adds a check for the minimum range as well.

Fixes #3785
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants