Move to Go 1.5.2 when released (was: 0.9.5-nightly-827c513 panic: objectstart: bad pointer in unexpected span)

[daviesalex]

[http] 2015/10/23 11:45:16 7.132.32.31 - - [23/Oct/2015:11:44:56 -0500] POST /write?consistency=&db=telegraf&precision=s&rp= HTTP/1.1 500 32 - InfluxDBClient 649d91d5-79a5-11e5-9838-000000000000 19.892626801s
[http] 2015/10/23 11:45:16 7.132.32.31 - - [23/Oct/2015:11:45:03 -0500] POST /write?consistency=&db=telegraf&precision=s&rp= HTTP/1.1 500 32 - InfluxDBClient 68434da2-79a5-11e5-9cd7-000000000000 13.77702484s
runtime:objectstart Span weird: p=0xd06e0b7320 k=0x683705b s.start=0xd06e0b2000 s.limit=0xd06e0b7f80 s.state=4
fatal error: objectstart: bad pointer in unexpected span

goroutine 133 [running]:
runtime.throw(0xd397a0, 0x2b)
        /root/.gvm/gos/go1.5.1/src/runtime/panic.go:527 +0x90 fp=0xc8206b9e18 sp=0xc8206b9e00
runtime.heapBitsForObject(0xd06e0b7320, 0x0, 0x0, 0xc800000000, 0x7fff210a8100)
        /root/.gvm/gos/go1.5.1/src/runtime/mbitmap.go:217 +0x287 fp=0xc8206b9e50 sp=0xc8206b9e18
runtime.scanobject(0xd94363a990, 0xc820038720)
        /root/.gvm/gos/go1.5.1/src/runtime/mgcmark.go:878 +0x239 fp=0xc8206b9f20 sp=0xc8206b9e50
runtime.gcDrain(0xc820038720, 0x7d0)
        /root/.gvm/gos/go1.5.1/src/runtime/mgcmark.go:689 +0xf4 fp=0xc8206b9f50 sp=0xc8206b9f20
runtime.gcBgMarkWorker(0xc820037500)
        /root/.gvm/gos/go1.5.1/src/runtime/mgc.go:1320 +0x281 fp=0xc8206b9fb8 sp=0xc8206b9f50
runtime.goexit()
        /root/.gvm/gos/go1.5.1/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8206b9fc0 sp=0xc8206b9fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5.1/src/runtime/mgc.go:1239 +0x93

goroutine 1 [chan receive, 83 minutes]:
main.(*Main).Run(0xc8202bdf10, 0xc82000a060, 0x4, 0x4, 0x0, 0x0)
        /tmp/tmp.kvLu2bJaVO/src/github.com/influxdb/influxdb/cmd/influxd/main.go:96 +0xa39
main.main()
        /tmp/tmp.kvLu2bJaVO/src/github.com/influxdb/influxdb/cmd/influxd/main.go:46 +0x29b

goroutine 17 [syscall, 83 minutes, locked to thread]:
runtime.goexit()
        /root/.gvm/gos/go1.5.1/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall, 83 minutes]:
os/signal.loop()
        /root/.gvm/gos/go1.5.1/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /root/.gvm/gos/go1.5.1/src/os/signal/signal_unix.go:28 +0x37

goroutine 18 [chan receive, 83 minutes]:
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc820052158, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.kvLu2bJaVO/src/github.com/influxdb/influxdb/tcp/mux.go:129 +0x6f
github.com/influxdb/influxdb/meta.(*Store).serveExecListener(0xc8200c08c0)
        /tmp/tmp.kvLu2bJaVO/src/github.com/influxdb/influxdb/meta/store.go:622 +0xa2
created by github.com/influxdb/influxdb/meta.(*Store).Open
        /tmp/tmp.kvLu2bJaVO/src/github.com/influxdb/influxdb/meta/store.go:239 +0x364

[daviesalex]

The stack trace is 216mb (!) but if somebody wants it, I can send it to you. I've kept a copy of it (its only 10mb compressed)

[adavies@ljoit-adl1 ~]$ zcat issue-4554-influxdb.log.gz | wc -l
3020501
[adavies@ljoit-adl1 ~]$ ls -lh issue-4554-influxdb.log.gz
-rw-r--r-- 1 adavies it 10M Oct 23 12:33 issue-4554-influxdb.log.gz
[adavies@ljoit-adl1 ~]$

[otoolep]

This issue has been marked "tsm".

@daviesalex -- are you running the "tsm1" engine?

[daviesalex]

@otoolep - yes.

[otoolep]

OK, thanks @daviesalex -- I wanted to ensure this issue was labelled correctly.

[jwilder]

This might be a Go bug. Code where panic is raised: https://github.com/golang/go/blob/release-branch.go1.5/src/runtime/mbitmap.go#L217

[jwilder]

It's possible this is related to golang/go#12233 and should be fixed in go 1.5.2. The same stack trace for this issue and #4548 are reported in that issue.

[daviesalex]

@otoolep , @jwilder - fantastic stuff, thanks for tracking that down! I guess rename this issue 'Move to Go 1.5.2' as several other Influx users have hit this? (also seems that golang/go#12932 is tagged for that release, and affects InfluxDB)

We are using your nightly builds right now, but if this panic becomes a big drama for us we will move to building our own against golang head...

[sebito91]

FYI, as a test we're going to build go from HEAD along with InfluxDB from HEAD to see if this addresses the issue...will post the results through the weekend.

[otoolep]

https://github.com/golang/go/milestones is what we need to watch for the release of Go 1.5.2.

[otoolep]

We're reverting to Go 1.4.2 for the 0.9.5 release.

[sebito91]

Will tonight's nightly reflect that? Was trying to build against HEAD today
and was getting bad file descriptor issues using 1.5.1...
On Nov 11, 2015 5:56 PM, "otoolep" notifications@github.com wrote:

We're reverting to Go 1.4.2 for the 0.9.5 release.

—
Reply to this email directly or view it on GitHub
#4554 (comment).

[otoolep]

Yes, tonight's build should be built with 1.4.2. The start-up log will confirm. If this it not what you see, let us know.

[sebito91]

Thanks @otoolep, absolutely will do.

On Wednesday, November 11, 2015, otoolep notifications@github.com wrote:

Yes, tonight's build should be built with 1.4.2. The start-up log will
confirm. If this it not what you see, let us know.

—
Reply to this email directly or view it on GitHub
#4554 (comment).

Sebastian Borza

PGP: EDC2 BF61 4B91 14F2 AAB4 06C9 3744 7F3F E411 0D3E

[daviesalex]

We can confirm that the 1.4.2 revert fixes the problems we had. We also, at the request of @pauldix, tested the latest 1.5.2 golang branch and that appeared vastly improved vs 1.5.1, although we still saw higher system load than 1.4.2.

As soon as 1.5.2 is released, we will happily test again. As our InfluxDB setup becomes more stable, it becomes easier to test golang change in isolation.

[huhongbo]

1.5.2 golang has released
will influxdb switch to 1.5.2 ?

[otoolep]

The plan is not to move to 1.5.2 until after 0.9.6 is released.

[otoolep]

Dupe of #4760