net/http: nil pointer dereference in closeConnIfStillIdle

[reusee]

1. What version of Go are you using (go version)?
   go version devel +b75b063 Tue Jun 28 04:49:33 2016 +0000 linux/amd64
2. What operating system and processor architecture are you using (go env)?
   GOARCH="amd64"
   GOBIN=""
   GOEXE=""
   GOHOSTARCH="amd64"
   GOHOSTOS="linux"
   GOOS="linux"
   GOPATH="/home/reus"
   GORACE=""
   GOROOT="/home/reus/go"
   GOTOOLDIR="/home/reus/go/pkg/tool/linux_amd64"
   CC="gcc"
   GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build483039895=/tmp/go-build -gno-record-gcc-switches"
   CXX="g++"
   CGO_ENABLED="1"
3. What did you see instead?

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4de58f]

goroutine 54787 [running]:
panic(0x7a2f40, 0xc4200120c0)
    /home/reus/go/src/runtime/panic.go:500 +0x1a1
net/http.(*persistConn).closeConnIfStillIdle(0xc4249cfb00)
    /home/reus/go/src/net/http/transport.go:1255 +0x2f
net/http.(*persistConn).(net/http.closeConnIfStillIdle)-fm()
    /home/reus/go/src/net/http/transport.go:643 +0x2a
created by time.goFunc
    /home/reus/go/src/time/sleep.go:154 +0x44

[ALTree]

You should include a recipe to reproduce the error in your report. Point 3 in the issue report list says

3. What did you do?

If possible, provide a recipe for reproducing the error.

A complete runnable program is good.

A link on play.golang.org is best.

[bradfitz]

In addition to code, as @ALTree mentioned, does your program run under the race detector?

[reusee]

@bradfitz No, not with race detector.

I don't think it will be easy to reproduce by a short example. It's a large long-running program. I believe it's introduced by recent changes. I will try to do some git bisect to find out which commit.

[bradfitz]

No, not with race detector.

Does that mean you haven't tried, or it fails?

I will try to do some git bisect to find out which commit.

Please do. I labeled this "WaitingForInfo" since there's nothing for us to help with until we have more to go on. If you get us something soon, there's a chance it could be fixed for Go 1.7, if there's a problem.

[reusee]

@bradfitz I didn't run the program with race detector, I will give it a try.

[reusee]

Close because the program ran for several days with no crash at all, with race detector enabled / disabled. No way to reproduce.

[vadmeste]

This bug occurred with me too (minio/minio#2456). However, it is not reproducible anymore. So, just wanted to encourage you to review the code if many other people experiment the same issue.

[bradfitz]

@vadmeste, if you find a reproducible example, please file a new bug. You can reference this one. But we don't re-use old bugs.

[kidoman]

Just got this today after upgrading to Go 1.7. Try to see if it occurs multiple times and will file a new bug. As this occurs in a go routine outside our control, it ended up bringing down the entire service.

[bradfitz]

@kidoman, if you find a reproducible example, please file a new bug. You can reference this one. But we don't re-use old bugs.

[bradfitz]

Okay, I have a fuzzy theory, so let's just use this bug, since it was never fixed anyway.

The crash is because persistConn.t is nil and we're crashing on the mutex Lock line here:

func (pc *persistConn) closeConnIfStillIdle() {
        t := pc.t
        t.idleMu.Lock()  // <--- BOOM, because t == nil

As far as I can tell, the only place wheret is nil (since it's never set to nil once created) is in the HTTP/2 (alt) path:

   return &persistConn{alt: next(cm.targetAddr, pconn.conn.(*tls.Conn))}, nil

So somehow that persistConn is getting into the tryPutIdleConn path. I suspect it's via t.putOrCloseIdleConn, in this path:

        handlePendingDial := func() {
                testHookPrePendingDial()
                go func() {
                        if v := <-dialc; v.err == nil {
                                t.putOrCloseIdleConn(v.pc)  // <--- here
                        }
                        testHookPostPendingDial()
                }()
        }

I don't have a repro yet, but it looks suspicious.

[mrjrieke]

Looks promising. Possibly race between setup and teardown?

[gopherbot]

CL https://golang.org/cl/27450 mentions this issue.

[sudersen]

What version of Go are you using (go version)?

go version go1.7.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build191499881=/tmp/go-build"
CXX="g++"
CGO_ENABLED="1"

What did you do?

This is the first time I'm seeing this crash on the same code running for months and It happened for me sporadically on high load. Following are the initial trace.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x5241a0]

What did you expect to see?

We make an HTTP request to a service, process the response and return it as reference to the caller. The reference is never expected to be out of box since we have created an object and this failure did not occur by far for us in the service running for months.

Caller here is spawned using standard waitGroup method. Are there any other race conditions that could happen?

[bradfitz]

@sudersen, you're commenting on a closed issue. Closed issues aren't tracked.

If you're experiencing a problem, please file a new code and include code. There's nothing in your comment to use to help you.