os/user: LookupId panics on Linux+glibc static build

[kolyshkin]

Since Go 1.10 beta, using tar.FileInfoHeader() is not possible from a static binary compiled on Linux/glibc. This is caused by commit 0564e30 ("archive/tar: populate uname/gname/devmajor/devminor in FileInfoHeader") which makes use of os/user functions LookupId and LookupGroupId, which, in turn, use glibc's getpw* and getgrp* calls (in case CGO is available).

This breaks both Docker (moby/moby#35739 (comment)) and containerd.

This issue is a manifestation of a problem described in more details at #23265 and remedied by commit 62f0127 ("os/user: add a way to enforce pure Go implementation"). Unfortunately, the remedy will only be available in Go 1.11, and requires setting a osusergo build tag for static build.

PS this report is serving merely to document the issue and maybe help others who see the same issue. I do not expect this to be fixed in Go 1.10. As a workaround, a fork of archive/tar is created with the partial revert of 0564e30 here: https://github.com/kolyshkin/go-tar/tree/go-1.10

What version of Go are you using (go version)?

1.10.1

Does this issue reproduce with the latest release?

yes (1.10.1)

What operating system and processor architecture are you using (go env)?

Linux/amd64

What did you do?

Switched to Go 1.10 to compile Docker (moby/moby#35739).

What did you expect to see?

no panic

What did you see instead?

panic while running CI

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe5 pc=0x7fa8342d77f8]

runtime stack:
runtime.throw(0x183d93d, 0x2a)
        /usr/local/go/src/runtime/panic.go:616 +0x83
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:372 +0x292

goroutine 192 [syscall]:
runtime.cgocall(0x1748570, 0xc42095b5a8, 0x1839600)
        /usr/local/go/src/runtime/cgocall.go:128 +0x66 fp=0xc42095b560 sp=0xc42095b528 pc=0x402236
os/user._Cfunc_mygetpwuid_r(0x3, 0xc4207f94a0, 0x34734c0, 0x400, 0xc4200c4578, 0x0)
        _cgo_gotypes.go:170 +0x4f fp=0xc42095b5a8 sp=0xc42095b560 pc=0x97257f
os/user.lookupUnixUid.func1.1(0x7fa800000003, 0xc4207f94a0, 0x34734c0, 0x400, 0xc4200c4578, 0xc42095b638)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:100 +0x141 fp=0xc42095b5e8 sp=0xc42095b5a8 pc=0x9741c1
os/user.lookupUnixUid.func1(0x10)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:100 +0x52 fp=0xc42095b628 sp=0xc42095b5e8 pc=0x974262
os/user.retryWithBuffer(0xc420b01c60, 0xc42095b718, 0xc420b01c60, 0x2199980)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:253 +0x3d fp=0xc42095b688 sp=0xc42095b628 pc=0x97366d
os/user.lookupUnixUid(0x3, 0x0, 0x0, 0x0)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:96 +0x132 fp=0xc42095b750 sp=0xc42095b688 pc=0x972ac2
os/user.lookupUserId(0x187f9c0, 0x1, 0x1, 0x47df00, 0x3)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:86 +0x75 fp=0xc42095b788 sp=0xc42095b750 pc=0x972955
os/user.LookupId(0x187f9c0, 0x1, 0x1, 0x0, 0x0)
        /usr/local/go/src/os/user/lookup.go:41 +0x53 fp=0xc42095b7d8 sp=0xc42095b788 pc=0x971f23
archive/tar.statUnix(0x230c640, 0xc4213e45b0, 0xc420ede2a0, 0x17ecf6e, 0x1)
        /usr/local/go/src/archive/tar/stat_unix.go:39 +0x4c3 fp=0xc42095b858 sp=0xc42095b7d8 pc=0x97dd33
archive/tar.FileInfoHeader(0x230c640, 0xc4213e45b0, 0x0, 0x0, 0xc420559360, 0x99, 0x230c640)
        /usr/local/go/src/archive/tar/common.go:699 +0x493 fp=0xc42095b9d8 sp=0xc42095b858 pc=0x9771b3
github.com/docker/docker/pkg/archive.FileInfoHeader(0xc4205593f6, 0x3, 0x230c640, 0xc4213e45b0, 0x0, 0x0, 0x7412596f, 0xf943ee04c5fd7027, 0x0)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:360 +0x55 fp=0xc42095ba38 sp=0xc42095b9d8 pc=0xd73705
github.com/docker/docker/pkg/archive.(*tarAppender).addTarFile(0xc4211b70c0, 0xc420559360, 0x99, 0xc4205593f6, 0x3, 0x0, 0x0)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:478 +0xd9 fp=0xc42095bad0 sp=0xc42095ba38 pc=0xd73f99
github.com/docker/docker/pkg/archive.TarWithOptions.func1.2(0xc420559360, 0x99, 0x230c640, 0xc4213e44e0, 0x0, 0x0, 0x0, 0xc42095bd28)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:888 +0x612 fp=0xc42095bca0 sp=0xc42095bad0 pc=0xd7ed42
path/filepath.walk(0xc420559360, 0x99, 0x230c640, 0xc4213e44e0, 0xc4200c74f0, 0x0, 0x0)
        /usr/local/go/src/path/filepath/path.go:361 +0xe7 fp=0xc42095bd78 sp=0xc42095bca0 pc=0x50fd17
path/filepath.walk(0xc4213e79a0, 0x97, 0x230c640, 0xc420eac8f0, 0xc4200c74f0, 0x0, 0x50)
        /usr/local/go/src/path/filepath/path.go:381 +0x2c4 fp=0xc42095be50 sp=0xc42095bd78 pc=0x50fef4
path/filepath.Walk(0xc4213e79a0, 0x97, 0xc4200c74f0, 0x17ecf6e, 0x1)
        /usr/local/go/src/path/filepath/path.go:403 +0x108 fp=0xc42095beb0 sp=0xc42095be50 pc=0x510178
github.com/docker/docker/pkg/archive.TarWithOptions.func1(0xc4202a25a0, 0x22eb160, 0xc4203cbec0, 0xc4200c5e48, 0xc4200abfb0, 0xc4203cbea0)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:806 +0x2ee fp=0xc42095bfb0 sp=0xc42095beb0 pc=0xd7f42e
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42095bfb8 sp=0xc42095bfb0 pc=0x45bc71
created by github.com/docker/docker/pkg/archive.TarWithOptions
        /go/src/github.com/docker/docker/pkg/archive/archive.go:748 +0x26f

[dsnet]

This isn't really tar's fault per-say. The underlying issue is os/user, which shouldn't panic in the first place. However, you seem to indicate that os/user is already fixed. Is there anything to do here then?

[kolyshkin]

However, you seem to indicate that os/user is already fixed. Is there anything to do here then?

It is only fixed in Go 1.11 which is not here yet, and the archive/tar started to use os/user in 1.10. So, 1.11 should be fine, and 1.9 was fine, but Go 1.10 is broken in this very case.

[dsnet]

It is unfortunate that #23265 is only for Go1.11. We can't exactly revert the archive/tar change in the next Go1.10 point release, as that is too large of a change. The alternative seems to be back-porting #23265 to Go1.10.2 (if we have one).

[kolyshkin]

The alternative seems to be back-porting #23265 to Go1.10.2 (if we have one)

As this looks to be the only alternative, please let me know if you want me to do a packport.

[andybons]

@kolyshkin The CL to be backported would be https://golang.org/cl/92456, correct?

@ianlancetaylor any objection to cherry-picking?

[ianlancetaylor]

We can't backport that CL by itself, because it wasn't tested and didn't work (#24841, #24845). It would need at least also CLs 106837, 107299, 107304. And in my opinion the sequence of problems and failed fixes means that the risk is too high for a backport to a release branch. Not to mention the fact that this requires the user to understand the problem--it's really not obvious that a crash while using archive/tar should be fixed by using -tags osusergo.

In my opinion the latter problem means that there is still a problem to fix here. The osusergo tag is useful by itself, but as far as this bug is concerned it's only a workaround.

[ianlancetaylor]

I actually don't understand where this crash is coming from, and I don't understand how to recreate it. @kolyshkin Do you have step by step instructions I can use to create the problem?