os/user: LookupId panics on Linux+glibc static build #24787

kolyshkin · 2018-04-09T23:56:46Z

Since Go 1.10 beta, using tar.FileInfoHeader() is not possible from a static binary compiled on Linux/glibc. This is caused by commit 0564e30 ("archive/tar: populate uname/gname/devmajor/devminor in FileInfoHeader") which makes use of os/user functions LookupId and LookupGroupId, which, in turn, use glibc's getpw* and getgrp* calls (in case CGO is available).

This breaks both Docker (moby/moby#35739 (comment)) and containerd.

This issue is a manifestation of a problem described in more details at #23265 and remedied by commit 62f0127 ("os/user: add a way to enforce pure Go implementation"). Unfortunately, the remedy will only be available in Go 1.11, and requires setting a osusergo build tag for static build.

PS this report is serving merely to document the issue and maybe help others who see the same issue. I do not expect this to be fixed in Go 1.10. As a workaround, a fork of archive/tar is created with the partial revert of 0564e30 here: https://github.com/kolyshkin/go-tar/tree/go-1.10

What version of Go are you using (`go version`)?

1.10.1

Does this issue reproduce with the latest release?

yes (1.10.1)

What operating system and processor architecture are you using (`go env`)?

Linux/amd64

What did you do?

Switched to Go 1.10 to compile Docker (moby/moby#35739).

What did you expect to see?

no panic

What did you see instead?

panic while running CI

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe5 pc=0x7fa8342d77f8]

runtime stack:
runtime.throw(0x183d93d, 0x2a)
        /usr/local/go/src/runtime/panic.go:616 +0x83
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:372 +0x292

goroutine 192 [syscall]:
runtime.cgocall(0x1748570, 0xc42095b5a8, 0x1839600)
        /usr/local/go/src/runtime/cgocall.go:128 +0x66 fp=0xc42095b560 sp=0xc42095b528 pc=0x402236
os/user._Cfunc_mygetpwuid_r(0x3, 0xc4207f94a0, 0x34734c0, 0x400, 0xc4200c4578, 0x0)
        _cgo_gotypes.go:170 +0x4f fp=0xc42095b5a8 sp=0xc42095b560 pc=0x97257f
os/user.lookupUnixUid.func1.1(0x7fa800000003, 0xc4207f94a0, 0x34734c0, 0x400, 0xc4200c4578, 0xc42095b638)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:100 +0x141 fp=0xc42095b5e8 sp=0xc42095b5a8 pc=0x9741c1
os/user.lookupUnixUid.func1(0x10)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:100 +0x52 fp=0xc42095b628 sp=0xc42095b5e8 pc=0x974262
os/user.retryWithBuffer(0xc420b01c60, 0xc42095b718, 0xc420b01c60, 0x2199980)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:253 +0x3d fp=0xc42095b688 sp=0xc42095b628 pc=0x97366d
os/user.lookupUnixUid(0x3, 0x0, 0x0, 0x0)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:96 +0x132 fp=0xc42095b750 sp=0xc42095b688 pc=0x972ac2
os/user.lookupUserId(0x187f9c0, 0x1, 0x1, 0x47df00, 0x3)
        /usr/local/go/src/os/user/cgo_lookup_unix.go:86 +0x75 fp=0xc42095b788 sp=0xc42095b750 pc=0x972955
os/user.LookupId(0x187f9c0, 0x1, 0x1, 0x0, 0x0)
        /usr/local/go/src/os/user/lookup.go:41 +0x53 fp=0xc42095b7d8 sp=0xc42095b788 pc=0x971f23
archive/tar.statUnix(0x230c640, 0xc4213e45b0, 0xc420ede2a0, 0x17ecf6e, 0x1)
        /usr/local/go/src/archive/tar/stat_unix.go:39 +0x4c3 fp=0xc42095b858 sp=0xc42095b7d8 pc=0x97dd33
archive/tar.FileInfoHeader(0x230c640, 0xc4213e45b0, 0x0, 0x0, 0xc420559360, 0x99, 0x230c640)
        /usr/local/go/src/archive/tar/common.go:699 +0x493 fp=0xc42095b9d8 sp=0xc42095b858 pc=0x9771b3
github.com/docker/docker/pkg/archive.FileInfoHeader(0xc4205593f6, 0x3, 0x230c640, 0xc4213e45b0, 0x0, 0x0, 0x7412596f, 0xf943ee04c5fd7027, 0x0)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:360 +0x55 fp=0xc42095ba38 sp=0xc42095b9d8 pc=0xd73705
github.com/docker/docker/pkg/archive.(*tarAppender).addTarFile(0xc4211b70c0, 0xc420559360, 0x99, 0xc4205593f6, 0x3, 0x0, 0x0)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:478 +0xd9 fp=0xc42095bad0 sp=0xc42095ba38 pc=0xd73f99
github.com/docker/docker/pkg/archive.TarWithOptions.func1.2(0xc420559360, 0x99, 0x230c640, 0xc4213e44e0, 0x0, 0x0, 0x0, 0xc42095bd28)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:888 +0x612 fp=0xc42095bca0 sp=0xc42095bad0 pc=0xd7ed42
path/filepath.walk(0xc420559360, 0x99, 0x230c640, 0xc4213e44e0, 0xc4200c74f0, 0x0, 0x0)
        /usr/local/go/src/path/filepath/path.go:361 +0xe7 fp=0xc42095bd78 sp=0xc42095bca0 pc=0x50fd17
path/filepath.walk(0xc4213e79a0, 0x97, 0x230c640, 0xc420eac8f0, 0xc4200c74f0, 0x0, 0x50)
        /usr/local/go/src/path/filepath/path.go:381 +0x2c4 fp=0xc42095be50 sp=0xc42095bd78 pc=0x50fef4
path/filepath.Walk(0xc4213e79a0, 0x97, 0xc4200c74f0, 0x17ecf6e, 0x1)
        /usr/local/go/src/path/filepath/path.go:403 +0x108 fp=0xc42095beb0 sp=0xc42095be50 pc=0x510178
github.com/docker/docker/pkg/archive.TarWithOptions.func1(0xc4202a25a0, 0x22eb160, 0xc4203cbec0, 0xc4200c5e48, 0xc4200abfb0, 0xc4203cbea0)
        /go/src/github.com/docker/docker/pkg/archive/archive.go:806 +0x2ee fp=0xc42095bfb0 sp=0xc42095beb0 pc=0xd7f42e
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42095bfb8 sp=0xc42095bfb0 pc=0x45bc71
created by github.com/docker/docker/pkg/archive.TarWithOptions
        /go/src/github.com/docker/docker/pkg/archive/archive.go:748 +0x26f

The text was updated successfully, but these errors were encountered:

dsnet · 2018-04-10T00:47:35Z

This isn't really tar's fault per-say. The underlying issue is os/user, which shouldn't panic in the first place. However, you seem to indicate that os/user is already fixed. Is there anything to do here then?

kolyshkin · 2018-04-10T02:00:45Z

However, you seem to indicate that os/user is already fixed. Is there anything to do here then?

It is only fixed in Go 1.11 which is not here yet, and the archive/tar started to use os/user in 1.10. So, 1.11 should be fine, and 1.9 was fine, but Go 1.10 is broken in this very case.

dsnet · 2018-04-10T02:06:17Z

It is unfortunate that #23265 is only for Go1.11. We can't exactly revert the archive/tar change in the next Go1.10 point release, as that is too large of a change. The alternative seems to be back-porting #23265 to Go1.10.2 (if we have one).

kolyshkin · 2018-04-13T01:20:16Z

The alternative seems to be back-porting #23265 to Go1.10.2 (if we have one)

As this looks to be the only alternative, please let me know if you want me to do a packport.

golang/go#24787

andybons · 2018-04-27T19:58:20Z

@kolyshkin The CL to be backported would be https://golang.org/cl/92456, correct?

@ianlancetaylor any objection to cherry-picking?

ianlancetaylor · 2018-04-27T20:08:59Z

We can't backport that CL by itself, because it wasn't tested and didn't work (#24841, #24845). It would need at least also CLs 106837, 107299, 107304. And in my opinion the sequence of problems and failed fixes means that the risk is too high for a backport to a release branch. Not to mention the fact that this requires the user to understand the problem--it's really not obvious that a crash while using archive/tar should be fixed by using -tags osusergo.

In my opinion the latter problem means that there is still a problem to fix here. The osusergo tag is useful by itself, but as far as this bug is concerned it's only a workaround.

ianlancetaylor · 2018-04-27T20:14:41Z

I actually don't understand where this crash is coming from, and I don't understand how to recreate it. @kolyshkin Do you have step by step instructions I can use to create the problem?

kolyshkin · 2018-05-04T00:50:51Z

Do you have step by step instructions I can use to create the problem?

@ianlancetaylor unfortunately not. Before filing this bug I spent some time trying to create a simple reproducer but failed. Perhaps it requires some specific version of glibc, or there is something else absent from the simple reproducers I tried. What we see in docker CI is here moby/moby#35739 (comment)

Let me try again to create a repro

kolyshkin · 2018-05-09T03:16:32Z

It looks like the reason of segfault is indeed a glibc bug described here: #13470

kolyshkin · 2018-05-09T03:32:53Z

and here's a reproducer, modeled after the one in C from #13470 (comment)

package main

import (
	"fmt"
	"os/user"
)

func main() {
	id := "3"
	wait := make(chan bool)
	ret := make(chan *user.User)
	go func() {
		<-wait
		u, _ := user.LookupId(id)
		ret <- u
	}()
	u, _ := user.LookupId(id)
	wait <- true
	fmt.Printf("%+v\n", u)
	u = <-ret
	fmt.Printf("%+v\n", u)
}

When compiled statically (i.e. go build -ldflags '-extldflags "-fno-PIC -static"' -buildmode pie), it crashes on my machine (Ubuntu 17.10, amd64, go-1.10, glibc 2.26):

&{Uid:3 Gid:3 Username:sys Name:sys HomeDir:/dev}
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe5 pc=0x7ff160f4f698]

runtime stack:
runtime.throw(0x53c045, 0x2a)
	/usr/lib/go-1.10/src/runtime/panic.go:619 +0x83
runtime.sigpanic()
	/usr/lib/go-1.10/src/runtime/signal_unix.go:372 +0x292

goroutine 5 [syscall]:
runtime.cgocall(0x48a4d0, 0xc420040d50, 0x53bdad)
	/usr/lib/go-1.10/src/runtime/cgocall.go:128 +0x66 fp=0xc420040d08 sp=0xc420040cd0 pc=0x402266
os/user._Cfunc_mygetpwuid_r(0x3, 0xc42009c000, 0x7ff14c000b10, 0x400, 0xc42009e000, 0x0)
	_cgo_gotypes.go:170 +0x4f fp=0xc420040d50 sp=0xc420040d08 pc=0x488a1f
os/user.lookupUnixUid.func1.1(0x3, 0xc42009c000, 0x7ff14c000b10, 0x400, 0xc42009e000, 0xc420035de0)
	/usr/lib/go-1.10/src/os/user/cgo_lookup_unix.go:100 +0x141 fp=0xc420040d90 sp=0xc420040d50 pc=0x489921
os/user.lookupUnixUid.func1(0xc420035db0)
	/usr/lib/go-1.10/src/os/user/cgo_lookup_unix.go:100 +0x52 fp=0xc420040dd0 sp=0xc420040d90 pc=0x4899c2
os/user.retryWithBuffer(0xc4200a0000, 0xc420040ec0, 0xc4200a0000, 0x7aa060)
	/usr/lib/go-1.10/src/os/user/cgo_lookup_unix.go:253 +0x3d fp=0xc420040e30 sp=0xc420040dd0 pc=0x48963d
os/user.lookupUnixUid(0x3, 0x0, 0x0, 0x0)
	/usr/lib/go-1.10/src/os/user/cgo_lookup_unix.go:96 +0x132 fp=0xc420040ef8 sp=0xc420040e30 pc=0x488f62
os/user.lookupUserId(0x53614a, 0x1, 0x0, 0x404e4b, 0xc42007c060)
	/usr/lib/go-1.10/src/os/user/cgo_lookup_unix.go:86 +0x75 fp=0xc420040f30 sp=0xc420040ef8 pc=0x488df5
os/user.LookupId(0x53614a, 0x1, 0x0, 0x0, 0x0)
	/usr/lib/go-1.10/src/os/user/lookup.go:41 +0x53 fp=0xc420040f80 sp=0xc420040f30 pc=0x488763
main.main.func1(0xc42007c060, 0x53614a, 0x1, 0xc42007c0c0)
	/home/kir/go/src/github.com/kolyshkin/test/lookup.go:14 +0x4e fp=0xc420040fc0 sp=0xc420040f80 pc=0x48a2fe
runtime.goexit()
	/usr/lib/go-1.10/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420040fc8 sp=0xc420040fc0 pc=0x450c51
created by main.main
	/home/kir/go/src/github.com/kolyshkin/test/lookup.go:12 +0xb3

goroutine 1 [chan receive]:
main.main()
	/home/kir/go/src/github.com/kolyshkin/test/lookup.go:20 +0x164

kolyshkin · 2018-05-09T03:47:48Z

it's really not obvious that a crash while using archive/tar should be fixed by using -tags osusergo. <...>
In my opinion the latter problem means that there is still a problem to fix here. The osusergo tag is useful by itself, but as far as this bug is concerned it's only a workaround.

Producing a static build already requires a non-trivial amount of flags passed to go build, which on Linux currently amounts to something like

-ldflags '-extldflags "-fno-PIC -static"' -buildmode pie -tags 'osusergo netgo static_build'

...and this magic string keeps growing.

Perhaps the solution is to encapsulate this knowledge internally, exposing it via a new -static flag for go build and friends?

One other good thing such a flag could also do is to add --static flag to pkg-config invocations (those initiated by // #cgo pkg-config: lib lines in the source code). It will solve another issue for which a somewhat verbose workaround is currently required (for example, see ploop_link_static.go and ploop_link_dynamic.go). In fact I have already suggested it some time ago in #12058.

ianlancetaylor · 2018-05-09T17:55:57Z

@kolyshkin I suggest that you open a new bug for a -static option.

I'm inclined to close this bug. I don't see what we can change in Go to fix it. There is a workaround, as you know.

kolyshkin · 2018-07-20T00:00:15Z

I suggest that you open a new bug for a -static option.

Done, see #26492

FiloSottile · 2018-08-31T19:12:10Z

Closing based on #24787 (comment)

1. LookupId panics on Linux+glibc static build (golang/go#24787) 2. Updated go version to 1.11.0 in Dockerfile.build 3. added go build tags netgo osusergo in static binary

1. LookupId panics on Linux+glibc static build (golang/go#24787) 2. Updated go version to 1.11.0 in Dockerfile.build 3. added go build tags netgo osusergo in static binary Signed-off-by: vijaysamanuri <vijay.samanuri@wavemaker.com>

dsnet changed the title ~~1.10 archive/tar: using tar.FileInfoHeader() results in panic on Linux+glibc static build~~ os/user: LookupId panics on Linux+glibc static build Apr 10, 2018

kolyshkin mentioned this issue Apr 10, 2018

Bump Golang to 1.10.1 moby/moby#35739

Merged

dsnet added this to the Go1.10.2 milestone Apr 10, 2018

AkihiroSuda mentioned this issue Apr 10, 2018

[Go 1.10 archive/tar] static build is broken when built with glibc moby/buildkit#333

Closed

tamalsaha added a commit to appscodelabs/libbuild that referenced this issue Apr 15, 2018

Use go 1.9 compiler since tar package breaks static build

0e54dac

golang/go#24787

FiloSottile added the CherryPickCandidate Used during the release process for point releases label Apr 24, 2018

andybons removed the CherryPickCandidate Used during the release process for point releases label Apr 27, 2018

andybons modified the milestones: Go1.10.2, Go1.10.3 Apr 27, 2018

kolyshkin mentioned this issue May 3, 2018

[DO NOT MERGE] remove vendored archive/tar moby/moby#36997

Closed

alexellis mentioned this issue May 12, 2018

of-builder: use Go client instead of buildctl binary openfaas/openfaas-cloud#44

Merged

FiloSottile modified the milestones: Go1.10.3, Go1.10.4 Jun 7, 2018

kolyshkin mentioned this issue Jun 9, 2018

Vendoring standard packages LK4D4/vndr#72

Open

tamalsaha mentioned this issue Jun 23, 2018

reproducible builds appscodelabs/libbuild#18

Open

kolyshkin mentioned this issue Jul 19, 2018

cmd/go: build: add -static flag #26492

Open

andybons removed this from the Go1.10.4 milestone Aug 24, 2018

andybons added this to the Go1.10.5 milestone Aug 24, 2018

FiloSottile closed this as completed Aug 31, 2018

anitgandhi mentioned this issue Nov 16, 2018

Docker 18.09.0 docker-archive/for-aws#183

Closed

zeripath mentioned this issue May 6, 2019

Debian 9.6 AMD64: Segfault on start at os/user.Current (1.7.0+) go-gitea/gitea#6841

Closed

7 tasks

vijaySamanuri mentioned this issue Jun 11, 2019

Fixing static binary build using container containers/skopeo#670

Closed

golang locked and limited conversation to collaborators Aug 31, 2019

gopherbot added the FrozenDueToAge label Aug 31, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os/user: LookupId panics on Linux+glibc static build #24787

os/user: LookupId panics on Linux+glibc static build #24787

kolyshkin commented Apr 9, 2018

dsnet commented Apr 10, 2018

kolyshkin commented Apr 10, 2018

dsnet commented Apr 10, 2018

kolyshkin commented Apr 13, 2018

andybons commented Apr 27, 2018 •

edited

Loading

ianlancetaylor commented Apr 27, 2018

ianlancetaylor commented Apr 27, 2018

kolyshkin commented May 4, 2018

kolyshkin commented May 9, 2018

kolyshkin commented May 9, 2018

kolyshkin commented May 9, 2018

ianlancetaylor commented May 9, 2018

kolyshkin commented Jul 20, 2018

FiloSottile commented Aug 31, 2018

os/user: LookupId panics on Linux+glibc static build #24787

os/user: LookupId panics on Linux+glibc static build #24787

Comments

kolyshkin commented Apr 9, 2018

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

dsnet commented Apr 10, 2018

kolyshkin commented Apr 10, 2018

dsnet commented Apr 10, 2018

kolyshkin commented Apr 13, 2018

andybons commented Apr 27, 2018 • edited Loading

ianlancetaylor commented Apr 27, 2018

ianlancetaylor commented Apr 27, 2018

kolyshkin commented May 4, 2018

kolyshkin commented May 9, 2018

kolyshkin commented May 9, 2018

kolyshkin commented May 9, 2018

ianlancetaylor commented May 9, 2018

kolyshkin commented Jul 20, 2018

FiloSottile commented Aug 31, 2018

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

andybons commented Apr 27, 2018 •

edited

Loading