Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix BranchSimpleIterator panic on close after failed SeekGE #5516

Merged
merged 1 commit into from
Mar 18, 2023

Conversation

nopcoder
Copy link
Contributor

@nopcoder nopcoder commented Mar 18, 2023

While listing branches error from the underlying kv can cause a panic in the iterator Close.
The following occur while the iterator SeekGE failed, caused the following iterator Close to try access nil pointer.

2023/03/18 03:45:38 http: panic serving [::1]:59624: runtime error: invalid memory address or nil pointer dereference
goroutine 521 [running]:
net/http.(*conn).serve.func1()
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:1850 +0x118
panic({0x107acbda0, 0x1089c4110})
        /Users/barak/sdk/go1.19.3/src/runtime/panic.go:890 +0x26c
github.com/treeverse/lakefs/pkg/kv.(*PrimaryIterator).Close(0x0)
        /Users/barak/work/lakeFS/pkg/kv/iterators.go:118 +0x1c
github.com/treeverse/lakefs/pkg/graveler/ref.(*BranchSimpleIterator).Close(0x14000949590)
        /Users/barak/work/lakeFS/pkg/graveler/ref/branch_iterator.go:96 +0x74
github.com/treeverse/lakefs/pkg/catalog.(*Catalog).ListBranches(0x1400070e840, {0x107dac7d0, 0x1400095d260}, {0x14000962069, 0x4}, {0x0, 0x0}, 0x3, {0x1400096207d, 0x4})
        /Users/barak/work/lakeFS/pkg/catalog/catalog.go:562 +0x718
github.com/treeverse/lakefs/pkg/api.(*Controller).ListBranches(0x1400c6cee00, {0x107daaab0, 0x1400094b590}, 0x1400df02600, {0x14000962069, 0x4}, {0x0, 0x140005abbc0, 0x1400095a750})
        /Users/barak/work/lakeFS/pkg/api/controller.go:1740 +0x1a0
github.com/treeverse/lakefs/pkg/api.(*ServerInterfaceWrapper).ListBranches.func1({0x107daaab0, 0x1400094b590}, 0x1400df02600)
        /Users/barak/work/lakeFS/pkg/api/lakefs.gen.go:19608 +0x7c
github.com/treeverse/lakefs/pkg/api.(*ServerInterfaceWrapper).ListBranches(0x1400c9a97a0, {0x107daaab0, 0x1400094b590}, 0x1400df02500)
        /Users/barak/work/lakeFS/pkg/api/lakefs.gen.go:19615 +0xb5c
net/http.HandlerFunc.ServeHTTP(0x1400d00a960, {0x107daaab0, 0x1400094b590}, 0x1400df02500)
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:2109 +0x44
github.com/treeverse/lakefs/pkg/api.MetricsMiddleware.func1.1({0x11073c3e0, 0x1400095d140}, 0x1400df02500)
        /Users/barak/work/lakeFS/pkg/api/metrics_middleware.go:27 +0xe4
net/http.HandlerFunc.ServeHTTP(0x1400d03c0c0, {0x11073c3e0, 0x1400095d140}, 0x1400df02500)
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:2109 +0x44
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x107daaae0, 0x140005ac380}, 0x1400df02500)
        /Users/barak/go/pkg/mod/github.com/prometheus/client_golang@v1.14.0/prometheus/promhttp/instrument_server.go:146 +0xc0
net/http.HandlerFunc.ServeHTTP(0x1400d03c270, {0x107daaae0, 0x140005ac380}, 0x1400df02500)
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:2109 +0x44
github.com/treeverse/lakefs/pkg/api.AuthMiddleware.func1.1({0x107daaae0, 0x140005ac380}, 0x1400df02500)
        /Users/barak/work/lakeFS/pkg/api/auth_middleware.go:72 +0x3e0
net/http.HandlerFunc.ServeHTTP(0x1400cb15280, {0x107daaae0, 0x140005ac380}, 0x1400df02400)
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:2109 +0x44
github.com/treeverse/lakefs/pkg/httputil.DefaultLoggingMiddleware.func1.1({0x107dab2c0, 0x14000944700}, 0x1400df02400)
        /Users/barak/work/lakeFS/pkg/httputil/logging.go:87 +0x534
net/http.HandlerFunc.ServeHTTP(0x1400d03be00, {0x107dab2c0, 0x14000944700}, 0x1400df02200)
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:2109 +0x44
github.com/treeverse/lakefs/pkg/api.OapiRequestValidatorWithOptions.func1.1({0x107dab2c0, 0x14000944700}, 0x1400df02200)
        /Users/barak/work/lakeFS/pkg/api/serve.go:171 +0x2e8
net/http.HandlerFunc.ServeHTTP(0x1400d03c2a0, {0x107dab2c0, 0x14000944700}, 0x1400df02100)
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:2109 +0x44
github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP(0x1400d03be40, {0x107dab2c0, 0x14000944700}, 0x1400df02100)
        /Users/barak/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.0/chain.go:31 +0x48
github.com/go-chi/chi/v5.(*Mux).routeHTTP(0x140001152c0, {0x107dab2c0, 0x14000944700}, 0x1400df02100)
        /Users/barak/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.0/mux.go:436 +0x22c
net/http.HandlerFunc.ServeHTTP(0x1400c999f40, {0x107dab2c0, 0x14000944700}, 0x1400df02100)
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:2109 +0x44
github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0x140001152c0, {0x107dab2c0, 0x14000944700}, 0x1400df02100)
        /Users/barak/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.0/mux.go:87 +0x250
github.com/treeverse/lakefs/cmd/lakefs/cmd.glob..func8.3({0x107dab2c0, 0x14000944700}, 0x1400df02000)
        /Users/barak/work/lakeFS/cmd/lakefs/cmd/run.go:298 +0x144
net/http.HandlerFunc.ServeHTTP(0x1400de08840, {0x107dab2c0, 0x14000944700}, 0x1400df02000)
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:2109 +0x44
net/http.serverHandler.ServeHTTP({0x1400014e960}, {0x107dab2c0, 0x14000944700}, 0x1400df02000)
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:2947 +0x41c
net/http.(*conn).serve(0x140005f8000, {0x107dac728, 0x14000113a00})
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:1991 +0x14e0
created by net/http.(*Server).Serve
        /Users/barak/sdk/go1.19.3/src/net/http/server.go:3102 +0x7e4

Fix #5517

@nopcoder nopcoder added bug Something isn't working include-changelog PR description should be included in next release changelog labels Mar 18, 2023
@nopcoder nopcoder self-assigned this Mar 18, 2023
@nopcoder nopcoder requested a review from a team March 18, 2023 01:55
Copy link
Member

@N-o-Z N-o-Z left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nopcoder nopcoder merged commit 6e9a137 into master Mar 18, 2023
@nopcoder nopcoder deleted the fix/branch-itr-close-panic branch March 18, 2023 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working include-changelog PR description should be included in next release changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

List branch iterator panic after failed SeekGE
2 participants