Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GetRegions: stop process and release resources when the connection is disconnected. #6835

Closed
AndreMouche opened this issue Jul 24, 2023 · 1 comment · Fixed by #6838
Closed

Comments

@AndreMouche
Copy link
Member

Enhancement Task

As we have observed, the GetRegions request will return information about all regions. If there are a large number of regions, it will occupy a significant amount of memory in a short period of time.

3: 2013265920 [15: 10066329600] @ 0xedfb76 0xedf50f 0xedf706 0xf4f2ce 0xf50194 0xf51c15 0xf51727 0xf50194 0xf51c15 0xf51727 0xf50194 0xf51fde 0xf4dbb8 0xf4d6ba 0xf59ede 0x1cfb777 0x1cfb5c8 0x1cfdc30 0x1cfe3c5 0x237bdb1 0x1dcdd71 0x1dcd9f3 0x1dcdaf6 0x2375b7e 0x1dcdaf6 0x23752a3 0x1dcdaf6 0x2374a3a 0x1dcdaf6 0x1dce0c5 0x2374daa 0x11db10f
#       0xedfb75        bytes.growSlice+0x95                                                    /usr/local/go/src/bytes/buffer.go:240
#       0xedf50e        bytes.(*Buffer).grow+0x14e                                              /usr/local/go/src/bytes/buffer.go:142
#       0xedf705        bytes.(*Buffer).Write+0x65                                              /usr/local/go/src/bytes/buffer.go:170
#       0xf4f2cd        encoding/json.uintEncoder+0xed                                          /usr/local/go/src/encoding/json/encode.go:566
#       0xf50193        encoding/json.structEncoder.encode+0x1f3                                /usr/local/go/src/encoding/json/encode.go:760
#       0xf51c14        encoding/json.arrayEncoder.encode+0xd4                                  /usr/local/go/src/encoding/json/encode.go:915
#       0xf51726        encoding/json.sliceEncoder.encode+0x366                                 /usr/local/go/src/encoding/json/encode.go:888
#       0xf50193        encoding/json.structEncoder.encode+0x1f3                                /usr/local/go/src/encoding/json/encode.go:760
#       0xf51c14        encoding/json.arrayEncoder.encode+0xd4                                  /usr/local/go/src/encoding/json/encode.go:915
#       0xf51726        encoding/json.sliceEncoder.encode+0x366                                 /usr/local/go/src/encoding/json/encode.go:888
#       0xf50193        encoding/json.structEncoder.encode+0x1f3                                /usr/local/go/src/encoding/json/encode.go:760
#       0xf51fdd        encoding/json.ptrEncoder.encode+0x25d                                   /usr/local/go/src/encoding/json/encode.go:944
#       0xf4dbb7        encoding/json.(*encodeState).reflectValue+0x77                          /usr/local/go/src/encoding/json/encode.go:359
#       0xf4d6b9        encoding/json.(*encodeState).marshal+0xf9                               /usr/local/go/src/encoding/json/encode.go:331
#       0xf59edd        encoding/json.(*Encoder).Encode+0x7d                                    /usr/local/go/src/encoding/json/stream.go:206
#       0x1cfb776       github.com/unrolled/render.JSON.renderStreamingJSON+0x136               /go/pkg/mod/github.com/unrolled/render@v1.0.1/engine.go:147
#       0x1cfb5c7       github.com/unrolled/render.JSON.Render+0x347                            /go/pkg/mod/github.com/unrolled/render@v1.0.1/engine.go:105
#       0x1cfdc2f       github.com/unrolled/render.(*Render).Render+0x4f                        /go/pkg/mod/github.com/unrolled/render@v1.0.1/render.go:341
#       0x1cfe3c4       github.com/unrolled/render.(*Render).JSON+0x184                         /go/pkg/mod/github.com/unrolled/render@v1.0.1/render.go:408
#       0x237bdb0       github.com/tikv/pd/server/api.(*regionsHandler).GetRegions+0x90         /pd/server/api/region.go:334
#       0x1dcdd70       github.com/urfave/negroni.WrapFunc.func1+0x30                           /go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:56
#       0x1dcd9f2       github.com/urfave/negroni.HandlerFunc.ServeHTTP+0x32                    /go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29
#       0x1dcdaf5       github.com/urfave/negroni.middleware.ServeHTTP+0xb5                     /go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38
#       0x2375b7d       github.com/tikv/pd/server/api.(*rateLimitMiddleware).ServeHTTP+0x39d    /pd/server/api/middleware.go:170
#       0x1dcdaf5       github.com/urfave/negroni.middleware.ServeHTTP+0xb5                     /go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38
#       0x23752a2       github.com/tikv/pd/server/api.(*auditMiddleware).ServeHTTP+0x3a2        /pd/server/api/middleware.go:150
#       0x1dcdaf5       github.com/urfave/negroni.middleware.ServeHTTP+0xb5                     /go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38
#       0x2374a39       github.com/tikv/pd/server/api.(*requestInfoMiddleware).ServeHTTP+0x2f9  /pd/server/api/middleware.go:76
#       0x1dcdaf5       github.com/urfave/negroni.middleware.ServeHTTP+0xb5                     /go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38
#       0x1dce0c4       github.com/urfave/negroni.(*Negroni).ServeHTTP+0x124                    /go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96
#       0x2374da9       github.com/tikv/pd/server/api.clusterMiddleware.Middleware.func1+0x1a9  /pd/server/api/middleware.go:99
#       0x11db10e       net/http.HandlerFunc.ServeHTTP+0x2e                                     /usr/local/go/src/net/http/server.go:2109

Currently, we have found that after the connection is disconnected, GetRegions will continue to be processed and occupy a large amount of memory until it is completely handled before the memory is released.This is not a very good design, it will cause resource waste. We hope that after the connection is disconnected, PD can immediately stop processing requests and release resources.

@AndreMouche AndreMouche added the type/enhancement The issue or PR belongs to an enhancement. label Jul 24, 2023
@AndreMouche
Copy link
Member Author

@lhy1024 Please take a look, thanks.

ti-chi-bot bot added a commit that referenced this issue Jul 27, 2023
close #6835

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Jul 27, 2023
close tikv#6835

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot bot pushed a commit that referenced this issue Aug 15, 2023
close #6835

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <admin@liudos.us>
Co-authored-by: Hu# <jinhao.hu@pingcap.com>
ti-chi-bot pushed a commit to ti-chi-bot/pd that referenced this issue Oct 8, 2023
close tikv#6835

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot bot pushed a commit that referenced this issue Oct 8, 2023
close #6835

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: lhy1024 <admin@liudos.us>
rleungx pushed a commit to rleungx/pd that referenced this issue Dec 1, 2023
close tikv#6835

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants