Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Looks like there is a memory leak #1780

Closed
hktalent opened this issue Jan 12, 2023 · 20 comments
Closed

Looks like there is a memory leak #1780

hktalent opened this issue Jan 12, 2023 · 20 comments

Comments

@hktalent
Copy link

profile.pb.gz
Screenshot 2023-01-12 at 11 48 36

go tool pprof -http=:9999 profile.pb.gz
@hktalent
Copy link
Author

Screenshot 2023-01-15 at 17 07 28

[profile.pb.gz](https://github.com/blevesearch/bleve/files/10419809/profile.pb.gz)

@hktalent
Copy link
Author 

// 并发 默认 16，thread 配置,fnCbk 在成功保存
//Concurrency defaults to 16, thread configuration, fncbk is saved successfully
func SaveIndexDoc(szIndex, docID string, doc interface{}, fnCbk func(), fnEnd func()) {
	fmt.Printf("SaveThread len %d\r", len(SaveThread))
	SaveThread <- struct{}{}
	util1.DefaultPool.Submit(func() {
		defer func() {
			<-SaveThread
			if nil != fnEnd {
				fnEnd()
			}
		}()
		index := http.IndexByName(szIndex)
		if index == nil || nil == doc {
			fmt.Sprintf("no such index '%s' \n", szIndex)
			return
		}
		if err := index.Index(docID, doc); err != nil {
			fmt.Printf("error indexing document '%s': %v\n", docID, err)
		} else if nil != fnCbk {
			fnCbk()
		}
	})
}

@hktalent
Copy link
Author

hktalent commented Jan 15, 2023 •
edited
Loading

Screenshot 2023-01-15 at 22 55 10

Screenshot 2023-01-15 at 22 56 39

@hktalent
Copy link
Author

profile.pb.gz

@hktalent
Copy link
Author 

heap profile: 7916: 56499325256 [3786847: 415286849928] @ heap/1048576
63: 1585446912 [75: 1887436800] @ 0x100099b50 0x100099078 0x100099305 0x1017e120c 0x1017e7f02 0x1017e80e5 0x1017ef394 0x1017fa83b 0x1017f4405 0x1017f3045 0x1017f2cd5 0x101829669 0x1019038f5 0x101dcbb43 0x10114e50e 0x100077bc1
#	0x100099b4f	bytes.growSlice+0x16f								/usr/local/opt/go/libexec/src/bytes/buffer.go:240
#	0x100099077	bytes.(*Buffer).grow+0x3b7							/usr/local/opt/go/libexec/src/bytes/buffer.go:142
#	0x100099304	bytes.(*Buffer).Write+0xc4							/usr/local/opt/go/libexec/src/bytes/buffer.go:170
#	0x1017e120b	github.com/blevesearch/zapx/v15.(*CountHashWriter).Write+0x8b			/Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/count.go:44
#	0x1017e7f01	github.com/blevesearch/zapx/v15.(*chunkedIntCoder).Write+0x3e1			/Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/intcoder.go:161
#	0x1017e80e4	github.com/blevesearch/zapx/v15.(*chunkedIntCoder).writeAt+0x104		/Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/intcoder.go:181
#	0x1017ef393	github.com/blevesearch/zapx/v15.writePostings+0x253				/Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/merge.go:613
#	0x1017fa83a	github.com/blevesearch/zapx/v15.(*interim).writeDicts+0x1a5a			/Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/new.go:737
#	0x1017f4404	github.com/blevesearch/zapx/v15.(*interim).convert+0x504			/Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/new.go:277
#	0x1017f3044	github.com/blevesearch/zapx/v15.(*ZapPlugin).newWithChunkMode+0x284		/Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/new.go:72
#	0x1017f2cd4	github.com/blevesearch/zapx/v15.(*ZapPlugin).New+0x94				/Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/new.go:47
#	0x101829668	github.com/blevesearch/bleve/v2/index/scorch.(*Scorch).Batch+0x748		/Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/bleve/v2/index/scorch/scorch.go:412
#	0x1019038f4	github.com/blevesearch/bleve/v2.(*indexImpl).Batch+0xf4				/Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/bleve/v2/index_impl.go:310
#	0x101dcbb42	github.com/hktalent/51pwnPlatform/pkg/blevExp.SaveIndexDoc4Batch.func1+0x262	/Users/51pwn/MyWork/goSqlite_gorm/pkg/blevExp/indexImp.go:375

@abhinavdangeti
Sorry to bother you again

Can you guide me to solve this memory leak problem?

After one night's operation, the memory gradually increased to 100G

I build indexes in batches, with 2000 entries each time

And that's it

@hktalent
Copy link
Author

hktalent commented Jan 17, 2023 •
edited
Loading

After several analyses, it is determined that there is a memory leak
@mschoch @abhinavdangeti @sreekanth-cb
profile.pb.gz

go tool pprof -http=:9999 profile.pb.gz

@hktalent hktalent reopened this Jan 27, 2023
@hktalent
Copy link
Author

hktalent commented Jan 27, 2023 •
edited
Loading

Screenshot 2023-01-27 at 09 04 15

profile.pb.gz

@hktalent
Copy link
Author

hktalent commented Feb 21, 2023 •
edited
Loading

@mschoch @abhinavdangeti @sreekanth-cb
memory leak,If I can, can I pay to support you to solve this problem?

@abhinavdangeti
Copy link
Member

@hktalent You mention that you're batching your documents into the bleve.Index. Would you share the exact snippet of code where you're doing this.

@abhinavdangeti
Copy link
Member

We've seen a similar report elsewhere as well - we're trying to understand the root cause for this at the moment.
A sample program reproducing the situation above would help greatly!

abhinavdangeti added a commit to blevesearch/zapx that referenced this issue Feb 23, 2023
@abhinavdangeti
Place a ceiling for initial capacity allocated for bytes.Buffer
b6800de 
+ Initializing the ceiling to 10MB. This means that at least
  10 MB of space is guaranteed if the estimate is larger than
  that. Any additional space will be allocated on demand.
+ Intended to address: blevesearch/bleve#1780
@abhinavdangeti abhinavdangeti mentioned this issue Feb 23, 2023
Open
@abhinavdangeti abhinavdangeti self-assigned this Feb 23, 2023
@hktalent
Copy link
Author

@abhinavdangeti

// 批量保存,并发限制16
func SaveIndexDoc4Batch(szIndex string, doc chan *IndexData, fnCbk func(), fnEnd func()) {
	log.Println("SaveIndexDoc4Batch len", len(cBath))
	cBath <- struct{}{}
	go func() {
		defer func() {
			if nil != fnEnd {
				fnEnd()
			}
			<-cBath
		}()
		index := http.IndexByName(szIndex)
		if index == nil || nil == doc {
			fmt.Sprintf("no such index '%s' \n", szIndex)
			return
		}
		badd := index.NewBatch()
		n := len(doc)
		if 0 < n {
			for i := 0; i < n; i++ {
				d1 := <-doc
				badd.Index(d1.Id, d1.Doc)
			}
		}
		if err := index.Batch(badd); nil != err {
			log.Println("index.Batch ", err)
		} else {
			fnCbk()
			log.Printf("save ok, %d", n)
		}
	}()
}





if 0 < n1 && 10000 <= n1 {
		dox := r.docs
		SaveIndexDoc4Batch(r.Index, dox, r.fnCbk, func() {
			close(dox)
			log.Println("close dox for SaveIndexDoc4Batch")
		})
		r.docs = make(chan *IndexData, r.MaxBatch)
	}

@hktalent
Copy link
Author

hktalent commented Feb 23, 2023 •
edited
Loading

@abhinavdangeti
I update

github.com/blevesearch/zapx/v11 v11.3.7 // indirect
	github.com/blevesearch/zapx/v12 v12.3.7 // indirect
	github.com/blevesearch/zapx/v13 v13.3.7 // indirect
	github.com/blevesearch/zapx/v14 v14.3.7 // indirect
	github.com/blevesearch/zapx/v15 v15.3.9 // indirect

After, the problem still exists, thank you

profile.pb.gz

go tool pprof -http=:9999 profile.pb.gz

http://localhost:9999/ui/

@abhinavdangeti
Copy link
Member

Without the docs, that code snippet you've shared will not help me reproduce the problem :)

Would you add this line in your application somewhere (maybe after indexing every 10 or so batches) and let us know of the behavior you observe ..

debug.FreeOSMemory() // need to import runtime/debug

This forces a garbage collection followed by an attempt to return as much memory to the operating system as possible. See https://pkg.go.dev/runtime/debug#FreeOSMemory

@hktalent
Copy link
Author

1、download
https://github.com/blevesearch/bleve/files/10810564/profile.pb.gz
2、run

go tool pprof -http=:9999 profile.pb.gz

3、open
http://localhost:9999/ui/

4、http://localhost:9999/ui/
image
5、
http://localhost:9999/ui/source?f=bytes%5C.growSlice%7Cbytes%5C.%5C%28%5C*Buffer%5C%29%5C.grow%7Cbytes%5C.%5C%28%5C*Buffer%5C%29%5C.Write%7Cgithub%5C.com%2Fblevesearch%2Fzapx%2Fv15%5C.%5C%28%5C*CountHashWriter%5C%29%5C.Write%7Cgithub%5C.com%2Fblevesearch%2Fzapx%2Fv15%5C.writePostings%7Cgithub%5C.com%2Fblevesearch%2Fzapx%2Fv15%5C.%5C%28%5C*interim%5C%29%5C.writeDicts

image

@abhinavdangeti
Copy link
Member

abhinavdangeti commented Feb 23, 2023 •
edited
Loading

@hktalent any chance you could accommodate this -

Would you add this line in your application somewhere (maybe after indexing every 10 or so batches) and let us know of the behavior you observe ..

debug.FreeOSMemory() // need to import runtime/debug

This forces a garbage collection followed by an attempt to return as much memory to the operating system as possible. See https://pkg.go.dev/runtime/debug#FreeOSMemory

Or is your last profile with this in place already?

@hktalent
Copy link
Author

I added

debug.FreeOSMemory（）

Memory is still growing. Tomorrow I will grab another profile.pb.gz to have a look

@abhinavdangeti
Copy link
Member

Also, something else to test would be to limit the amount of memory used by your application with : https://pkg.go.dev/runtime/debug#SetMemoryLimit .

@hktalent
Copy link
Author

hktalent commented Mar 6, 2023 •
edited
Loading

@abhinavdangeti
Still the same
image

Type: alloc_space
Time: Mar 6, 2023 at 9:07pm (CST)
Duration: 599.99s, Total samples = 106890.40MB 
Showing nodes accounting for 106890.40MB, 100% of 106890.40MB total
----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context 	 	 
----------------------------------------------------------+-------------
                                        43082.33MB 50.02% |   github.com/RoaringBitmap/roaring.(*arrayContainer).toBitmapContainer /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/RoaringBitmap/roaring/arraycontainer.go:230
                                        43034.97MB 49.97% |   github.com/RoaringBitmap/roaring.(*runContainer16).toBitmapContainer /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/RoaringBitmap/roaring/runcontainer.go:2439
                                            7.05MB 0.0082% |   github.com/RoaringBitmap/roaring.(*bitmapContainer).andNotBitmap /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/RoaringBitmap/roaring/bitmapcontainer.go:920
86124.36MB 80.57% 80.57% 86124.36MB 80.57%                | github.com/RoaringBitmap/roaring.newBitmapContainer /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/RoaringBitmap/roaring/bitmapcontainer.go:24
----------------------------------------------------------+-------------
                                         2003.21MB 50.93% |   github.com/blevesearch/zapx/v15.(*PostingsList).iterator /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/posting.go:223
                                         1930.21MB 49.07% |   github.com/blevesearch/zapx/v15.(*PostingsList).iterator /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/posting.go:226
 3933.42MB  3.68% 84.25%  3933.42MB  3.68%                | github.com/RoaringBitmap/roaring.(*Bitmap).Iterator /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/RoaringBitmap/roaring/roaring.go:572
----------------------------------------------------------+-------------
                                         1457.61MB 51.46% |   github.com/blevesearch/zapx/v15.(*PostingsList).read /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/posting.go:299
                                         1375.10MB 48.54% |   github.com/RoaringBitmap/roaring.AndNot /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/RoaringBitmap/roaring/roaring.go:1351
 2832.72MB  2.65% 86.90%  2832.72MB  2.65%                | github.com/RoaringBitmap/roaring.NewBitmap /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/RoaringBitmap/roaring/roaring.go:183
----------------------------------------------------------+-------------
                                         1732.58MB 80.43% |   github.com/blevesearch/vellum.(*unfinishedNodes).addSuffix /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/vellum/builder.go:309
                                          421.52MB 19.57% |   github.com/blevesearch/vellum.(*unfinishedNodes).pushEmpty /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/vellum/builder.go:256
 2154.10MB  2.02% 88.92%  2154.10MB  2.02%                | github.com/blevesearch/vellum.(*builderNodePool).Get /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/vellum/builder.go:438
----------------------------------------------------------+-------------
                                         1940.15MB   100% |   github.com/blevesearch/zapx/v15.(*Dictionary).postingsListFromOffset /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/dict.go:74
 1940.15MB  1.82% 90.73%  1940.15MB  1.82%                | github.com/blevesearch/zapx/v15.(*Dictionary).postingsListInit /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/zapx/v15/dict.go:86
----------------------------------------------------------+-------------
                                         1780.37MB   100% |   bytes.(*Buffer).grow /usr/local/opt/go/libexec/src/bytes/buffer.go:142
 1780.37MB  1.67% 92.40%  1780.37MB  1.67%                | bytes.growSlice /usr/local/opt/go/libexec/src/bytes/buffer.go:240
----------------------------------------------------------+-------------
                                         1001.05MB 74.64% |   github.com/blevesearch/vellum.(*unfinishedNodes).popFreeze /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/vellum/builder.go:274
                                          340.06MB 25.36% |   github.com/blevesearch/vellum.(*unfinishedNodes).topLastFreeze /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/vellum/builder.go:296
 1341.11MB  1.25% 93.65%  1341.11MB  1.25%                | github.com/blevesearch/vellum.(*builderNodeUnfinished).lastCompiled /Users/51pwn/MyWork/goSqlite_gorm/vendor/github.com/blevesearch/vellum/builder.go:331
----------------------------------------------------------+-------------

@hktalent
Copy link
Author 

 $ go tool pprof  https://127.0.0.1:8081/debug/pprof/heap
Fetching profile over HTTP from https://127.0.0.1:8081/debug/pprof/heap
Saved profile in /Users/51pwn/pprof/pprof.alloc_objects.alloc_space.inuse_objects.inuse_space.004.pb.gz
Type: inuse_space
Time: Mar 30, 2023 at 9:15am (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 1291.54MB, 94.12% of 1372.16MB total
Dropped 135 nodes (cum <= 6.86MB)
Showing top 10 nodes out of 90
      flat  flat%   sum%        cum   cum%
  739.94MB 53.93% 53.93%   739.94MB 53.93%  github.com/blevesearch/zapx/v15.(*SegmentBase).loadFieldDocValueReader
  353.65MB 25.77% 79.70%   353.65MB 25.77%  bytes.growSlice
   83.20MB  6.06% 85.76%    83.20MB  6.06%  github.com/dgraph-io/badger/skl.newArena
   27.50MB  2.00% 87.77%    32.50MB  2.37%  net.ParseCIDR
   19.80MB  1.44% 89.21%    19.80MB  1.44%  github.com/golang/snappy.Decode
   17.62MB  1.28% 90.49%    17.62MB  1.28%  go.etcd.io/bbolt.(*freelist).reindex
      15MB  1.09% 91.59%    15.51MB  1.13%  github.com/blevesearch/bleve/v2/index/scorch/mergeplan.plan
   13.62MB  0.99% 92.58%    33.43MB  2.44%  github.com/blevesearch/zapx/v15.mergeStoredAndRemap
   13.13MB  0.96% 93.54%    18.64MB  1.36%  github.com/hktalent/51pwnPlatform/pkg/server/torrentImp.Visit.func1
    8.06MB  0.59% 94.12%     8.06MB  0.59%  github.com/RoaringBitmap/roaring.newBitmapContainer

@hktalent hktalent closed this as completed Apr 5, 2023
@almostinf
Copy link

Hello, @hktalent! Could you please advise what ended up solving the problem? I'm facing the exact same problem

@abhinavdangeti abhinavdangeti removed their assignment Jul 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@abhinavdangeti @hktalent @almostinf