WithInMemory(true) panic: runtime error: index out of range [0] with length 0

[dpgarrick]

What version of Go are you using (go version)?

$ go version
go version go1.13.8 darwin/amd64

What version of Badger are you using?

2.0.2

Does this issue reproduce with the latest master?

Yes

What are the hardware specifications of the machine (RAM, OS, Disk)?

Model Identifier: MacBookPro16,1
Processor Name: 6-Core Intel Core i7
Processor Speed: 2.6 GHz
Number of Processors: 1
Total Number of Cores: 6
L2 Cache (per Core): 256 KB
L3 Cache: 12 MB
Hyper-Threading Technology: Enabled
Memory: 16 GB

SSD drive

What did you do?

Trying to store a ~2MB protobuf message in a badgerdb instance. Running badgerdb in disk mode and storing to ./db runs perfectly but changing the db instance to disk-less using WithInMemory(true) fails.

Decreasing the data size also runs fine in memory.

I've made a sparse repo just with source code reproducing the problem here:
https://github.com/dpgarrick/badgerdb-inmem

Should be able to reproduce via

go get github.com/dpgarrick/badgerdb-inmem
go run main.go manifest.pb.go > testOutput

and just commenting/uncommenting the database instance lines to go from disk to disk-less.

Unsure if I am just overlooking something obvious (pretty new to golang and badgerdb) but if I take only the first 40,000 lines of the data file instead of all ~88000 lines it runs fine WithInMemory(true) and it always runs fine when using badgerdb in disk mode so it seems more like a bug to me.

What did you expect to see?

Disk-less mode to run identical to the disk mode.

What did you see instead?

panic: runtime error: index out of range [0] with length 0

goroutine 23 [running]:
github.com/dgraph-io/badger.(*DB).writeToLSM(0xc00078a800, 0xc0015bc720, 0x0, 0x0)
	/Users/dpg/go/src/github.com/dgraph-io/badger/db.go:667 +0x341
github.com/dgraph-io/badger.(*DB).writeRequests(0xc00078a800, 0xc00013a050, 0x1, 0xa, 0x0, 0x0)
	/Users/dpg/go/src/github.com/dgraph-io/badger/db.go:720 +0x364
github.com/dgraph-io/badger.(*DB).doWrites.func1(0xc00013a050, 0x1, 0xa)
	/Users/dpg/go/src/github.com/dgraph-io/badger/db.go:762 +0x5a
created by github.com/dgraph-io/badger.(*DB).doWrites
	/Users/dpg/go/src/github.com/dgraph-io/badger/db.go:815 +0x329
exit status 2

[jarifibrahim]

Thanks for raising this @dpgarrick. This definitely looks like a bug to me. I'll look into it.

[jarifibrahim]

@dpgarrick I've raised #1235 to fix this. The issue was that I had set a hard limit of 1 MB on the maximum size of value in in-memory mode. Your code was inserting value of size 2 MB and that's why badger crashed. I've changed the code so that the maximum size of value in in-memory mode will be math.MaxInt32 (around 2 GB).

[dpgarrick]

@jarifibrahim Great thanks, I wondered if it was something simple like that.

2GB should be more than enough but also nice to know I can just increase the options.ValueThreshold to address.

[jarifibrahim]

@dpgarrick If you're running badger in in-memory mode, the value of ValueTheshold will not be used. We over-write it while opening DB.

badger/db.go

Lines 309 to 312 in 617ed7c

	if db.opt.InMemory {
	db.opt.SyncWrites = false
	db.opt.ValueThreshold = maxValueThreshold
	}

[dpgarrick]

@jarifibrahim Ahh right, thanks for the clarification.