Introduce in-memory mode in badger

[jarifibrahim]

This PR introduces disk-less mode in badger. The disk-less mode can be enabled by setting options.DiskLess = true. When badger is running in disk-less mode no files are created and everything is stored in memory.

On DB close, all stored data is lost.

NOTE - An existing DB cannot be opened in diskless mode.

Fixes #1001

---

This change is 

[coveralls]

Coverage increased (+0.3%) to 77.625% when pulling 1b4b6eb on ibrahim/diskless-mode into 3eb4e72 on master.

[manishrjain]

Use a unix tool which tracks file system calls, and ensure that Badger is not making any such calls in InMemory mode.

Reviewed 10 of 15 files at r1, 3 of 3 files at r2, 4 of 4 files at r5.
Reviewable status: all files reviewed, 11 unresolved discussions (waiting on @ashish-goswami and @jarifibrahim)

---

db.go, line 191 at r5 (raw file):

// Open returns a new DB object.
func Open(opt Options) (db *DB, err error) {
	if opt.DiskLess && (opt.Dir != "" || opt.ValueDir != "") {

opt.InMemory

---

db.go, line 635 at r5 (raw file):

func (db *DB) shouldWriteValueToLSM(e Entry) bool {
	return db.opt.DiskLess || (len(e.Value) < db.opt.ValueThreshold)

Avoid this. Just set ValueThreshold to max upfront.

---

db.go, line 685 at r5 (raw file):

	// When DB is running in diskless mode, we don't write anything to the value log.
	// The value log doesn't exists, there are no files on the disk.
	if !db.opt.DiskLess {

don't need this if.

---

db.go, line 1120 at r5 (raw file):

func (db *DB) RunValueLogGC(discardRatio float64) error {
	if db.opt.DiskLess {
		return errors.New("Cannot run value log GC when DB is opened in diskless mode")

Create that error variable upfront.

---

levels.go, line 256 at r5 (raw file):

		}
		changeSet := pb.ManifestChangeSet{Changes: changes}
		if err := s.kv.manifest.addChanges(changeSet.Changes); err != nil {

Move the changes to manifest.

---

options.go, line 53 at r5 (raw file):

	Compression         options.CompressionType
	EventLogging        bool
	DiskLess            bool

InMemory bool

---

options.go, line 551 at r5 (raw file):

// created. In case of a crash all data will be lost.
func (opt Options) WithDiskLess(b bool) Options {
	opt.DiskLess = b

opt.path = "", opt.vpath = "", opt.Sync = false, opt.ValueThreshold = max possible.

---

stream_writer.go, line 248 at r5 (raw file):

	// Now sync the directories, so all the files are registered.
	if !sw.db.opt.DiskLess && (sw.db.opt.ValueDir != sw.db.opt.Dir) {
		if err := syncDir(sw.db.opt.ValueDir); err != nil {

There's a side-effect logic change, where we sync the same dir twice (if vdir == dir). Instead, syncDir should be the one deciding what it needs to do.

---

txn.go, line 303 at r5 (raw file):

		size = txn.size + int64(len(e.Key)) + int64(len(e.Value)) + 2 /* Meta and user meta */
	} else {
		size = txn.size + int64(e.estimateSize(txn.db.opt.ValueThreshold))

Not required.

---

value.go, line 1245 at r5 (raw file):

// fid < vlog.maxFid. To sync irrespective of file id just call it with math.MaxUint32.
func (vlog *valueLog) sync(fid uint32) error {
	if vlog.opt.SyncWrites || vlog.db.opt.DiskLess {

No need to have this change.

[jarifibrahim]

Dismissed @golangcibot from a discussion.
Reviewable status: 2 of 19 files reviewed, 10 unresolved discussions (waiting on @ashish-goswami and @manishrjain)

---

db.go, line 310 at r3 (raw file):

Previously, golangcibot (Bot from GolangCI) wrote…

printf: Println call has possible formatting directive %s (from govet)

Done.

---

db.go, line 191 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

opt.InMemory

Done.

---

db.go, line 635 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Avoid this. Just set ValueThreshold to max upfront.

Done.

---

db.go, line 685 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

don't need this if.

Done.

---

db.go, line 1120 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Create that error variable upfront.

Done.

---

levels.go, line 256 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Move the changes to manifest.

Done.

---

options.go, line 53 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

InMemory bool

Done.

---

options.go, line 551 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

opt.path = "", opt.vpath = "", opt.Sync = false, opt.ValueThreshold = max possible.

These values are being set in the db.Open() function. If we set them here they might get overwritten by user.

---

stream_writer.go, line 248 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

There's a side-effect logic change, where we sync the same dir twice (if vdir == dir). Instead, syncDir should be the one deciding what it needs to do.

Yeah. Fixed.

---

txn.go, line 303 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Not required.

Done.

---

value.go, line 1245 at r5 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

No need to have this change.

Done.

[manishrjain]

Reviewed 8 of 17 files at r6, 9 of 9 files at r7.
Reviewable status: all files reviewed, 11 unresolved discussions (waiting on @ashish-goswami, @jarifibrahim, and @manishrjain)

---

db.go, line 266 at r7 (raw file):

	}

	manifestFile, manifest, err := openOrCreateManifestFile(opt.Dir, opt.ReadOnly, opt.InMemory)

pass in opt.