change the way purge works #392

janardhan1993 · 2018-01-08T06:37:38Z

This change is

Ensure all versions of key are always written to same table.

manishrjain · 2018-01-10T05:50:15Z

Reviewed 1 of 7 files at r1, 7 of 7 files at r2.
Review status: all files reviewed at latest revision, 15 unresolved discussions, some commit checks broke.

db.go, line 896 at r1 (raw file):

	defer it.Close()

	var lastPurgeTs uint64

Do a blind write here. And do a select push into a channel, which can be run via a goroutine, which can update the GC stats.

db.go, line 976 at r1 (raw file):

		// or else same entry would be counted as discarded everytime we call PurgeOlderVersions.
		db.vlog.resetGCStats()
		for it.Rewind(); it.Valid(); it.Next() {

No need for all versions.

db.go, line 243 at r2 (raw file):

		flushChan:     make(chan flushTask, opt.NumMemtables),
		writeCh:       make(chan *request, kvWriteChCapacity),
		gcStatsCh:     make(chan updateGcStatTask, 10000),

1000

db.go, line 243 at r2 (raw file):

		flushChan:     make(chan flushTask, opt.NumMemtables),
		writeCh:       make(chan *request, kvWriteChCapacity),
		gcStatsCh:     make(chan updateGcStatTask, 10000),

purgeUpdateCh : chan purgeUpdate

db.go, line 317 at r2 (raw file):

	db.closers.gcStats = y.NewCloser(1)
	go db.periodicUpdateGCStats(db.closers.gcStats)

db.updateGCStats

db.go, line 461 at r2 (raw file):

// get returns the value in memtable or disk for given key.
// Note that value will include meta byte.
// IMP: We should never write an entry with a older timestamp for same key,

IMPORTANT

db.go, line 961 at r2 (raw file):

	}
	select {
	case db.gcStatsCh <- updateGcTask:

default:

db_test.go, line 326 at r1 (raw file):

	err = db.RunValueLogGC(0.2)
	require.NoError(t, err)

Explain in db.go why we should never write a key with an older version to LSM tree.

levels.go, line 673 at r1 (raw file):

			return vs, nil
		}
		if maxVs.Version < vs.Version {

When generating LSM tables, continue iterating until all the versions have been picked up by a single SSTable. That is, don't split a key between 2 tables on the same level.

levels.go, line 661 at r2 (raw file):

// get returns the found value if any. If not found, we return nil.
func (s *levelsController) get(key []byte) (vs y.ValueStruct, err error) {

Don't declare the return vars here.

levels.go, line 676 at r2 (raw file):

			continue
		}
		return

return vs, nil

levels.go, line 678 at r2 (raw file):

		return
	}
	return

return empty vs, nil?

value.go, line 319 at r1 (raw file):

	if err != nil {
		return false, err
	} else if isDeletedOrExpired(vs.Meta, vs.ExpiresAt) {

Add a comment //pass or explain why this.

value.go, line 314 at r2 (raw file):

}

func (vlog *valueLog) purgeEntry(key []byte, version uint64) (bool, error) {

Can just take the fully loaded keyWithTs, and parse the version and key out of it.

y/sparse_linux.go, line 28 at r1 (raw file):

// ErrPurged is returned when a transaction tries to access an entry which
// has been purged.
var ErrPurged = errors.New("This version of key has been purged")

full stop?

Comments from Reviewable

janardhan1993 · 2018-01-10T06:01:04Z

Review status: all files reviewed at latest revision, 15 unresolved discussions, some commit checks broke.

db.go, line 896 at r1 (raw file):