backend: implement disk quota #493

kennytm · 2020-11-26T23:24:03Z

What problem does this PR solve?

Implement #446.

What is changed and how it works?

Check List

Tests

Unit test
Integration test

Side effects

Possible performance regression
Increased code complexity

Related changes

Need to update the documentation
Need to be included in the release note

Little-Wallace · 2020-12-01T06:09:04Z

lightning/backend/backend.go

+	sort.Slice(sizes, func(i, j int) bool {
+		a, b := &sizes[i], &sizes[j]
+		if a.IsImporting != b.IsImporting {
+			return a.IsImporting


What does it mean? Do you want to sort all importing engine before other engines?

Yes. We can't import engines that are already importing, but their size do count towards the disk quota.

kennytm · 2020-12-02T20:39:02Z

The test case depends on #505 in order to quickly generate huge SSTs without checking in a 1 GB database or introduce dbgen.

lightning/backend/local.go

kennytm · 2021-01-19T11:10:13Z

lightning/backend/local.go

+	})
+	return
+}
+
 type LocalWriter struct {


PTAL @Little-Wallace, the LocalWriter is basically entirely rewritten, there may be some performance implication of the original code that I may have missed. Thanks.

glorv

rest LGTM

lightning/backend/local.go

glorv · 2021-01-20T09:02:59Z

lightning/restore/restore.go

-			return
-		}
+		func() {
+			rc.diskQuotaLock.RLock()


yes, the current implementation can not handle this if the previous check isImporting is false and the disk-quote loop holds the lock and do importing.

lightning/backend/importer.go

lightning/backend/local.go

kennytm

~~test~~

lightning/backend/local.go

Little-Wallace · 2021-01-21T06:58:05Z

lightning/backend/local.go

+
+// append more KV pairs to the kvMemCache.
+func (m *kvMemCache) append(kvs []common.KvPair) {
+	if !m.notSorted {


Even if the kvMemCache has been sorted, we shall still check the order of kvs. The array kvs may be unordered and the first key of kvs may be greater than the lastkey of m.kvs

the loop inside does check the order of kvs.

var lastKey []byte if len(m.kvs) > 0 { lastKey = m.kvs[len(m.kvs)-1].Key } for _, kv := range kvs { if bytes.Compare(kv.Key, lastKey) <= 0 { // <---- m.notSorted = true break } lastKey = kv.Key // <---- }

lance6716 · 2021-01-29T05:48:22Z

/lgtm

glorv · 2021-01-29T09:19:13Z

/lgtm

glorv · 2021-01-29T09:19:42Z

/merge

ti-srebot · 2021-01-29T09:22:12Z

/run-all-tests

ti-srebot · 2021-01-29T09:32:11Z

@kennytm merge failed.

glorv · 2021-01-29T09:46:50Z

/run-all-tests

kennytm added the status/WIP Work in progress label Nov 26, 2020

Little-Wallace reviewed Dec 1, 2020

View reviewed changes

kennytm added status/DNM Do not merge, test is failing or blocked by another PR and removed status/WIP Work in progress labels Dec 2, 2020

kennytm force-pushed the kennytm/disk-quota branch 2 times, most recently from 3f12d8c to f6f87fc Compare December 2, 2020 20:31

kennytm marked this pull request as ready for review December 2, 2020 20:43

kennytm changed the title ~~[WIP] Implement disk quota~~ backend: implement disk quota Dec 5, 2020

kennytm added 12 commits December 6, 2020 07:22

common: copied the GetStorageSize function from DM

cec3d30

common: recognize multierr in IsRetryableError()

147851e

restore: refactor runPeriodicActions

bd02b59

config: fix test failure on Windows

360c491

*: implement disk quota

3a605bc

backend: split the disk size and mem size metrics

6afb02d

backend,restore: refactor, move the postProcessLock into the backend

657adea

go.mod1: update

1d7a411

backend: introduce a conversion factor between memtable and disk size

95891e1

backend: force the aes-256-cbc before anyone notices ;)

7bf5fbf

tests: split FailAfterWriteRows into two failpoints

cc65433

tests: add disk-quota test

dad23f5

kennytm force-pushed the kennytm/disk-quota branch from f6f87fc to dad23f5 Compare December 5, 2020 23:27

kennytm added status/PTAL This PR is ready for review. Add this label back after committing new changes and removed status/DNM Do not merge, test is failing or blocked by another PR labels Dec 8, 2020

lance6716 reviewed Dec 8, 2020

View reviewed changes

lightning/backend/local.go Outdated Show resolved Hide resolved

lance6716 reviewed Dec 8, 2020

View reviewed changes

lightning/backend/local.go Show resolved Hide resolved

kennytm added 2 commits December 8, 2020 23:39

Merge branch 'master' into kennytm/disk-quota

2575452

backend/local: replace the semaphore by a regular mutex

626f480

kennytm force-pushed the kennytm/disk-quota branch from 3636dc5 to 626f480 Compare December 8, 2020 17:04

tests/disk_quota: add that 2-second loop for SET GLOBAL

1ed6164

kennytm force-pushed the kennytm/disk-quota branch from 7f8a286 to f274838 Compare January 19, 2021 09:30

tests: use a unique sorted-kv-dir per test

66a250c

kennytm force-pushed the kennytm/disk-quota branch from f274838 to 66a250c Compare January 19, 2021 09:58

kennytm removed the status/WIP Work in progress label Jan 19, 2021

backend/local: fixed some doc comments

6dd5ef3

kennytm commented Jan 19, 2021

View reviewed changes

glorv reviewed Jan 20, 2021

View reviewed changes

lance6716 reviewed Jan 20, 2021

View reviewed changes

lightning/backend/importer.go Outdated Show resolved Hide resolved

lightning/backend/local.go Outdated Show resolved Hide resolved

lightning/backend/local.go Outdated Show resolved Hide resolved

backend: addressed comments

ca808b2

kennytm commented Jan 20, 2021

View reviewed changes

lance6716 reviewed Jan 21, 2021

View reviewed changes

lightning/backend/local.go Outdated Show resolved Hide resolved

lightning/backend/local.go Outdated Show resolved Hide resolved

Little-Wallace reviewed Jan 21, 2021

View reviewed changes

kennytm added 3 commits January 21, 2021 15:55

backend/local: just always save meta every time we flush

e401ffe

Merge branch 'master' into kennytm/disk-quota

6309d13

backend: addressed comments

98e02a7

ti-srebot removed the status/LGT1 One reviewer already commented LGTM (LGTM1) label Jan 29, 2021

ti-srebot approved these changes Jan 29, 2021

View reviewed changes

ti-srebot added the status/LGT2 Two reviewers already commented LGTM, ready for merge (LGTM2) label Jan 29, 2021

ti-srebot added status/LGT3 and removed status/LGT2 Two reviewers already commented LGTM, ready for merge (LGTM2) labels Jan 29, 2021

ti-srebot added the CanMerge label Jan 29, 2021

Merge branch 'master' into kennytm/disk-quota

93beae8

glorv merged commit 1f12021 into master Jan 29, 2021

glorv deleted the kennytm/disk-quota branch January 29, 2021 10:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

backend: implement disk quota #493

backend: implement disk quota #493

kennytm commented Nov 26, 2020

Little-Wallace Dec 1, 2020

kennytm Dec 1, 2020 •

edited

Loading

kennytm commented Dec 2, 2020

kennytm Jan 19, 2021

glorv left a comment

glorv Jan 20, 2021

kennytm left a comment •

edited

Loading

Little-Wallace Jan 21, 2021

kennytm Jan 21, 2021

lance6716 commented Jan 29, 2021

glorv commented Jan 29, 2021

glorv commented Jan 29, 2021

ti-srebot commented Jan 29, 2021

ti-srebot commented Jan 29, 2021

glorv commented Jan 29, 2021

backend: implement disk quota #493

backend: implement disk quota #493

Conversation

kennytm commented Nov 26, 2020

What problem does this PR solve?

What is changed and how it works?

Check List

Little-Wallace Dec 1, 2020

Choose a reason for hiding this comment

kennytm Dec 1, 2020 • edited Loading

Choose a reason for hiding this comment

kennytm commented Dec 2, 2020

kennytm Jan 19, 2021

Choose a reason for hiding this comment

glorv left a comment

Choose a reason for hiding this comment

glorv Jan 20, 2021

Choose a reason for hiding this comment

kennytm left a comment • edited Loading

Choose a reason for hiding this comment

Little-Wallace Jan 21, 2021

Choose a reason for hiding this comment

kennytm Jan 21, 2021

Choose a reason for hiding this comment

lance6716 commented Jan 29, 2021

glorv commented Jan 29, 2021

glorv commented Jan 29, 2021

ti-srebot commented Jan 29, 2021

ti-srebot commented Jan 29, 2021

glorv commented Jan 29, 2021

kennytm Dec 1, 2020 •

edited

Loading

kennytm left a comment •

edited

Loading