Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store total key-value size in table footer #1137

Merged
merged 5 commits into from
Dec 10, 2019
Merged

Conversation

jarifibrahim
Copy link
Contributor

@jarifibrahim jarifibrahim commented Nov 27, 2019

This PR stores the total key-value size in a table in the table footer. The key-value size can be accessed via db.Tables(..) call. It returns the list of all tables along with the total size of key-values.


This change is Reviewable

@jarifibrahim jarifibrahim marked this pull request as ready for review November 27, 2019 14:45
@jarifibrahim jarifibrahim requested a review from a team November 27, 2019 14:45
@coveralls
Copy link

Coverage Status

Coverage decreased (-0.1%) to 77.414% when pulling a243b6c on ibrahim/table-kv-size into f5b6321 on master.

@coveralls
Copy link

coveralls commented Nov 27, 2019

Coverage Status

Coverage decreased (-7.6%) to 69.939% when pulling 4130bba on ibrahim/table-kv-size into f5b6321 on master.

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 7 of 7 files at r1.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @ashish-goswami and @jarifibrahim)


table/builder.go, line 132 at r1 (raw file):

	v.EncodeTo(b.buf)
	b.tableIndex.KvSize += uint64(uint32(headerSize) + uint32(len(diffKey)) + v.EncodedSize())

EstimatedSize

  • Size of KV on SST
  • Size of KV on Value Log
  • Sum these two.
  • Do not consider compression / encryption.

Or, perhaps do consider compression. Use the compressed size of the block and add the value log sizes to it.


table/table.go, line 445 at r1 (raw file):

// TotalKVSize returns the total size of key-values stored in this table.
func (t *Table) TotalKVSize() uint64 { return t.kvSize }

EstimatedSize

Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 11 of 11 files at r2.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @ashish-goswami and @jarifibrahim)

@jarifibrahim jarifibrahim merged commit f46f8ea into master Dec 10, 2019
@jarifibrahim jarifibrahim deleted the ibrahim/table-kv-size branch December 10, 2019 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants