Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Orphan block scanner scripts [2.1] #1133

Merged
merged 6 commits into from
Apr 28, 2015
Merged

Orphan block scanner scripts [2.1] #1133

merged 6 commits into from
Apr 28, 2015

Conversation

shino
Copy link
Contributor

@shino shino commented Apr 28, 2015

Rebased version of #1128 for some reason.

Steps for scanning orphan blocks

  1. List keys in moss.buckets
  2. Iterate over all CS buckets (optionally ones listed in command line)
    1. List keys (streaming listkeys) on a <<0b:...>> bucket
      • We MUST list keys of blocks at first to avoid race from API calls
    2. Collect all blocks, as {UUID, [list of existing block seqs]}
    3. List manifest UUIDs on <<0o:...>> bucket (by streaming csbucketfold)
    4. Check existence of corresponding manifest by subtracting the two
      results of list keys
    5. Complete maybe orphaned UUID list
  3. Filter out false-orphaned block UUIDs.
    This is because list keys (or csbucketfold) of manifest buckets will
    be done by R=1, then replica deficit (e.g. by force-replace) leads to
    false-positiveness for orphan block extraction at previous step.
    R=quorum (or R=all) manifest GETs is needed to avoid false-positive.
    1. Get block object and get CS key of manifests from user metadata
    2. Get manifest for the key and verify existence of corresponding manifest
    3. Remove block UUIDs if manifest exists from orphaned list

Usage (for debugging)

% riak-cs escript /path/to/priv/tools/internal/block_audit.erl  \
      -h 127.0.0.1 -p 10017 \
      -dd \
      -o $PWD/blocks.local
% riak-cs escript /path/to/priv/tools/internal/ensure_orphan_blocks.erl \
      -h 127.0.0.1 -p 10017 \
      -dd \
      -i $PWD/blocks.local -o $PWD/blocks2.local

This PR adds scripts for gathering block {UUID, Sequence}s to be deleted.
For actual deletion, we can re-using block cleaner script in #1124.
But there is still some interface (file format) discrepancy, subsequent
PR is needed.

shino added 4 commits April 28, 2015 11:02
@shino
Import block audit script
7e76df0 
Originally written by Andrew J. Stone
@shino
Add handling of Multipart part UUIDs and some improvement for speed, …
2c8c38e 
…etc.

- Remove subtraction of UUIDs in GC bucket

  For the purpose to find leaked UUIDs, subtracting UUIDs in GC bucket
  is correct. However, it does no harm to remove blocks in GC bucket
  because they are already deleted from users' point of view.
  To remove subtraction, correctly tracked (not leaked) UUIDs may
  be included in the result, but we can reduce execution time of the
  script by skipping GC bucket scan.

- Use ETS folding to output to file
- Add options for specify buckets and skip unnecessary operations for efficiency
- Use getopt and lager
@shino
Add script to filter out false-orphaned blocks
4c0f714
@shino
Add riak_test case for block audit scripts
231888a
@shino shino added this to the 2.1.0 milestone Apr 28, 2015
kuenishi
kuenishi reviewed Apr 28, 2015
View reviewed changes
priv/tools/internal/block_audit.erl
{ok, Pid} ->
pong = riakc_pb_socket:ping(Pid),
audit(Pid, Options),
timer:sleep(100);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this sleep for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be old/historical workaround now. R15B01 (too old one 😅 ) fixed standard out flush issue when escript exits [1].
There is (small) possibility to backport this script to CS 1.5 env, so if you don't mind, I want to
leave this line as is. Thoughts?

[1] OTP-9985 http://www.erlang.org/download/otp_src_R15B01.readme

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

np. Let's leave as it is.

@kuenishi
Copy link
Contributor

Code is perfect, tests passed - while some riak_tests around upgrade/downgrade still failing, but that'd be another issue. Well done, man.

@shino
Copy link
Contributor Author

shino commented Apr 28, 2015

Thanks for review 😄 Addressed the comments and pushed ⏫

borshop added a commit that referenced this pull request Apr 28, 2015
@borshop
Merge pull request #1133 from basho/feature/block-audit
06be181 
Orphan block scanner scripts [2.1]

Reviewed-by: kuenishi
@kuenishi
Copy link
Contributor

@borshop merge

@borshop borshop merged commit 88fd61f into develop Apr 28, 2015
@kuenishi kuenishi deleted the feature/block-audit branch April 28, 2015 13:05
@kuenishi kuenishi modified the milestones: 2.2.0, 2.1.0 Apr 28, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
2.1.0
Development

Successfully merging this pull request may close these issues.
3 participants
@shino @kuenishi @borshop