Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dump tool. #1479

Merged
merged 12 commits into from
Dec 31, 2019
Merged

Dump tool. #1479

merged 12 commits into from
Dec 31, 2019

Conversation

CPWstatic
Copy link
Contributor

This tool is about to help to dump the records or statistics for a space from rocksdb directly. Please take a look about the usage, any suggestion is appreciated.

  ./db_dump --space=<space name>

required:
       --space=<space name>
         A space name must be given.

optional:
       --db_path=<path to rocksdb>
         Path to the rocksdb data directory. If nebula was installed in /usr/local/nebula, 
         the db_path would be /usr/local/nebula/data/storage/nebula/
         Default: ./

       --meta_server=<ip:port,...>
         A list of meta severs' ip:port seperated by comma.
         Default: 127.0.0.1:45500

       --mode= scan | stat
         scan: print to screen when records meet the condition, and also print statistics 
               to screen in final.
         stat: print statistics to screen.
         Defualt: scan

       --vids=<list of vid>
         A list of vid seperated by comma.
         Would scan the whole space's records if it is not given.

       --parts=<list of partition id>
         A list of partition id seperated by comma.
         Would output all patitions if it is not given.

       --tags=<list of tag name>
         A list of tag name seperated by comma.

       --edges=<list of edge name>
         A list of edge name seperated by comma.

       --limit=<N>
         A positive number that limits the output.
         Would output all if set 0 or negative number.

Example:

  • If you want statistics for a space
[william.chen@hp-server build]$ ./src/tools/db-dump/_build/db_dump --space=meituan --meta_server=192.168.8.5:46513 --db_path=/home/william.chen/nebula_run/data/storage/nebula/ --mode=stat --limit=0
=======================================================
mode: stat
seta server: 192.168.8.5:46513
space: meituan
path: /home/william.chen/nebula_run/data/storage/nebula/
parts: 
vids: 
tags: 
edges: 
limit: 0
=======================================================
=======================================================
COUNT: 9978598
VERTEX COUNT: 1001002
EDGE COUNT: 8977596
TAG STATISTICS: 
        user : 1000000
        poi : 1002
EDGE STATISTICS: 
        consume_poi : 4488664
        consume_poi_reverse : 4488932
=======================================================
  • if you want dump some records
[william.chen@hp-server build]$ ./src/tools/db-dump/_build/db_dump --space=meituan --meta_server=192.168.8.5:46513 --db_path=/home/william.chen/nebula_run/data/storage/nebula/ --mode=scan  --limit=10
=======================================================
mode: scan
seta server: 192.168.8.5:46513
space: meituan
path: /home/william.chen/nebula_run/data/storage/nebula/
parts: 
vids: 
tags: 
edges: 
limit: 10
=======================================================
[vertex] key: 1, 0, poi value:mid:0,8191765721868409651,8025713627522363385,1993089399535188613,3926276052777355165,5123607763506443893,2990089379644866415,poi_name_0,上海,华东,30.2824,120.016,poi_stat_0,poi_fc_0,poi_sc_0,0,poi_star_0,
[edge] key: 1, 0, consume_poi_reverse, 0, 656384 value:mid:656384,mid:0,7.19312,mid:656384,3897457441682646732,mun:656384,4038264117233984707,dun:656384,empe:656384,mobile:656384,gender:656384,age:656384,rs:656384,fpd:656384,0.75313,1.34433,fpd:656384,0.03567,7.56212,
[edge] key: 1, 0, consume_poi_reverse, 0, 853504 value:mid:853504,mid:0,0.48533,mid:853504,5913078351068176125,mun:853504,9057247552038005785,dun:853504,empe:853504,mobile:853504,gender:853504,age:853504,rs:853504,fpd:853504,2.98033,7.90627,fpd:853504,9.6636,9.37384,
[edge] key: 1, 0, consume_poi_reverse, 0, 469760 value:mid:469760,mid:0,3.23066,mid:469760,7207250750188099999,mun:469760,4810157463127675320,dun:469760,empe:469760,mobile:469760,gender:469760,age:469760,rs:469760,fpd:469760,0.00561,7.74479,fpd:469760,4.08251,1.51453,
[edge] key: 1, 0, consume_poi_reverse, 0, 601856 value:mid:601856,mid:0,7.47982,mid:601856,6697181152403783432,mun:601856,7276549530908530381,dun:601856,empe:601856,mobile:601856,gender:601856,age:601856,rs:601856,fpd:601856,4.44091,8.72814,fpd:601856,4.797,2.48794,
[edge] key: 1, 0, consume_poi_reverse, 0, 145408 value:mid:145408,mid:0,5.4381,mid:145408,7161523474603122361,mun:145408,5500149701677167180,dun:145408,empe:145408,mobile:145408,gender:145408,age:145408,rs:145408,fpd:145408,2.68552,0.75738,fpd:145408,1.39399,9.81429,
[edge] key: 1, 0, consume_poi_reverse, 0, 678912 value:mid:678912,mid:0,6.16487,mid:678912,937598200293099198,mun:678912,1472268951595502012,dun:678912,empe:678912,mobile:678912,gender:678912,age:678912,rs:678912,fpd:678912,2.94393,6.16802,fpd:678912,4.46143,7.63295,
[edge] key: 1, 0, consume_poi_reverse, 0, 93952 value:mid:93952,mid:0,5.52997,mid:93952,8994248903787227040,mun:93952,232800446877859067,dun:93952,empe:93952,mobile:93952,gender:93952,age:93952,rs:93952,fpd:93952,4.16682,5.80782,fpd:93952,3.75861,8.14151,
[edge] key: 1, 0, consume_poi_reverse, 0, 684288 value:mid:684288,mid:0,9.26951,mid:684288,6042250945445127247,mun:684288,8498942626451711080,dun:684288,empe:684288,mobile:684288,gender:684288,age:684288,rs:684288,fpd:684288,5.86881,4.82076,fpd:684288,0.02854,8.68384,
[edge] key: 1, 0, consume_poi_reverse, 0, 558592 value:mid:558592,mid:0,4.05633,mid:558592,8996879966948165152,mun:558592,2580898387119818270,dun:558592,empe:558592,mobile:558592,gender:558592,age:558592,rs:558592,fpd:558592,9.38898,4.50945,fpd:558592,9.39891,2.67842,
=======================================================
COUNT: 10
VERTEX COUNT: 1
EDGE COUNT: 9
TAG STATISTICS: 
        poi : 1
EDGE STATISTICS: 
        consume_poi_reverse : 9
=======================================================

@CPWstatic CPWstatic changed the title Db dump Dump tool. Dec 19, 2019
@CPWstatic CPWstatic added the ready-for-testing PR: ready for the CI test label Dec 20, 2019
@CPWstatic
Copy link
Contributor Author

Jenkins go!

@nebula-community-bot
Copy link
Member

Unit testing passed.

@nebula-community-bot
Copy link
Member

Unit testing passed.

}
}

std::cout << "===========================STATISTICS============================\n";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why don't use LOG(INFO)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we need log in such a tool?

@nebula-community-bot
Copy link
Member

Unit testing failed.

@CPWstatic
Copy link
Contributor Author

Jenkins go!

@nebula-community-bot
Copy link
Member

Unit testing passed.

Comment on lines +170 to +177
uint32_t bitmap =
(parts_.empty() ? 0 : 1 << 3)
| (vids_.empty() ? 0 : 1 << 2)
| (tags_.empty() ? 0 : 1 << 1)
| (edges_.empty() ? 0 : 1);
switch (bitmap) {
case 0b0000: {
// nothing specified,seek to first and print them all
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like this !

src/tools/db-dump/DbDumper.cpp Outdated Show resolved Hide resolved
src/tools/db-dump/DbDumper.cpp Outdated Show resolved Hide resolved
src/tools/db-dump/DbDumper.cpp Outdated Show resolved Hide resolved
src/tools/db-dump/DbDumper.h Show resolved Hide resolved
Comment on lines +516 to +521
auto name = schemaMng_->toTagName(spaceId_, tagId);
if (!name.ok()) {
return folly::stringPrintf("%d", tagId);
} else {
return name.value();
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha, A good idea ! Same with a film《Inception》😄

@bright-starry-sky
Copy link
Contributor

Overall, I think this PR is very good !

@nebula-community-bot
Copy link
Member

Unit testing passed.

@nebula-community-bot
Copy link
Member

Unit testing passed.

Copy link
Contributor

@bright-starry-sky bright-starry-sky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done. great job !

@jude-zhu
Copy link
Contributor

close #194

@jude-zhu jude-zhu mentioned this pull request Dec 25, 2019
src/meta/client/MetaClient.cpp Outdated Show resolved Hide resolved
src/tools/db-dump/DbDumper.cpp Show resolved Hide resolved
src/tools/db-dump/DbDumper.cpp Show resolved Hide resolved
@jude-zhu
Copy link
Contributor

close #194

laura-ding
laura-ding previously approved these changes Dec 27, 2019
@nebula-community-bot
Copy link
Member

Unit testing failed.

Copy link
Contributor

@laura-ding laura-ding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done

@nebula-community-bot
Copy link
Member

Unit testing passed.

@amber-moe amber-moe mentioned this pull request Dec 30, 2019
@nebula-community-bot
Copy link
Member

Unit testing passed.

@dutor dutor merged commit 4ed1882 into vesoft-inc:master Dec 31, 2019
tong-hao pushed a commit to tong-hao/nebula that referenced this pull request Jun 1, 2021
* Add dumper.

* Add dump.

* Add statistics.

* Update.

* Add to_tag_name.

* Rebase and update.

* Address @bright-starry-sky's comment.

* Address @laura-ding's comment.

* Delete FLAGS_load_data_interval_secs.

Co-authored-by: bright-starry-sky <56461666+bright-starry-sky@users.noreply.github.com>
Co-authored-by: dutor <440396+dutor@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready-for-testing PR: ready for the CI test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants