Add new command line option --database-handles (#16796)

[juergenhoetzel]

This allows tuning the go-leveldb OpenFilesCacheCapacity option (capacity of
the open files) and thus improve performance by reducing the number of
open/close syscalls.

By increasing the number of DB handles to 100000 I was able to reduce the number of open/close syscalls (10 seconds interval) significantly from:

  open                                                            219
  close                                                           221

to

  close                                                            26
  open                                                             48

and thus increase the sync rate.

This Graphs shows the sync rate (Blocks/Minute) running with defaults (1024 handles) till 7:20 and --database-handles 100000 from 7:20.

[hackmod]

travis lint error found at this line. simply remove space, IntFlag { to IntFlag{ will work.

[juergenhoetzel]

HackyMiner writes:

@hackmod commented on this pull request. travis lint error found at this line. simply remove space, IntFlag { to IntFlag{ will work.

Thanks for the heads-up! I amended the commit.

[ghost]

Can't wait for the merge 😄

[holiman]

LGTM! I will run some benchmarks on this an post to the PR

[holiman]

So here are some stats from a fast-sync, with experimental being this PR, and master the current master branch.

• experimental
  • Fast sync started at 09:15, completed at 15:55: 6h40m
  • Filehandles rose to about 30K during the sync
• master
  • Fast sync started at 09:15, completed at 18:15: 9h0m
  • Filehandles stayed steady at around 2k during the sync

So the fast sync was 35% faster, which is very good (but it's just one datapoint so far).

[holiman]

A full sync is also in progress, and the results so far are not showing any similar improvement. The experimental is slightly ahead (30m or so), but that slight difference may just as well be related to differences in hardware.

While master is keeping open files at around 2K, the experimental has (over 24 hours) slowly grown roughly linearly to 18k so far.

I guess the file handle is more of a bottleneck during fast sync, and not so much as bottleneck during full sync.

[fjl]

We might be able to get similar benefits with less file handles by increasing the compaction table sizes in leveldb config. Right now the DB keeps thousands of small files and the access pattern is random so they're all accessed occasionally.

[mrFranklin]

good job!

This allows tuning the go-leveldb OpenFilesCacheCapacity option (capacity of
the open files) and thus improve performance by reducing the number of
open/close syscalls.

By increasing the number of DB handles to 100000 I was able to reduce the number of open/close syscalls (10 seconds interval) significantly from:

  open                                                            219
  close                                                           221

to

  close                                                            26
  open                                                             48

and thus increase the sync rate.

This Graphs shows the sync rate (Blocks/Minute) running with defaults (1024 handles) till 7:20 and --database-handles 100000 from 7:20.

HackyMiner writes:
@hackmod commented on this pull request. travis lint error found at this line. simply remove space, IntFlag { to IntFlag{ will work.
Thanks for the heads-up! I amended the commit.

good job!
could you tell me which tool do you use to generate the beautiful graph above?

[mrFranklin]

So here are some stats from a fast-sync, with experimental being this PR, and master the current master branch.

• experimental

  • Fast sync started at 09:15, completed at 15:55: 6h40m
  • Filehandles rose to about 30K during the sync

• master

  • Fast sync started at 09:15, completed at 18:15: 9h0m
  • Filehandles stayed steady at around 2k during the sync

So the fast sync was 35% faster, which is very good (but it's just one datapoint so far).

Could you tell me which tool can generate the graphs above? thanks

[holiman]

@mrFranklin it's from datadog. We export metrics through datadog to get nice charts from our benchmarking nodes.
@fjl so, what should we do about this? I think it seems to be a good PR, but if we can get similar improvements through some other means, I'm all ears...

[karalabe]

Superseded by #18211, which should solve the same problem, but automagically.

[hadv]

@mrFranklin it's from datadog. We export metrics through datadog to get nice charts from our benchmarking nodes.

@holiman Do you have a link that document detail steps to do this? Thank you!