[Keep open] Benchmark log #31

raulk · 2018-08-31T12:47:16Z

This issue tracks benchmark results as we continue tuning the peerstore. It will remain open until such efforts subside.

Please post the commit hash you're testing against and your CPU information.

raulk · 2018-08-31T12:55:46Z

Commit b1e626f, go 1.11, after running gx-go rw. Three runs:

→ sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:master ✖︎ [16:39:45]
→ go test -run=XXX -bench=.
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore
BenchmarkPeerstore/PeerstoreBasic-8         	  200000	     11118 ns/op
2018/09/03 16:39:49 Replaying from value pointer: {Fid:0 Len:0 Offset:0}
2018/09/03 16:39:49 Iterating file id: 0
2018/09/03 16:39:49 Iteration took: 16.665µs
BenchmarkPeerstore/PeerstoreDatastore-8     	   10000	    180383 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	4.305s

# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:master ✖︎ [16:39:50]
→ go test -run=XXX -bench=.
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore
BenchmarkPeerstore/PeerstoreBasic-8         	  200000	     11214 ns/op
2018/09/03 16:39:54 Replaying from value pointer: {Fid:0 Len:0 Offset:0}
2018/09/03 16:39:54 Iterating file id: 0
2018/09/03 16:39:54 Iteration took: 48.486µs
BenchmarkPeerstore/PeerstoreDatastore-8     	   10000	    180221 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	4.310s

# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:master ✖︎ [16:39:56]
→ go test -run=XXX -bench=.
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore
BenchmarkPeerstore/PeerstoreBasic-8         	  200000	     11399 ns/op
2018/09/03 16:40:00 Replaying from value pointer: {Fid:0 Len:0 Offset:0}
2018/09/03 16:40:00 Iterating file id: 0
2018/09/03 16:40:00 Iteration took: 44.098µs
BenchmarkPeerstore/PeerstoreDatastore-8     	   10000	    181797 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	4.373s

raulk · 2018-09-03T14:55:18Z

After optimising benchmark code, go 1.11:

→ sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz

# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:e59d8fc ✖︎ [16:41:53]
→ go test -run=XXX -bench=. ./...
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	0.032s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.006s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/ds
BenchmarkBadgerDsPeerstore/Peerstore-8         	   10000	    144338 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/ds	1.529s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/mem
BenchmarkInMemoryPeerstore/Peerstore-8         	  100000	     17512 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/mem	1.966s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/queue	0.035s
?   	github.com/libp2p/go-libp2p-peerstore/test	[no test files]

# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:e59d8fc ✖︎ [16:41:58]
→ go test -run=XXX -bench=. ./...
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	0.031s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.007s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/ds
BenchmarkBadgerDsPeerstore/Peerstore-8         	   10000	    145296 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/ds	1.533s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/mem
BenchmarkInMemoryPeerstore/Peerstore-8         	  100000	     17621 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/mem	1.979s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/queue	0.033s
?   	github.com/libp2p/go-libp2p-peerstore/test	[no test files]

# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:e59d8fc ✖︎ [16:42:03]
→ go test -run=XXX -bench=. ./...
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	0.032s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.006s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/ds
BenchmarkBadgerDsPeerstore/Peerstore-8         	   10000	    146030 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/ds	1.537s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/mem
BenchmarkInMemoryPeerstore/Peerstore-8         	  100000	     17413 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/mem	1.961s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/queue	0.033s
?   	github.com/libp2p/go-libp2p-peerstore/test	[no test files]

raulk · 2018-09-03T15:01:49Z

After migrating to TxnDatastore, go 1.11, with transactions in closures:

# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:txndatastore ✖︎ [16:42:56]
→ go test -run=XXX -bench=. ./...
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	0.014s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.008s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/mem
BenchmarkInMemoryPeerstore/Peerstore-8         	  100000	     17659 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/mem	1.971s
2018/09/03 16:43:01 Replaying from value pointer: {Fid:0 Len:0 Offset:0}
2018/09/03 16:43:01 Iterating file id: 0
2018/09/03 16:43:01 Iteration took: 22.089µs
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/pstoreds
BenchmarkBadgerDsPeerstore/Peerstore-8         	   10000	    149227 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/pstoreds	1.614s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/queue	0.015s
?   	github.com/libp2p/go-libp2p-peerstore/test	[no test files]

# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:txndatastore ✖︎ [16:43:03]
→ go test -run=XXX -bench=. ./...
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	0.014s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.007s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/mem
BenchmarkInMemoryPeerstore/Peerstore-8         	  100000	     17690 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/mem	1.972s
2018/09/03 16:43:08 Replaying from value pointer: {Fid:0 Len:0 Offset:0}
2018/09/03 16:43:08 Iterating file id: 0
2018/09/03 16:43:08 Iteration took: 24.236µs
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/pstoreds
BenchmarkBadgerDsPeerstore/Peerstore-8         	   10000	    151545 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/pstoreds	1.585s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/queue	0.016s
?   	github.com/libp2p/go-libp2p-peerstore/test	[no test files]

# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:txndatastore ✖︎ [16:43:09]
→ go test -run=XXX -bench=. ./...
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	0.014s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.007s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/mem
BenchmarkInMemoryPeerstore/Peerstore-8         	  100000	     17357 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/mem	1.942s
2018/09/03 16:43:13 Replaying from value pointer: {Fid:0 Len:0 Offset:0}
2018/09/03 16:43:13 Iterating file id: 0
2018/09/03 16:43:13 Iteration took: 26.184µs
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/pstoreds
BenchmarkBadgerDsPeerstore/Peerstore-8         	   10000	    148695 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/pstoreds	1.557s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/queue	0.015s
?   	github.com/libp2p/go-libp2p-peerstore/test	[no test files]

raulk · 2018-09-03T15:31:37Z

Procedural style, go 1.11 -- no practical difference. Will make use of the functional style.

# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:txndatastore-nofunctionaltxs ✖︎ [16:43:59]
→ go test -run=XXX -bench=. ./...
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	0.014s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.007s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/mem
BenchmarkInMemoryPeerstore/Peerstore-8         	  100000	     17612 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/mem	1.963s
2018/09/03 16:44:04 Replaying from value pointer: {Fid:0 Len:0 Offset:0}
2018/09/03 16:44:04 Iterating file id: 0
2018/09/03 16:44:04 Iteration took: 24.742µs
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/pstoreds
BenchmarkBadgerDsPeerstore/Peerstore-8         	   10000	    145249 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/pstoreds	1.522s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/queue	0.016s
?   	github.com/libp2p/go-libp2p-peerstore/test	[no test files]

# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:txndatastore-nofunctionaltxs ✖︎ [16:44:05]
→ go test -run=XXX -bench=. ./...
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	0.014s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.007s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/mem
BenchmarkInMemoryPeerstore/Peerstore-8         	  100000	     17491 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/mem	1.949s
2018/09/03 16:44:10 Replaying from value pointer: {Fid:0 Len:0 Offset:0}
2018/09/03 16:44:10 Iterating file id: 0
2018/09/03 16:44:10 Iteration took: 26.155µs
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/pstoreds
BenchmarkBadgerDsPeerstore/Peerstore-8         	   10000	    146738 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/pstoreds	1.531s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/queue	0.014s
?   	github.com/libp2p/go-libp2p-peerstore/test	[no test files]

# raul at Rauls-MacBook-Pro.local in ~/go/src/github.com/libp2p/go-libp2p-peerstore on git:txndatastore-nofunctionaltxs ✖︎ [16:44:11]
→ go test -run=XXX -bench=. ./...
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	0.014s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.007s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/mem
BenchmarkInMemoryPeerstore/Peerstore-8         	  100000	     17742 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/mem	1.978s
2018/09/03 16:44:15 Replaying from value pointer: {Fid:0 Len:0 Offset:0}
2018/09/03 16:44:15 Iterating file id: 0
2018/09/03 16:44:15 Iteration took: 22.872µs
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/pstoreds
BenchmarkBadgerDsPeerstore/Peerstore-8         	   10000	    146782 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/pstoreds	1.598s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/queue	0.015s
?   	github.com/libp2p/go-libp2p-peerstore/test	[no test files]

raulk · 2018-09-10T20:33:18Z

commit 686aaca

→ go test -v ./... -run=XXX -bench=.
PASS
ok  	github.com/libp2p/go-libp2p-peerstore	0.036s
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.007s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/pstoreds
BenchmarkBadgerDsPeerstore/GetAddrs-1Addrs-Caching-8         	  100000	     18607 ns/op
BenchmarkBadgerDsPeerstore/GetAddrs-10Addrs-Caching-8        	   30000	     42535 ns/op
BenchmarkBadgerDsPeerstore/GetAddrs-100Addrs-Caching-8       	   10000	    186330 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-1Addrs-Caching-8 	    5000	   1317656 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-10Addrs-Caching-8         	    2000	   3559630 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-100Addrs-Caching-8        	     500	  10097091 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-1Addrs-Caching-8     	     500	   4867320 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-10Addrs-Caching-8    	     100	  14738815 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-100Addrs-Caching-8   	      20	 131244485 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-1Addrs-Caching-8                  	   20000	    104391 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-10Addrs-Caching-8                 	    5000	    245969 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-100Addrs-Caching-8                	    2000	    742966 ns/op
BenchmarkBadgerDsPeerstore/SetAddrs-1Addrs-Caching-8                  	   20000	     95248 ns/op
BenchmarkBadgerDsPeerstore/SetAddrs-10Addrs-Caching-8                 	    5000	    245479 ns/op
BenchmarkBadgerDsPeerstore/SetAddrs-100Addrs-Caching-8                	    2000	    757456 ns/op
BenchmarkBadgerDsPeerstore/GetAddrs-1Addrs-Cacheless-8                	  100000	     19788 ns/op
BenchmarkBadgerDsPeerstore/GetAddrs-10Addrs-Cacheless-8               	   50000	     33343 ns/op
BenchmarkBadgerDsPeerstore/GetAddrs-100Addrs-Cacheless-8              	   10000	    104965 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-1Addrs-Cacheless-8        	    3000	    861505 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-10Addrs-Cacheless-8       	    2000	   3708932 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-100Addrs-Cacheless-8      	     500	  10793205 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-1Addrs-Cacheless-8   	     500	   4861002 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-10Addrs-Cacheless-8  	     100	  14688661 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-100Addrs-Cacheless-8 	      20	 130403586 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-1Addrs-Cacheless-8                	   20000	    108384 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-10Addrs-Cacheless-8               	    5000	    265131 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-100Addrs-Cacheless-8              	    2000	    887931 ns/op
BenchmarkBadgerDsPeerstore/SetAddrs-1Addrs-Cacheless-8                	   10000	    111406 ns/op
BenchmarkBadgerDsPeerstore/SetAddrs-10Addrs-Cacheless-8               	    5000	    261645 ns/op
BenchmarkBadgerDsPeerstore/SetAddrs-100Addrs-Cacheless-8              	    2000	    896938 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/pstoreds	139.817s

raulk · 2018-09-11T11:00:47Z

Big improvement in Addrs() now that we can return fully from cache if the entry is present. 90% performance improvement on cache hits, and way lower amplification as multiaddr count grows.

Commit raulk@03ee26f

ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.006s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/pstoreds
BenchmarkBadgerDsPeerstore/SetAddrs-1Addrs-Caching-8         	   10000	    118601 ns/op
BenchmarkBadgerDsPeerstore/SetAddrs-10Addrs-Caching-8        	    5000	    253708 ns/op
BenchmarkBadgerDsPeerstore/SetAddrs-100Addrs-Caching-8       	    2000	    782324 ns/op
*** BenchmarkBadgerDsPeerstore/GetAddrs-1Addrs-Caching-8         	  500000	      2651 ns/op
*** BenchmarkBadgerDsPeerstore/GetAddrs-10Addrs-Caching-8        	  500000	      2687 ns/op
*** BenchmarkBadgerDsPeerstore/GetAddrs-100Addrs-Caching-8       	  500000	      3314 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-1Addrs-Caching-8 	    3000	    947853 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-10Addrs-Caching-8         	    2000	   3643233 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-100Addrs-Caching-8        	     500	   9864702 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-1Addrs-Caching-8     	     500	   4802422 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-10Addrs-Caching-8    	     100	  14171103 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-100Addrs-Caching-8   	      20	 125427860 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-1Addrs-Caching-8                  	   10000	    104702 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-10Addrs-Caching-8                 	    5000	    246570 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-100Addrs-Caching-8                	    2000	    751040 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-1Addrs-Cacheless-8                	   20000	    109331 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-10Addrs-Cacheless-8               	    5000	    245606 ns/op
BenchmarkBadgerDsPeerstore/AddAddrs-100Addrs-Cacheless-8              	    2000	    763814 ns/op
BenchmarkBadgerDsPeerstore/SetAddrs-1Addrs-Cacheless-8                	   20000	    115151 ns/op
BenchmarkBadgerDsPeerstore/SetAddrs-10Addrs-Cacheless-8               	    5000	    253255 ns/op
BenchmarkBadgerDsPeerstore/SetAddrs-100Addrs-Cacheless-8              	    2000	    745076 ns/op
BenchmarkBadgerDsPeerstore/GetAddrs-1Addrs-Cacheless-8                	  100000	     22827 ns/op
BenchmarkBadgerDsPeerstore/GetAddrs-10Addrs-Cacheless-8               	   30000	     65168 ns/op
BenchmarkBadgerDsPeerstore/GetAddrs-100Addrs-Cacheless-8              	    5000	    228338 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-1Addrs-Cacheless-8        	    3000	    902894 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-10Addrs-Cacheless-8       	    2000	   3722929 ns/op
BenchmarkBadgerDsPeerstore/AddAndClearAddrs-100Addrs-Cacheless-8      	     500	  10161723 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-1Addrs-Cacheless-8   	     500	   4794067 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-10Addrs-Cacheless-8  	     100	  14503596 ns/op
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-100Addrs-Cacheless-8 	      20	 128094557 ns/op
PASS

raulk · 2018-09-11T11:41:38Z

Changed the AddAndClearAddrs benchmark to add a middle Get step that allows us to characterise cache performance.

ok  	github.com/libp2p/go-libp2p-peerstore/addr	0.006s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/pstoreds
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-1Addrs-Caching-8         	    3000	   1452368 ns/op
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-10Addrs-Caching-8        	    2000	   6044407 ns/op
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-100Addrs-Caching-8       	     300	  13057546 ns/op
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-1Addrs-Cacheless-8       	    3000	   1415757 ns/op
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-10Addrs-Cacheless-8      	    2000	   6454347 ns/op
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-100Addrs-Cacheless-8     	     500	  17702449 ns/op
PASS

raulk · 2018-09-11T12:02:27Z

Commit raulk@903aa12 gives us close to a 30-40% reduction in cache-bound ClearAddrs.

pkg: github.com/libp2p/go-libp2p-peerstore/pstoreds
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-1Addrs-Caching-8         	    3000	    970293 ns/op
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-10Addrs-Caching-8        	    2000	   3601837 ns/op
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-100Addrs-Caching-8       	     500	   9939502 ns/op
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-1Addrs-Cacheless-8       	    3000	   1398470 ns/op
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-10Addrs-Cacheless-8      	    1000	   3672528 ns/op
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-100Addrs-Cacheless-8     	     300	  14108666 ns/op
PASS
ok  	github.com/libp2p/go-libp2p-peerstore/pstoreds	48.637s

raulk · 2018-09-14T17:43:06Z

benchcmp between baseline (prior to refactorings) and #34.

→ benchcmp orig.txt ../ttl.txt
benchmark                                                               old ns/op     new ns/op     delta
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-1Addrs-Caching-8         1278381       1119209       -12.45%
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-10Addrs-Caching-8        5882370       3405595       -42.11%
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-100Addrs-Caching-8       17658285      9621773       -45.51%
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-1Addrs-Caching-8       6608083       6086530       -7.89%
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-10Addrs-Caching-8      41895212      18584244      -55.64%
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-100Addrs-Caching-8     429926167     160347525     -62.70%
BenchmarkBadgerDsPeerstore/AddAddrs-1Addrs-Caching-8                    124759        119486        -4.23%
BenchmarkBadgerDsPeerstore/AddAddrs-10Addrs-Caching-8                   286197        250963        -12.31%
BenchmarkBadgerDsPeerstore/AddAddrs-100Addrs-Caching-8                  1248950       750596        -39.90%
BenchmarkBadgerDsPeerstore/SetAddrs-1Addrs-Caching-8                    117801        104276        -11.48%
BenchmarkBadgerDsPeerstore/SetAddrs-10Addrs-Caching-8                   288753        246731        -14.55%
BenchmarkBadgerDsPeerstore/SetAddrs-100Addrs-Caching-8                  1208837       742194        -38.60%
BenchmarkBadgerDsPeerstore/GetAddrs-1Addrs-Caching-8                    19064         1694          -91.11%
BenchmarkBadgerDsPeerstore/GetAddrs-10Addrs-Caching-8                   36066         1765          -95.11%
BenchmarkBadgerDsPeerstore/GetAddrs-100Addrs-Caching-8                  99075         2140          -97.84%

benchmark                                                               old allocs     new allocs     delta
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-1Addrs-Caching-8         6779           5450           -19.60%
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-10Addrs-Caching-8        45392          23771          -47.63%
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-100Addrs-Caching-8       149315         77388          -48.17%
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-1Addrs-Caching-8       39134          39136          +0.01%
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-10Addrs-Caching-8      260122         170121         -34.60%
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-100Addrs-Caching-8     2600130        1610119        -38.08%
BenchmarkBadgerDsPeerstore/AddAddrs-1Addrs-Caching-8                    129            124            -3.88%
BenchmarkBadgerDsPeerstore/AddAddrs-10Addrs-Caching-8                   814            673            -17.32%
BenchmarkBadgerDsPeerstore/AddAddrs-100Addrs-Caching-8                  7589           6092           -19.73%
BenchmarkBadgerDsPeerstore/SetAddrs-1Addrs-Caching-8                    129            124            -3.88%
BenchmarkBadgerDsPeerstore/SetAddrs-10Addrs-Caching-8                   814            673            -17.32%
BenchmarkBadgerDsPeerstore/SetAddrs-100Addrs-Caching-8                  7589           6092           -19.73%
BenchmarkBadgerDsPeerstore/GetAddrs-1Addrs-Caching-8                    71             7              -90.14%
BenchmarkBadgerDsPeerstore/GetAddrs-10Addrs-Caching-8                   149            7              -95.30%
BenchmarkBadgerDsPeerstore/GetAddrs-100Addrs-Caching-8                  782            7              -99.10%

benchmark                                                               old bytes     new bytes     delta
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-1Addrs-Caching-8         247723        182528        -26.32%
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-10Addrs-Caching-8        1491712       777324        -47.89%
BenchmarkBadgerDsPeerstore/AddGetAndClearAddrs-100Addrs-Caching-8       4907489       2589311       -47.24%
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-1Addrs-Caching-8       2684239       2684482       +0.01%
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-10Addrs-Caching-8      16537993      7321976       -55.73%
BenchmarkBadgerDsPeerstore/Get1000PeersWithAddrs-100Addrs-Caching-8     163418001     62041576      -62.04%
BenchmarkBadgerDsPeerstore/AddAddrs-1Addrs-Caching-8                    10428         9944          -4.64%
BenchmarkBadgerDsPeerstore/AddAddrs-10Addrs-Caching-8                   40567         33667         -17.01%
BenchmarkBadgerDsPeerstore/AddAddrs-100Addrs-Caching-8                  343654        273577        -20.39%
BenchmarkBadgerDsPeerstore/SetAddrs-1Addrs-Caching-8                    10359         9943          -4.02%
BenchmarkBadgerDsPeerstore/SetAddrs-10Addrs-Caching-8                   40569         33667         -17.01%
BenchmarkBadgerDsPeerstore/SetAddrs-100Addrs-Caching-8                  350473        273561        -21.95%
BenchmarkBadgerDsPeerstore/GetAddrs-1Addrs-Caching-8                    14614         272           -98.14%
BenchmarkBadgerDsPeerstore/GetAddrs-10Addrs-Caching-8                   18024         416           -97.69%
BenchmarkBadgerDsPeerstore/GetAddrs-100Addrs-Caching-8                  46089         2048          -95.56%

whyrusleeping · 2018-09-19T18:41:19Z

@raulk how do these numbers look with leveldb?

raulk · 2018-09-19T20:19:36Z

@whyrusleeping very good question, my friend! go-ds-leveldb will need a TTLDatastore shim, as it does not support TTL natively, so we'll need to emulate that capability.

Also, we migrated to TxnDatastore and leveldb doesn't have transactions per se, but it does support batching.

From the point of view of the peerstore, I don't think we need transactional guarantees anyway. @bigs do you remember the argument for requiring a transactional DB over a batch-capable one?

raulk · 2018-09-24T11:40:41Z

Ok, I see go-ds-leveldb uses go-leveldb and not the native leveldb. go-leveldb does support transactions, so I'll open a PR in go-ds-leveldb to make it implement TxnDatastore. We can then benchmark it against the commit prior to adopting TTLDatastore, just to get an initial relative comparison.

raulk · 2018-10-03T17:59:00Z

@whyrusleeping Benchmarks for badger vs. leveldb vs. in-memory. Numbers are for the version before implementing datastore-native TTL (we'd need a shim for leveldb), but they provide a good baseline.

For badger and leveldb, the numbers with and without the ARC cache are shown.

→ go test -run=XXX -bench=. -count 1 -v ./...
PASS
ok      github.com/libp2p/go-libp2p-peerstore   0.012s
PASS
ok      github.com/libp2p/go-libp2p-peerstore/addr  0.006s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/pstoreds
BenchmarkDsPeerstore/Badger/AddAddrs-1Addrs-Caching-8              10000        142463 ns/op
BenchmarkDsPeerstore/Badger/AddAddrs-10Addrs-Caching-8              5000        256604 ns/op
BenchmarkDsPeerstore/Badger/AddAddrs-100Addrs-Caching-8             2000        776413 ns/op
BenchmarkDsPeerstore/Badger/SetAddrs-1Addrs-Caching-8              10000        131172 ns/op
BenchmarkDsPeerstore/Badger/SetAddrs-10Addrs-Caching-8              5000        261044 ns/op
BenchmarkDsPeerstore/Badger/SetAddrs-100Addrs-Caching-8             2000        779919 ns/op
BenchmarkDsPeerstore/Badger/GetAddrs-1Addrs-Caching-8             500000          2828 ns/op
BenchmarkDsPeerstore/Badger/GetAddrs-10Addrs-Caching-8            500000          2886 ns/op
BenchmarkDsPeerstore/Badger/GetAddrs-100Addrs-Caching-8           500000          3743 ns/op
BenchmarkDsPeerstore/Badger/AddGetAndClearAddrs-1Addrs-Caching-8                3000        832585 ns/op
BenchmarkDsPeerstore/Badger/AddGetAndClearAddrs-10Addrs-Caching-8               2000       3511877 ns/op
BenchmarkDsPeerstore/Badger/AddGetAndClearAddrs-100Addrs-Caching-8               500       9602723 ns/op
BenchmarkDsPeerstore/Badger/Get1000PeersWithAddrs-1Addrs-Caching-8               500       4941457 ns/op
BenchmarkDsPeerstore/Badger/Get1000PeersWithAddrs-10Addrs-Caching-8              100      15008817 ns/op
BenchmarkDsPeerstore/Badger/Get1000PeersWithAddrs-100Addrs-Caching-8              20     139019626 ns/op
BenchmarkDsPeerstore/Badger/Get1000PeersWithAddrs-1Addrs-Cacheless-8             500       4964657 ns/op
BenchmarkDsPeerstore/Badger/Get1000PeersWithAddrs-10Addrs-Cacheless-8            100      14791194 ns/op
BenchmarkDsPeerstore/Badger/Get1000PeersWithAddrs-100Addrs-Cacheless-8            20     129484423 ns/op
BenchmarkDsPeerstore/Badger/AddAddrs-1Addrs-Cacheless-8                        10000        132617 ns/op
BenchmarkDsPeerstore/Badger/AddAddrs-10Addrs-Cacheless-8                        5000        269158 ns/op
BenchmarkDsPeerstore/Badger/AddAddrs-100Addrs-Cacheless-8                       2000        752294 ns/op
BenchmarkDsPeerstore/Badger/SetAddrs-1Addrs-Cacheless-8                        10000        132579 ns/op
BenchmarkDsPeerstore/Badger/SetAddrs-10Addrs-Cacheless-8                        5000        251542 ns/op
BenchmarkDsPeerstore/Badger/SetAddrs-100Addrs-Cacheless-8                       2000        764891 ns/op
BenchmarkDsPeerstore/Badger/GetAddrs-1Addrs-Cacheless-8                        50000         27171 ns/op
BenchmarkDsPeerstore/Badger/GetAddrs-10Addrs-Cacheless-8                       30000         68789 ns/op
BenchmarkDsPeerstore/Badger/GetAddrs-100Addrs-Cacheless-8                       5000        475009 ns/op
BenchmarkDsPeerstore/Badger/AddGetAndClearAddrs-1Addrs-Cacheless-8              3000       1313442 ns/op
BenchmarkDsPeerstore/Badger/AddGetAndClearAddrs-10Addrs-Cacheless-8             2000       6668670 ns/op
BenchmarkDsPeerstore/Badger/AddGetAndClearAddrs-100Addrs-Cacheless-8             500      19092934 ns/op
BenchmarkDsPeerstore/Leveldb/AddGetAndClearAddrs-1Addrs-Caching-8               1000       1496587 ns/op
BenchmarkDsPeerstore/Leveldb/AddGetAndClearAddrs-10Addrs-Caching-8              1000       1609880 ns/op
BenchmarkDsPeerstore/Leveldb/AddGetAndClearAddrs-100Addrs-Caching-8              500       2328996 ns/op
BenchmarkDsPeerstore/Leveldb/Get1000PeersWithAddrs-1Addrs-Caching-8               30      72825488 ns/op
BenchmarkDsPeerstore/Leveldb/Get1000PeersWithAddrs-10Addrs-Caching-8              30      79516329 ns/op
BenchmarkDsPeerstore/Leveldb/Get1000PeersWithAddrs-100Addrs-Caching-8             20     189617163 ns/op
BenchmarkDsPeerstore/Leveldb/AddAddrs-1Addrs-Caching-8                          2000       1444066 ns/op
BenchmarkDsPeerstore/Leveldb/AddAddrs-10Addrs-Caching-8                         2000       1413137 ns/op
BenchmarkDsPeerstore/Leveldb/AddAddrs-100Addrs-Caching-8                        1000       1255299 ns/op
BenchmarkDsPeerstore/Leveldb/SetAddrs-1Addrs-Caching-8                          2000       1419329 ns/op
BenchmarkDsPeerstore/Leveldb/SetAddrs-10Addrs-Caching-8                         2000       1411017 ns/op
BenchmarkDsPeerstore/Leveldb/SetAddrs-100Addrs-Caching-8                        1000       1199369 ns/op
BenchmarkDsPeerstore/Leveldb/GetAddrs-1Addrs-Caching-8                        500000          2886 ns/op
BenchmarkDsPeerstore/Leveldb/GetAddrs-10Addrs-Caching-8                       500000          3012 ns/op
BenchmarkDsPeerstore/Leveldb/GetAddrs-100Addrs-Caching-8                      500000          3277 ns/op
BenchmarkDsPeerstore/Leveldb/AddAddrs-1Addrs-Cacheless-8                        2000       1425005 ns/op
BenchmarkDsPeerstore/Leveldb/AddAddrs-10Addrs-Cacheless-8                       2000       1522823 ns/op
BenchmarkDsPeerstore/Leveldb/AddAddrs-100Addrs-Cacheless-8                      2000       1822587 ns/op
BenchmarkDsPeerstore/Leveldb/SetAddrs-1Addrs-Cacheless-8                        2000       1458652 ns/op
BenchmarkDsPeerstore/Leveldb/SetAddrs-10Addrs-Cacheless-8                       2000       1466791 ns/op
BenchmarkDsPeerstore/Leveldb/SetAddrs-100Addrs-Cacheless-8                      2000       1606746 ns/op
BenchmarkDsPeerstore/Leveldb/GetAddrs-1Addrs-Cacheless-8                       30000         51810 ns/op
BenchmarkDsPeerstore/Leveldb/GetAddrs-10Addrs-Cacheless-8                      20000         67545 ns/op
BenchmarkDsPeerstore/Leveldb/GetAddrs-100Addrs-Cacheless-8                     10000        180388 ns/op
BenchmarkDsPeerstore/Leveldb/AddGetAndClearAddrs-1Addrs-Cacheless-8             1000       1545687 ns/op
BenchmarkDsPeerstore/Leveldb/AddGetAndClearAddrs-10Addrs-Cacheless-8            1000       1791162 ns/op
BenchmarkDsPeerstore/Leveldb/AddGetAndClearAddrs-100Addrs-Cacheless-8            500       2574569 ns/op
BenchmarkDsPeerstore/Leveldb/Get1000PeersWithAddrs-1Addrs-Cacheless-8             30      74049144 ns/op
BenchmarkDsPeerstore/Leveldb/Get1000PeersWithAddrs-10Addrs-Cacheless-8            30      81618528 ns/op
BenchmarkDsPeerstore/Leveldb/Get1000PeersWithAddrs-100Addrs-Cacheless-8           20     188187383 ns/op
PASS
ok      github.com/libp2p/go-libp2p-peerstore/pstoreds  209.673s
goos: darwin
goarch: amd64
pkg: github.com/libp2p/go-libp2p-peerstore/pstoremem
BenchmarkInMemoryPeerstore/AddAddrs-1Addrs-InMem-8            100000         19356 ns/op
BenchmarkInMemoryPeerstore/AddAddrs-10Addrs-InMem-8            30000         52424 ns/op
BenchmarkInMemoryPeerstore/AddAddrs-100Addrs-InMem-8           10000        406217 ns/op
BenchmarkInMemoryPeerstore/SetAddrs-1Addrs-InMem-8            100000         18826 ns/op
BenchmarkInMemoryPeerstore/SetAddrs-10Addrs-InMem-8            30000         51079 ns/op
BenchmarkInMemoryPeerstore/SetAddrs-100Addrs-InMem-8           10000        405784 ns/op
BenchmarkInMemoryPeerstore/GetAddrs-1Addrs-InMem-8           5000000           285 ns/op
BenchmarkInMemoryPeerstore/GetAddrs-10Addrs-InMem-8          2000000           651 ns/op
BenchmarkInMemoryPeerstore/GetAddrs-100Addrs-InMem-8          300000          3871 ns/op
BenchmarkInMemoryPeerstore/AddGetAndClearAddrs-1Addrs-InMem-8             100000         19153 ns/op
BenchmarkInMemoryPeerstore/AddGetAndClearAddrs-10Addrs-InMem-8             30000         51645 ns/op
BenchmarkInMemoryPeerstore/AddGetAndClearAddrs-100Addrs-InMem-8            10000        350757 ns/op
BenchmarkInMemoryPeerstore/Get1000PeersWithAddrs-1Addrs-InMem-8            20000         89279 ns/op
BenchmarkInMemoryPeerstore/Get1000PeersWithAddrs-10Addrs-InMem-8           20000         93078 ns/op
BenchmarkInMemoryPeerstore/Get1000PeersWithAddrs-100Addrs-InMem-8          20000        106979 ns/op
PASS
ok      github.com/libp2p/go-libp2p-peerstore/pstoremem 38.852s
PASS
ok      github.com/libp2p/go-libp2p-peerstore/queue 0.043s
?       github.com/libp2p/go-libp2p-peerstore/test  [no test files]

raulk · 2018-10-03T18:03:37Z

Badger seems to come out several multiples on top of the leveldb Go implementation. Testing against RocksDB (cgo) could be interesting. Also, the peerstore could benefit from an indexed DB like sqlite.

bigs · 2018-10-04T19:33:28Z

good to know! i think rocksdb could be nice for the future, but i don’t think it needs to be a priority. i do however think a sqlite example could be worthwhile, though if all we really need from it is indexing, perhaps we could implement this in another badger db?

…

On Oct 3, 2018, 2:03 PM -0400, Raúl Kripalani ***@***.***>, wrote: Badger seems to come out several multiples on top of the leveldb Go implementation. Testing against RocksDB (cgo) could be interesting. Also, the peerstore could benefit from an indexed DB like sqlite. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

lanzafame · 2018-10-29T08:39:03Z

@raulk would be a good idea to add -benchmem to the go test command to get the number allocations per op

raulk · 2019-04-08T22:38:38Z

No longer necessary to keep this trail since datastore-backed peerstore is now merged and performant.

raulk changed the title ~~Benchmark log~~ [Keep open] Benchmark log Aug 31, 2018

raulk mentioned this issue Aug 31, 2018

Implement datastore-backed peerstore #28

Closed

6 tasks

This was referenced Sep 14, 2018

Store-native TTL management #34

Merged

peerstore: next steps for datastore-backed peerstore libp2p/go-libp2p#1702

Open

bigs added topic/perf Performance research status/deferred Conscious decision to pause or backlog labels Sep 18, 2018

raulk mentioned this issue Oct 4, 2018

Replace b58-encoded keys with b32 (no padding) + test against multiple ds #40

Merged

raulk closed this as completed Apr 8, 2019

ghost removed the status/deferred Conscious decision to pause or backlog label Apr 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Keep open] Benchmark log #31

[Keep open] Benchmark log #31

raulk commented Aug 31, 2018 •

edited

Loading

raulk commented Aug 31, 2018 •

edited

Loading

raulk commented Sep 3, 2018 •

edited

Loading

raulk commented Sep 3, 2018 •

edited

Loading

raulk commented Sep 3, 2018 •

edited

Loading

raulk commented Sep 10, 2018

raulk commented Sep 11, 2018 •

edited

Loading

raulk commented Sep 11, 2018

raulk commented Sep 11, 2018

raulk commented Sep 14, 2018

whyrusleeping commented Sep 19, 2018

raulk commented Sep 19, 2018 •

edited

Loading

raulk commented Sep 24, 2018

raulk commented Oct 3, 2018

raulk commented Oct 3, 2018

bigs commented Oct 4, 2018 via email

lanzafame commented Oct 29, 2018

raulk commented Apr 8, 2019

[Keep open] Benchmark log #31

[Keep open] Benchmark log #31

Comments

raulk commented Aug 31, 2018 • edited Loading

raulk commented Aug 31, 2018 • edited Loading

raulk commented Sep 3, 2018 • edited Loading

raulk commented Sep 3, 2018 • edited Loading

raulk commented Sep 3, 2018 • edited Loading

raulk commented Sep 10, 2018

raulk commented Sep 11, 2018 • edited Loading

raulk commented Sep 11, 2018

raulk commented Sep 11, 2018

raulk commented Sep 14, 2018

whyrusleeping commented Sep 19, 2018

raulk commented Sep 19, 2018 • edited Loading

raulk commented Sep 24, 2018

raulk commented Oct 3, 2018

raulk commented Oct 3, 2018

bigs commented Oct 4, 2018 via email

lanzafame commented Oct 29, 2018

raulk commented Apr 8, 2019

raulk commented Aug 31, 2018 •

edited

Loading

raulk commented Aug 31, 2018 •

edited

Loading

raulk commented Sep 3, 2018 •

edited

Loading

raulk commented Sep 3, 2018 •

edited

Loading

raulk commented Sep 3, 2018 •

edited

Loading

raulk commented Sep 11, 2018 •

edited

Loading

raulk commented Sep 19, 2018 •

edited

Loading