add benchmark test #62

akhilkumarpilli · 2022-03-03T08:52:01Z

Initially, I did load testing on tracelistener with multiple traces and here are the stats:

Traces sent	Time took
100 K	42 secs
1M	391 secs
10M	4535 secs

After that, I tried to do benchmarking testing and here is the result of it:

goos: linux
goarch: amd64
pkg: github.com/allinbits/tracelistener/tracelistener
cpu: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
BenchmarkTraceListener
BenchmarkTraceListener-4           25924             44233 ns/op            1899 B/op         27 allocs/op

Please run make benchmark-<sdk-version> to run benchmark test locally

gsora · 2022-03-03T11:05:06Z

tracelistener/benchmark_test.go

+	if err != nil {
+		return err
+	}
+	trace := tracelistener.TraceOperation{


You cannot use this object to marshal trace operations, you must use the (non-exported) traceOperationInter object.

This one is just a high-level representation of a trace, not the wire-level format the chain outputs.

Also, what key/value are you sending? What's the aim of this benchmark? Speed of processing or just ingestion speed?

We are just checking ingestion speed and not considering processing speed.

gsora · 2022-03-03T11:06:00Z

tracelistener/benchmark_test.go

+		Operation:   string(tracelistener.WriteOp),
+		Key:         []byte{0x68, 0x65, 0x6c, 0x6c, 0x6f, 0xa},
+		Value:       []byte{0x68, 0x65, 0x6c, 0x6c, 0x6f, 0xa},
+		BlockHeight: uint64(height),


Leaving the same block height for all traces will never trigger a database write, since tracelistener will only write back to database when the block height changes.

I am passing blockheight from benchmark function which will be dynamic and different.

Oh right! 👯

gsora · 2022-03-03T11:08:25Z

tracelistener/benchmark_test.go

+	}
+}
+
+func loadTest(height int, file string) error {


Suggested change

func loadTest(height int, file string) error {

func loadTest(b *testing.B, height int, file string) error {

b.Helper()

This function should be marked as helper, so that it gets ignored in performance counting.

gsora · 2022-03-03T11:09:06Z

tracelistener/benchmark_test.go

+}
+
+func loadTest(height int, file string) error {
+	ff, err := fifo.OpenFifo(context.Background(), file, syscall.O_WRONLY, 0655)


Do not open and close the FIFO, just open it at the beginning of the test and close it when done.

This way it resembles more how real execution behaves.

Even just doing this improves dramatically the performance of the test.

Yeah, I tried with opening file only once in the beginning. But, tracelistener is considering all traces into one single trace and just parsing it.

What do you mean?

Can you please check these logs

Fifo opening and closing each time: https://pastebin.com/22SSVnaq

Fifo opening only once in beginning; https://pastebin.com/RQjeRXSg

When fifo is opened in the beginning, new trace is being appended to previous trace everytime.

Just use fmt.Fprintf instead of writing directly, and append a newline at the end of each serialized trace.

gsora · 2022-03-03T11:09:51Z

Makefile

@@ -56,6 +57,11 @@ $(TEST_VERSIONS):
 	go test -v -failfast -race -count=1 \
 		-tags $(shell echo $@ | sed -e 's/test-/sdk_/g' -e 's/-/_/g'),muslc \
 		./...
+
+$(BENCHMARK_VERSIONS):


It would be useful to enable CPU and memory profiling while running benchmarks.

If I am not wrong, -benchmem flag will give you the same right?

You have to supply flags like those as well to create the profiling files:

-cpuprofile=cpu.out -memprofile=mem.out -trace=trace.out

gsora · 2022-03-03T11:24:30Z

Hmm I got wildly different numbers.

The estimated execution time calculated by go test (mind you, with the suggested improvements) for the two biggest sizes are the following:

BenchmarkTraceListener1M-10      	       1	3498110750 ns/op	501296464 B/op	 6201905 allocs/op
BenchmarkTraceListener10M-10     	       1	41465110541 ns/op	5070649768 B/op	62435038 allocs/op

So for 10M traces processed, I have a runtime of about 41s.

I ran those tests on an ARM M1 Pro machine.

gsora · 2022-03-03T11:30:58Z

From a quick test, looks like the json.Marshal method is actually bottlenecking the whole thing... Incredible :|

gsora · 2022-03-03T11:42:51Z

Memory-wise, we could greatly improve this fast path by relying on other JSON libraries, which avoid doing all those allocations I think.

gsora · 2022-03-03T11:44:02Z

This is the CPU flame graph from the hot path in tracelistener ingestion routines. The JSON unmarshaling is killing us on the CPU side as well.

add benchmark test

5dc09fd

gsora suggested changes Mar 3, 2022

View reviewed changes

akhilkumarpilli added 2 commits March 3, 2022 19:31

Address few suggestions

641336f

open fifo only once

0b15dca

akhilkumarpilli marked this pull request as ready for review March 4, 2022 05:04

akhilkumarpilli marked this pull request as draft March 4, 2022 10:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add benchmark test #62

add benchmark test #62

akhilkumarpilli commented Mar 3, 2022 •

edited

Loading

gsora Mar 3, 2022

akhilkumarpilli Mar 3, 2022

gsora Mar 3, 2022

akhilkumarpilli Mar 3, 2022

gsora Mar 3, 2022

gsora Mar 3, 2022

gsora Mar 3, 2022

gsora Mar 3, 2022

akhilkumarpilli Mar 3, 2022

gsora Mar 3, 2022

akhilkumarpilli Mar 3, 2022

gsora Mar 3, 2022

gsora Mar 3, 2022

akhilkumarpilli Mar 3, 2022

gsora Mar 3, 2022

gsora commented Mar 3, 2022

gsora commented Mar 3, 2022

gsora commented Mar 3, 2022

gsora commented Mar 3, 2022

	func loadTest(height int, file string) error {
	func loadTest(b *testing.B, height int, file string) error {
	b.Helper()

add benchmark test #62

Are you sure you want to change the base?

add benchmark test #62

Conversation

akhilkumarpilli commented Mar 3, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gsora commented Mar 3, 2022

gsora commented Mar 3, 2022

gsora commented Mar 3, 2022

gsora commented Mar 3, 2022

akhilkumarpilli commented Mar 3, 2022 •

edited

Loading