Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-allocate res change-items based on statistic #197

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

laskoviymishka
Copy link
Contributor

Calc avg row size so we can pre-allocate change-item slice.

Before:

BenchmarkGenericParser
BenchmarkGenericParser/metrika
BenchmarkGenericParser/metrika/batch_size_1
BenchmarkGenericParser/metrika/batch_size_1-10         	   42182	     27948 ns/op	 146.38 MB/s	   40584 B/op	     321 allocs/op
BenchmarkGenericParser/metrika/batch_size_5
BenchmarkGenericParser/metrika/batch_size_5-10         	    2776	    429816 ns/op	 152.36 MB/s	  650278 B/op	    5020 allocs/op
BenchmarkGenericParser/metrika/batch_size_10
BenchmarkGenericParser/metrika/batch_size_10-10        	      81	  13386086 ns/op	 156.55 MB/s	20820405 B/op	  160525 allocs/op
BenchmarkGenericParser/taxi
BenchmarkGenericParser/taxi/batch_size_1
BenchmarkGenericParser/taxi/batch_size_1-10            	   30919	     36305 ns/op	  54.43 MB/s	   35698 B/op	     385 allocs/op
BenchmarkGenericParser/taxi/batch_size_5
BenchmarkGenericParser/taxi/batch_size_5-10            	    2134	    565053 ns/op	  56.01 MB/s	  577095 B/op	    6004 allocs/op
BenchmarkGenericParser/taxi/batch_size_10
BenchmarkGenericParser/taxi/batch_size_10-10           	      63	  17794249 ns/op	  56.91 MB/s	18091908 B/op	  194000 allocs/op
BenchmarkGenericParser/mdb
BenchmarkGenericParser/mdb/batch_size_1
BenchmarkGenericParser/mdb/batch_size_1-10             	  130441	      9152 ns/op	  74.63 MB/s	    7584 B/op	     103 allocs/op
BenchmarkGenericParser/mdb/batch_size_5
BenchmarkGenericParser/mdb/batch_size_5-10             	    8551	    138940 ns/op	  78.87 MB/s	  123364 B/op	    1544 allocs/op
BenchmarkGenericParser/mdb/batch_size_10
BenchmarkGenericParser/mdb/batch_size_10-10            	     274	   4359217 ns/op	  80.45 MB/s	 3920205 B/op	   49315 allocs/op
BenchmarkGenericParser/json-sample
BenchmarkGenericParser/json-sample/batch_size_1
BenchmarkGenericParser/json-sample/batch_size_1-10     	   17776	     68195 ns/op	  95.55 MB/s	   93827 B/op	     986 allocs/op
BenchmarkGenericParser/json-sample/batch_size_5
BenchmarkGenericParser/json-sample/batch_size_5-10     	    1160	   1024875 ns/op	 101.75 MB/s	 1471232 B/op	   15896 allocs/op
BenchmarkGenericParser/json-sample/batch_size_10
BenchmarkGenericParser/json-sample/batch_size_10-10    	      34	  32322842 ns/op	 103.25 MB/s	52604432 B/op	  515963 allocs/op
PASS

After:

BenchmarkGenericParser
BenchmarkGenericParser/metrika
BenchmarkGenericParser/metrika/batch_size_1
BenchmarkGenericParser/metrika/batch_size_1-10         	   42922	     28136 ns/op	 145.40 MB/s	   40587 B/op	     321 allocs/op
BenchmarkGenericParser/metrika/batch_size_5
BenchmarkGenericParser/metrika/batch_size_5-10         	    2733	    432692 ns/op	 151.35 MB/s	  645956 B/op	    5016 allocs/op
BenchmarkGenericParser/metrika/batch_size_10
BenchmarkGenericParser/metrika/batch_size_10-10        	      82	  13368213 ns/op	 156.76 MB/s	20646278 B/op	  160527 allocs/op
BenchmarkGenericParser/taxi
BenchmarkGenericParser/taxi/batch_size_1
BenchmarkGenericParser/taxi/batch_size_1-10            	   32324	     36165 ns/op	  54.64 MB/s	   32783 B/op	     382 allocs/op
BenchmarkGenericParser/taxi/batch_size_5
BenchmarkGenericParser/taxi/batch_size_5-10            	    2126	    558261 ns/op	  56.69 MB/s	  522986 B/op	    5997 allocs/op
BenchmarkGenericParser/taxi/batch_size_10
BenchmarkGenericParser/taxi/batch_size_10-10           	      63	  17978731 ns/op	  56.33 MB/s	16610718 B/op	  193980 allocs/op
BenchmarkGenericParser/mdb
BenchmarkGenericParser/mdb/batch_size_1
BenchmarkGenericParser/mdb/batch_size_1-10             	  128125	      9239 ns/op	  73.93 MB/s	    7584 B/op	     103 allocs/op
BenchmarkGenericParser/mdb/batch_size_5
BenchmarkGenericParser/mdb/batch_size_5-10             	    8751	    139038 ns/op	  78.81 MB/s	  119005 B/op	    1540 allocs/op
BenchmarkGenericParser/mdb/batch_size_10
BenchmarkGenericParser/mdb/batch_size_10-10            	     272	   4506998 ns/op	  77.82 MB/s	 3750451 B/op	   49303 allocs/op
BenchmarkGenericParser/json-sample
BenchmarkGenericParser/json-sample/batch_size_1
BenchmarkGenericParser/json-sample/batch_size_1-10     	   19152	     62470 ns/op	 104.31 MB/s	   66309 B/op	     980 allocs/op
BenchmarkGenericParser/json-sample/batch_size_5
BenchmarkGenericParser/json-sample/batch_size_5-10     	    1216	    980603 ns/op	 106.35 MB/s	 1064461 B/op	   15884 allocs/op
BenchmarkGenericParser/json-sample/batch_size_10
BenchmarkGenericParser/json-sample/batch_size_10-10    	      36	  31956435 ns/op	 104.43 MB/s	33880404 B/op	  515899 allocs/op
PASS

Calc avg row size so we can pre-allocate change-item slice.

Before:
```
BenchmarkGenericParser
BenchmarkGenericParser/metrika
BenchmarkGenericParser/metrika/batch_size_1
BenchmarkGenericParser/metrika/batch_size_1-10         	   42182	     27948 ns/op	 146.38 MB/s	   40584 B/op	     321 allocs/op
BenchmarkGenericParser/metrika/batch_size_5
BenchmarkGenericParser/metrika/batch_size_5-10         	    2776	    429816 ns/op	 152.36 MB/s	  650278 B/op	    5020 allocs/op
BenchmarkGenericParser/metrika/batch_size_10
BenchmarkGenericParser/metrika/batch_size_10-10        	      81	  13386086 ns/op	 156.55 MB/s	20820405 B/op	  160525 allocs/op
BenchmarkGenericParser/taxi
BenchmarkGenericParser/taxi/batch_size_1
BenchmarkGenericParser/taxi/batch_size_1-10            	   30919	     36305 ns/op	  54.43 MB/s	   35698 B/op	     385 allocs/op
BenchmarkGenericParser/taxi/batch_size_5
BenchmarkGenericParser/taxi/batch_size_5-10            	    2134	    565053 ns/op	  56.01 MB/s	  577095 B/op	    6004 allocs/op
BenchmarkGenericParser/taxi/batch_size_10
BenchmarkGenericParser/taxi/batch_size_10-10           	      63	  17794249 ns/op	  56.91 MB/s	18091908 B/op	  194000 allocs/op
BenchmarkGenericParser/mdb
BenchmarkGenericParser/mdb/batch_size_1
BenchmarkGenericParser/mdb/batch_size_1-10             	  130441	      9152 ns/op	  74.63 MB/s	    7584 B/op	     103 allocs/op
BenchmarkGenericParser/mdb/batch_size_5
BenchmarkGenericParser/mdb/batch_size_5-10             	    8551	    138940 ns/op	  78.87 MB/s	  123364 B/op	    1544 allocs/op
BenchmarkGenericParser/mdb/batch_size_10
BenchmarkGenericParser/mdb/batch_size_10-10            	     274	   4359217 ns/op	  80.45 MB/s	 3920205 B/op	   49315 allocs/op
BenchmarkGenericParser/json-sample
BenchmarkGenericParser/json-sample/batch_size_1
BenchmarkGenericParser/json-sample/batch_size_1-10     	   17776	     68195 ns/op	  95.55 MB/s	   93827 B/op	     986 allocs/op
BenchmarkGenericParser/json-sample/batch_size_5
BenchmarkGenericParser/json-sample/batch_size_5-10     	    1160	   1024875 ns/op	 101.75 MB/s	 1471232 B/op	   15896 allocs/op
BenchmarkGenericParser/json-sample/batch_size_10
BenchmarkGenericParser/json-sample/batch_size_10-10    	      34	  32322842 ns/op	 103.25 MB/s	52604432 B/op	  515963 allocs/op
PASS
```

After:
```
BenchmarkGenericParser
BenchmarkGenericParser/metrika
BenchmarkGenericParser/metrika/batch_size_1
BenchmarkGenericParser/metrika/batch_size_1-10         	   42922	     28136 ns/op	 145.40 MB/s	   40587 B/op	     321 allocs/op
BenchmarkGenericParser/metrika/batch_size_5
BenchmarkGenericParser/metrika/batch_size_5-10         	    2733	    432692 ns/op	 151.35 MB/s	  645956 B/op	    5016 allocs/op
BenchmarkGenericParser/metrika/batch_size_10
BenchmarkGenericParser/metrika/batch_size_10-10        	      82	  13368213 ns/op	 156.76 MB/s	20646278 B/op	  160527 allocs/op
BenchmarkGenericParser/taxi
BenchmarkGenericParser/taxi/batch_size_1
BenchmarkGenericParser/taxi/batch_size_1-10            	   32324	     36165 ns/op	  54.64 MB/s	   32783 B/op	     382 allocs/op
BenchmarkGenericParser/taxi/batch_size_5
BenchmarkGenericParser/taxi/batch_size_5-10            	    2126	    558261 ns/op	  56.69 MB/s	  522986 B/op	    5997 allocs/op
BenchmarkGenericParser/taxi/batch_size_10
BenchmarkGenericParser/taxi/batch_size_10-10           	      63	  17978731 ns/op	  56.33 MB/s	16610718 B/op	  193980 allocs/op
BenchmarkGenericParser/mdb
BenchmarkGenericParser/mdb/batch_size_1
BenchmarkGenericParser/mdb/batch_size_1-10             	  128125	      9239 ns/op	  73.93 MB/s	    7584 B/op	     103 allocs/op
BenchmarkGenericParser/mdb/batch_size_5
BenchmarkGenericParser/mdb/batch_size_5-10             	    8751	    139038 ns/op	  78.81 MB/s	  119005 B/op	    1540 allocs/op
BenchmarkGenericParser/mdb/batch_size_10
BenchmarkGenericParser/mdb/batch_size_10-10            	     272	   4506998 ns/op	  77.82 MB/s	 3750451 B/op	   49303 allocs/op
BenchmarkGenericParser/json-sample
BenchmarkGenericParser/json-sample/batch_size_1
BenchmarkGenericParser/json-sample/batch_size_1-10     	   19152	     62470 ns/op	 104.31 MB/s	   66309 B/op	     980 allocs/op
BenchmarkGenericParser/json-sample/batch_size_5
BenchmarkGenericParser/json-sample/batch_size_5-10     	    1216	    980603 ns/op	 106.35 MB/s	 1064461 B/op	   15884 allocs/op
BenchmarkGenericParser/json-sample/batch_size_10
BenchmarkGenericParser/json-sample/batch_size_10-10    	      36	  31956435 ns/op	 104.43 MB/s	33880404 B/op	  515899 allocs/op
PASS
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant