Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Neo]: add benchmark that adopts existing tests. #3108

Closed
wants to merge 18 commits into from
Closed

[Neo]: add benchmark that adopts existing tests. #3108

wants to merge 18 commits into from

Conversation

Jim8y
Copy link
Contributor

@Jim8y Jim8y commented Jan 24, 2024 •
edited
Loading

Description

This pr adds benchmark to the neo project, currentlty the benchmark use the VM tests json.

about the block benchmark:

block.txt contains the block and its start states, by loading the text, you can run the block as you run them in a complete ledger.

format of the block.txt:

block.toarray
storagekey.id | storagekey.length | storagekey.Key | storageItem.length | storageitem.Value
...
...

Why those blocks?

100 blocks that have the heavest IO, then the rest have the heavest computation.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Existing tests passes.

Test Configuration: None

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@Jim8y
Copy link
Contributor Author

Jim8y commented Jan 24, 2024

BenchmarkDotNet v0.13.12, macOS Sonoma 14.2.1 (23C71) [Darwin 23.2.0]
Apple M1 Pro, 1 CPU, 10 logical and 10 physical cores
.NET SDK 7.0.404
[Host] : .NET 7.0.14 (7.0.1423.51910), Arm64 RyuJIT AdvSIMD
DefaultJob : .NET 7.0.14 (7.0.1423.51910), Arm64 RyuJIT AdvSIMD

Method Mean Error StdDev
TestOthers 2,611.5 us 13.54 us 12.00 us
TestOpCodesArrays 3,562.4 us 9.69 us 7.56 us
TestOpCodesStack 1,518.5 us 3.94 us 3.69 us
TestOpCodesSlot 2,151.1 us 10.13 us 8.98 us
TestOpCodesSplice 23,359.8 us 42.08 us 37.30 us
TestOpCodesControl 2,905.6 us 11.11 us 9.84 us
TestOpCodesPush 319.4 us 2.07 us 1.84 us
TestOpCodesArithmetic 2,815.1 us 11.94 us 11.17 us
TestOpCodesBitwiseLogic 2,620.3 us 17.66 us 14.75 us
TestOpCodesTypes 480.9 us 1.40 us 1.24 us

@Jim8y
Copy link
Contributor Author

Jim8y commented Jan 24, 2024

BenchmarkDotNet v0.13.12, macOS Sonoma 14.2.1 (23C71) [Darwin 23.2.0]
Apple M1 Pro, 1 CPU, 10 logical and 10 physical cores
.NET SDK 7.0.404
[Host] : .NET 7.0.14 (7.0.1423.51910), Arm64 RyuJIT AdvSIMD
DefaultJob : .NET 7.0.14 (7.0.1423.51910), Arm64 RyuJIT AdvSIMD

Method Mean Error StdDev
NeoIssue2528 1,433,521.63 us 9,596.338 us 8,013.377 us
NeoVMIssue418 63,883.54 us 227.887 us 202.016 us
NeoIssue2723 23.85 us 0.135 us 0.105 us

@Jim8y
Copy link
Contributor Author

Jim8y commented Jan 24, 2024

@superboyiii you will not like to test this,it will take many many hours to run......

@cschuchardt88
Copy link
Member

cschuchardt88 commented Jan 25, 2024 •
edited
Loading

What do those numbers even mean? Are they bad, normal or good? A better test would be CPU and Memory. At least we know what is good and bad. What kind of load was put on the VM? These benchmarks shows nothing that we can work with. You need something to compare too. Maybe put up ethereum and other blockchain VM engines to see the different. Or compare to the NEO 2 or the old NEO blockchain.

@Jim8y
Copy link
Contributor Author

Jim8y commented Jan 25, 2024 •
edited
Loading

// * Summary *

BenchmarkDotNet v0.13.12, macOS Sonoma 14.2.1 (23C71) [Darwin 23.2.0]
Apple M1 Pro, 1 CPU, 10 logical and 10 physical cores
.NET SDK 7.0.404
[Host] : .NET 7.0.14 (7.0.1423.51910), Arm64 RyuJIT AdvSIMD
DefaultJob : .NET 7.0.14 (7.0.1423.51910), Arm64 RyuJIT AdvSIMD

Method Mean Error StdDev Median Gen0 Allocated
TestOpCodesArraysClearItems NA NA NA NA NA NA
TestOpCodesArraysPack 95,476.112 ns 773.4156 ns 645.8371 ns 95,256.668 ns 0.3662 2880 B
TestOpCodesArraysNewArrayT 23.536 ns 0.0863 ns 0.0674 ns 23.521 ns - -
TestOpCodesArraysPickItem 135,657.912 ns 207.5807 ns 184.0149 ns 135,706.457 ns 0.4883 4161 B
TestOpCodesArraysPackStruct 96,874.729 ns 316.5346 ns 264.3207 ns 96,797.368 ns 0.3662 2880 B
TestOpCodesArraysSetItem 97,435.914 ns 1,305.0611 ns 1,089.7851 ns 96,934.926 ns 0.3662 2880 B
TestOpCodesArraysNewStruct0 2.983 ns 0.0108 ns 0.0090 ns 2.978 ns - -
TestOpCodesArraysReverseItems 40,458.545 ns 794.1920 ns 1,212.8151 ns 39,884.181 ns 0.1831 1152 B
TestOpCodesArraysNewMap 2.950 ns 0.0123 ns 0.0096 ns 2.948 ns - -
TestOpCodesArraysHasKey 58,410.787 ns 398.5594 ns 332.8152 ns 58,303.978 ns 0.2441 1856 B
TestOpCodesArraysNewArray 77,280.406 ns 226.8296 ns 177.0936 ns 77,293.170 ns 0.3662 2304 B
TestOpCodesArraysKeys 19,403.475 ns 49.9656 ns 41.7235 ns 19,401.651 ns 0.0916 680 B
TestOpCodesArraysAppend 38,986.206 ns 137.1839 ns 114.5547 ns 38,959.190 ns 0.1831 1152 B
TestOpCodesArraysRemove 77,648.787 ns 240.2417 ns 187.5649 ns 77,580.147 ns 0.3662 2432 B
TestOpCodesArraysPackMap 96,356.373 ns 236.6401 ns 197.6052 ns 96,275.884 ns 0.3662 2880 B
TestOpCodesArraysValues 39,105.539 ns 171.5065 ns 143.2157 ns 39,086.461 ns 0.1831 1256 B
TestOpCodesArraysNewArray0 2.940 ns 0.0114 ns 0.0101 ns 2.937 ns - -
TestOpCodesArraysUnpack 38,888.565 ns 195.0242 ns 162.8541 ns 38,837.110 ns 0.1831 1280 B
TestOpCodesArraysSize 38,815.475 ns 106.6800 ns 89.0826 ns 38,801.201 ns 0.1831 1152 B
TestOpCodesArraysNewStruct 77,633.435 ns 138.4228 ns 115.5893 ns 77,628.097 ns 0.3662 2304 B
TestOpCodesStackXDrop 38,422.055 ns 155.6063 ns 121.4872 ns 38,419.944 ns 0.1831 1152 B
TestOpCodesStackReverseN 57,495.637 ns 103.0159 ns 86.0230 ns 57,486.051 ns 0.2441 1728 B
TestOpCodesStackReverse4 19,086.639 ns 55.0453 ns 48.7962 ns 19,075.730 ns 0.0916 576 B
TestOpCodesStackClear 5.704 ns 0.0287 ns 0.0224 ns 5.697 ns - -
TestOpCodesStackReverse3 19,351.415 ns 81.8230 ns 68.3259 ns 19,329.057 ns 0.0916 576 B
TestOpCodesStackRot 57,610.061 ns 230.3782 ns 179.8642 ns 57,604.323 ns 0.2441 1728 B
TestOpCodesStackPick 76,878.799 ns 250.4480 ns 209.1354 ns 76,868.632 ns 0.3662 2304 B
TestOpCodesStackNip 38,249.946 ns 71.5273 ns 63.4071 ns 38,258.109 ns 0.1831 1152 B
TestOpCodesStackRoll 96,081.753 ns 205.4577 ns 160.4078 ns 96,084.045 ns 0.3662 2880 B
TestOpCodesStackDepth 8.250 ns 0.0786 ns 0.0656 ns 8.248 ns - -
TestOpCodesStackSwap 38,456.206 ns 55.9375 ns 49.5872 ns 38,451.763 ns 0.1831 1152 B
TestOpCodesStackTuck 38,584.372 ns 398.4348 ns 353.2021 ns 38,483.638 ns 0.1831 1264 B
TestOpCodesStackOver 38,177.789 ns 27.2507 ns 24.1571 ns 38,172.759 ns 0.1831 1264 B
TestOpCodesStackDrop 19,129.216 ns 20.6485 ns 16.1210 ns 19,130.192 ns 0.0916 576 B
TestOpCodesSlotStsFld6 38,645.590 ns 53.0260 ns 47.0061 ns 38,640.994 ns 0.1831 1232 B
TestOpCodesSlotInitSlot 56,989.522 ns 62.1905 ns 58.1730 ns 56,977.745 ns 0.1831 1411 B
TestOpCodesSlotLdLoc4 38,646.846 ns 113.5627 ns 100.6704 ns 38,616.109 ns 0.1831 1232 B
TestOpCodesSlotLdArg 38,233.586 ns 38.6605 ns 36.1631 ns 38,228.477 ns 0.1831 1232 B
TestOpCodesSlotStArg0 19,298.741 ns 128.9483 ns 114.3093 ns 19,266.357 ns 0.0610 560 B
TestOpCodesSlotLdSFLD0 19,217.195 ns 74.7758 ns 69.9453 ns 19,195.265 ns 0.0610 560 B
TestOpCodesSlotLdArg4 38,405.126 ns 71.5741 ns 55.8803 ns 38,399.782 ns 0.1831 1232 B
TestOpCodesSlotLdArg5 38,292.661 ns 69.9107 ns 65.3946 ns 38,285.965 ns 0.1831 1232 B
TestOpCodesSlotLdSFLD1 38,359.060 ns 122.5430 ns 108.6312 ns 38,329.508 ns 0.1831 1232 B
TestOpCodesSlotLdLoc 38,356.156 ns 99.4513 ns 93.0268 ns 38,407.684 ns 0.1831 1232 B
TestOpCodesSlotStArg1 38,276.018 ns 133.4440 ns 118.2946 ns 38,259.453 ns 0.1831 1232 B
TestOpCodesSlotLdLoc5 38,328.672 ns 57.6126 ns 53.8908 ns 38,334.346 ns 0.1831 1232 B
TestOpCodesSlotLdSFLD6 38,292.269 ns 46.7788 ns 43.7570 ns 38,302.228 ns 0.1831 1232 B
TestOpCodesSlotLdArg2 38,335.066 ns 43.6779 ns 38.7193 ns 38,327.572 ns 0.1831 1232 B
TestOpCodesSlotStsFld0 19,131.298 ns 25.0075 ns 20.8824 ns 19,134.590 ns 0.0610 560 B
TestOpCodesSlotLdLoc2 38,317.278 ns 71.8289 ns 67.1888 ns 38,287.369 ns 0.1831 1232 B
TestOpCodesSlotStArg6 38,343.566 ns 97.4422 ns 81.3687 ns 38,325.762 ns 0.1831 1232 B
TestOpCodesSlotLdLoc3 38,199.541 ns 73.7426 ns 65.3709 ns 38,193.058 ns 0.1831 1232 B
TestOpCodesSlotStsFld1 38,385.990 ns 80.1609 ns 66.9379 ns 38,390.142 ns 0.1831 1232 B
TestOpCodesSlotInitSSlot 37,835.773 ns 66.1470 ns 51.6432 ns 37,830.222 ns 0.1221 954 B
TestOpCodesSlotLdArg3 38,043.998 ns 110.3186 ns 103.1920 ns 38,022.301 ns 0.1831 1232 B
TestOpCodesSlotLdSFLD 38,517.087 ns 78.0817 ns 69.2174 ns 38,534.278 ns 0.1831 1232 B
TestOpCodesSlotLdArg0 19,144.253 ns 23.4959 ns 20.8285 ns 19,140.458 ns 0.0610 560 B
TestOpCodesSlotLdSFLD4 38,272.673 ns 86.1336 ns 80.5694 ns 38,273.529 ns 0.1831 1232 B
TestOpCodesSlotStArg4 38,369.547 ns 83.2295 ns 73.7808 ns 38,381.138 ns 0.1831 1232 B
TestOpCodesSlotLdLoc0 18,761.669 ns 27.2416 ns 25.4818 ns 18,765.900 ns 0.0610 560 B
TestOpCodesSlotStsFld2 38,771.062 ns 231.8051 ns 193.5678 ns 38,684.123 ns 0.1831 1232 B
TestOpCodesSlotStsFld3 38,383.088 ns 63.5832 ns 59.4757 ns 38,391.947 ns 0.1831 1232 B
TestOpCodesSlotLdLoc1 38,276.354 ns 80.4156 ns 71.2864 ns 38,249.681 ns 0.1831 1232 B
TestOpCodesSlotStArg5 38,481.845 ns 109.1567 ns 102.1053 ns 38,496.630 ns 0.1831 1232 B
TestOpCodesSlotLdSFLD5 38,327.282 ns 95.8860 ns 85.0005 ns 38,358.181 ns 0.1831 1232 B
TestOpCodesSlotLdArg1 38,213.025 ns 55.9467 ns 52.3326 ns 38,200.963 ns 0.1831 1232 B
TestOpCodesSlotStArg2 38,526.006 ns 91.0826 ns 85.1987 ns 38,528.392 ns 0.1831 1232 B
TestOpCodesSlotLdLoc6 38,271.898 ns 47.8651 ns 42.4311 ns 38,279.059 ns 0.1831 1232 B
TestOpCodesSlotStArg 38,415.999 ns 61.2439 ns 51.1414 ns 38,405.291 ns 0.1831 1232 B
TestOpCodesSlotStsFld4 38,476.872 ns 64.7117 ns 57.3652 ns 38,466.885 ns 0.1831 1232 B
TestOpCodesSlotLdArg6 38,432.306 ns 35.5100 ns 29.6524 ns 38,435.394 ns 0.1831 1232 B
TestOpCodesSlotLdSFLD2 38,392.652 ns 67.3424 ns 59.6972 ns 38,407.003 ns 0.1831 1232 B
TestOpCodesSlotLdSFLD3 38,334.715 ns 75.5306 ns 66.9559 ns 38,327.171 ns 0.1831 1232 B
TestOpCodesSlotStLoc 38,882.881 ns 170.7175 ns 142.5569 ns 38,851.473 ns 0.1831 1360 B
TestOpCodesSlotStsFld 38,592.569 ns 56.1854 ns 46.9173 ns 38,578.623 ns 0.1831 1232 B
TestOpCodesSlotStsFld5 38,517.218 ns 184.8137 ns 154.3278 ns 38,460.510 ns 0.1831 1232 B
TestOpCodesSlotStArg3 38,513.355 ns 114.7421 ns 101.7159 ns 38,483.279 ns 0.1831 1232 B
TestOpCodesSpliceSubStr 174,054.374 ns 1,375.6073 ns 1,073.9837 ns 173,735.240 ns 0.7324 5184 B
TestOpCodesSpliceCat 77,393.696 ns 256.6099 ns 200.3441 ns 77,303.423 ns 0.3662 2304 B
TestOpCodesSpliceMemCpy 117,471.057 ns 2,335.5704 ns 3,495.7718 ns 115,270.561 ns 0.4883 3456 B
TestOpCodesSpliceLeft 77,148.896 ns 228.7955 ns 202.8212 ns 77,119.275 ns 0.3662 2304 B
TestOpCodesSpliceRight 76,919.871 ns 289.2230 ns 241.5143 ns 76,901.072 ns 0.3662 2304 B
TestOpCodesSpliceNewBuffer 77,429.094 ns 491.2340 ns 383.5232 ns 77,456.324 ns 0.3662 2304 B
TestOpCodesControlTryCatchFinally6 18,748.917 ns 169.3863 ns 141.4452 ns 18,681.252 ns 0.0610 433 B
TestOpCodesControlJmpLeL 38,250.963 ns 66.9027 ns 59.3075 ns 38,225.632 ns 0.1831 1152 B
TestOpCodesControlJmpEqL 38,389.354 ns 134.9277 ns 119.6099 ns 38,366.796 ns 0.1831 1152 B
TestOpCodesControlJmpLe 38,308.947 ns 121.6646 ns 101.5954 ns 38,329.107 ns 0.1831 1152 B
TestOpCodesControlJmpNe 38,307.078 ns 106.1382 ns 94.0887 ns 38,270.700 ns 0.1831 1152 B
TestOpCodesControlTryCatchFinally7 18,947.012 ns 38.3092 ns 31.9899 ns 18,942.066 ns 0.0610 433 B
TestOpCodesControlAssertMsg 19,453.489 ns 57.0866 ns 47.6699 ns 19,461.123 ns 0.0916 576 B
TestOpCodesControlJmpGt 38,270.492 ns 108.9874 ns 91.0094 ns 38,244.753 ns 0.1831 1152 B
TestOpCodesControlJmpL 38,475.764 ns 104.3510 ns 92.5045 ns 38,443.998 ns 0.1831 1360 B
TestOpCodesControlJmpIfL 19,200.988 ns 51.7449 ns 48.4022 ns 19,213.182 ns 0.0916 576 B
TestOpCodesControlTryFinally 18,549.169 ns 35.1699 ns 29.3685 ns 18,550.669 ns 0.0305 344 B
TestOpCodesControlJmpNeL 38,215.114 ns 188.1668 ns 146.9083 ns 38,202.843 ns 0.1831 1152 B
TestOpCodesControlJmpIfNot 19,319.962 ns 25.2387 ns 21.0754 ns 19,318.281 ns 0.0916 576 B
TestOpCodesControlAbortMsg 19,762.930 ns 44.9133 ns 39.8145 ns 19,777.146 ns 0.0916 576 B
TestOpCodesControlCall 39,968.996 ns 140.3731 ns 124.4371 ns 39,956.236 ns 0.1831 1264 B
TestOpCodesControlCallL 39,954.244 ns 118.6756 ns 105.2029 ns 39,933.923 ns 0.1831 1264 B
TestOpCodesControlJmpGeL 38,683.008 ns 68.2173 ns 56.9645 ns 38,682.869 ns 0.1831 1152 B
TestOpCodesControlRet 3.013 ns 0.0023 ns 0.0018 ns 3.013 ns - -
TestOpCodesControlJmpGtL 38,114.794 ns 65.7059 ns 58.2466 ns 38,131.649 ns 0.1831 1152 B
TestOpCodesControlJmpEq 38,447.226 ns 95.3158 ns 84.4949 ns 38,445.847 ns 0.1831 1152 B
TestOpCodesControlSysCall 95,131.459 ns 252.2568 ns 210.6458 ns 95,167.643 ns 0.3662 3041 B
TestOpCodesControlCallA 19,288.770 ns 33.1302 ns 25.8659 ns 19,290.414 ns 0.0916 688 B
TestOpCodesControlAssert 19,151.383 ns 43.3988 ns 38.4719 ns 19,145.432 ns 0.0916 576 B
TestOpCodesControlTryCatchFinally2 3.170 ns 0.0028 ns 0.0025 ns 3.170 ns - -
TestOpCodesControlTryCatchFinally3 2.956 ns 0.0027 ns 0.0024 ns 2.957 ns - -
TestOpCodesControlNop 3.146 ns 0.0046 ns 0.0041 ns 3.146 ns - -
TestOpCodesControlJmpGe 38,279.556 ns 105.0763 ns 93.1474 ns 38,308.038 ns 0.1831 1152 B
TestOpCodesControlTryCatchFinally10 19,209.804 ns 365.8806 ns 342.2450 ns 19,070.993 ns 0.0305 344 B
TestOpCodesControlJmpLt 38,373.004 ns 104.0214 ns 81.2130 ns 38,386.987 ns 0.1831 1152 B
TestOpCodesControlJmpIf 19,060.537 ns 56.3196 ns 43.9707 ns 19,054.129 ns 0.0916 576 B
TestOpCodesControlAbort 18,722.664 ns 46.7518 ns 36.5007 ns 18,714.844 ns 0.0610 433 B
TestOpCodesControlJmpIfNotL 19,205.543 ns 167.6467 ns 148.6144 ns 19,133.621 ns 0.0916 576 B
TestOpCodesControlTryCatch 8.552 ns 0.1931 ns 0.2146 ns 8.444 ns - -
TestOpCodesControlTryCatchFinally4 2.969 ns 0.0066 ns 0.0058 ns 2.970 ns - -
TestOpCodesControlJmp 38,276.245 ns 56.1572 ns 43.8439 ns 38,282.485 ns 0.1831 1344 B
TestOpCodesControlTryCatchFinally8 18,558.700 ns 23.8142 ns 21.1107 ns 18,556.267 ns 0.0305 344 B
TestOpCodesControlTryCatchFinally9 3.052 ns 0.0053 ns 0.0041 ns 3.052 ns - -
TestOpCodesControlJmpLtL 37,763.102 ns 60.0366 ns 50.1333 ns 37,762.690 ns 0.1831 1152 B
TestOpCodesControlTryCatchFinally 3.051 ns 0.0069 ns 0.0057 ns 3.050 ns - -
TestOpCodesControlTryCatchFinally5 19,101.978 ns 28.1124 ns 23.4752 ns 19,109.838 ns 0.0916 576 B
TestOpCodesControlThrow 19,205.019 ns 54.2973 ns 45.3407 ns 19,195.484 ns 0.0916 576 B
TestOpCodesPushPushA 37,132.048 ns 220.7969 ns 172.3837 ns 37,060.229 ns 0.1221 848 B
TestOpCodesPushPushM1ToPush16 2.989 ns 0.0085 ns 0.0071 ns 2.989 ns - -
TestOpCodesPushPushInt8ToPushInt256 2.968 ns 0.0028 ns 0.0023 ns 2.968 ns - -
TestOpCodesPushPushData1 19,169.600 ns 28.3395 ns 23.6648 ns 19,170.968 ns 0.1221 800 B
TestOpCodesPushPushData2 19,541.478 ns 80.1702 ns 62.5916 ns 19,540.199 ns 0.1221 800 B
TestOpCodesPushPushNull 2.942 ns 0.0037 ns 0.0029 ns 2.942 ns - -
TestOpCodesPushPushData4 77,380.132 ns 131.7581 ns 110.0240 ns 77,401.799 ns 0.4883 3185 B
TestOpCodesArithmeticGe 58,091.587 ns 304.4691 ns 237.7094 ns 58,126.792 ns 0.2441 1728 B
TestOpCodesArithmeticLt 57,963.620 ns 70.3397 ns 62.3543 ns 57,951.169 ns 0.2441 1728 B
TestOpCodesArithmeticModMul 19,238.414 ns 44.2312 ns 36.9351 ns 19,229.671 ns 0.0916 576 B
TestOpCodesArithmeticNumNotEqual 210,207.565 ns 608.3355 ns 474.9484 ns 210,081.294 ns 0.9766 6336 B
TestOpCodesArithmeticNot 19,245.446 ns 31.0980 ns 27.5676 ns 19,245.665 ns 0.0916 576 B
TestOpCodesArithmeticModPow 19,201.356 ns 35.6527 ns 33.3495 ns 19,196.527 ns 0.0916 576 B
TestOpCodesArithmeticLe 57,567.218 ns 107.3161 ns 89.6138 ns 57,542.066 ns 0.2441 1728 B
TestOpCodesArithmeticShl 12.567 ns 0.0147 ns 0.0131 ns 12.563 ns - -
TestOpCodesArithmeticGt 58,221.557 ns 102.9047 ns 85.9301 ns 58,243.515 ns 0.2441 1728 B
TestOpCodesArithmeticPow 38,304.466 ns 59.5920 ns 49.7620 ns 38,306.513 ns 0.1831 1152 B
TestOpCodesArithmeticNumEqual 212,480.276 ns 337.9103 ns 263.8181 ns 212,507.995 ns 0.9766 6336 B
TestOpCodesArithmeticSign 96,228.650 ns 441.4115 ns 344.6250 ns 96,136.340 ns 0.3662 2880 B
TestOpCodesArithmeticSqrt 19,133.408 ns 45.2197 ns 42.2985 ns 19,139.097 ns 0.0916 576 B
TestOpCodesArithmeticShr 12.727 ns 0.0116 ns 0.0097 ns 12.725 ns - -
TestOpCodesBitwiseLogicOr 134,296.902 ns 332.9352 ns 278.0160 ns 134,349.437 ns 0.4883 4032 B
TestOpCodesBitwiseLogicEqual 38,453.372 ns 49.8019 ns 41.5868 ns 38,458.878 ns 0.1831 1152 B
TestOpCodesBitwiseLogicInvert 114,979.318 ns 143.5865 ns 119.9012 ns 114,968.185 ns 0.4883 3456 B
TestOpCodesBitwiseLogicXor 135,000.222 ns 239.9489 ns 200.3682 ns 134,952.443 ns 0.4883 4032 B
TestOpCodesBitwiseLogicNotEqual 38,363.671 ns 137.7108 ns 107.5155 ns 38,352.384 ns 0.1831 1152 B
TestOpCodesBitwiseLogicAnd 134,675.916 ns 293.8290 ns 245.3605 ns 134,626.068 ns 0.4883 4032 B
TestOpCodesTypesConvert 155,564.129 ns 239.1494 ns 211.9997 ns 155,567.876 ns 0.7324 4608 B
TestOpCodesTypesIsType 23.437 ns 0.1511 ns 0.1340 ns 23.366 ns - -
TestOpCodesTypesIsNull 19,326.088 ns 43.8447 ns 36.6123 ns 19,324.146 ns 0.0916 576 B

@Jim8y
Copy link
Contributor Author

Jim8y commented Jan 25, 2024 •
edited
Loading

What do those numbers even mean? Are they bad, normal or good? A better test would be CPU and Memory. At least we know what is good and bad. What kind of load was put on the VM? These benchmarks shows nothing that we can work with. You need something to compare too. Maybe put up ethereum and other blockchain VM engines to see the different. Or compare to the NEO 2 or the old NEO blockchain.

Have it first, improve it later. If this hurt you, i am sorry, but i need something to see if my optimization really works, if my later code update improve the benchmark or not. I will run benchmark after i change my code and compare the difference.

@cschuchardt88
Copy link
Member

cschuchardt88 commented Jan 25, 2024 •
edited
Loading

Its doesn't hurt. You just provide no information other than an output. What's the sample rate? Also running something on one machine doesn't show anything either. You run the benchmark in a headless vm like a github action or your own environment with nothing running other this test. A docker container would be best.

@Jim8y
Copy link
Contributor Author

Jim8y commented Jan 25, 2024

Its doesn't hurt. You just provide no information other than an output. What's the sample rate? Also running something on one machine doesn't show anything either. You run the benchmark in a headless vm like a github action or your own environment with nothing running other this test. A docker container would be best.

This benchmark is not for showing people how fast and how efficient it is, but for us developers (yes, me) to compare the performance difference after we make change to the core, in the same machine, same environment with high priority.

We need to know how much we improved.

@shargon
Copy link
Member

shargon commented Jan 25, 2024

What do those numbers even mean? Are they bad, normal or good? A better test would be CPU and Memory. At least we know what is good and bad. What kind of load was put on the VM? These benchmarks shows nothing that we can work with. You need something to compare too. Maybe put up ethereum and other blockchain VM engines to see the different. Or compare to the NEO 2 or the old NEO blockchain.

Have it first, improve it later. If this hurt you, i am sorry, but i need something to see if my optimization really works, if my later code update improve the benchmark or not. I will run benchmark after i change my code and compare the difference.

Totally agree

shargon
shargon reviewed Jan 25, 2024
View reviewed changes
Copy link
Member

@shargon shargon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more

benchmarks/Neo.VM.Benchmarks/Benchmark.OpCodes.cs Outdated Show resolved Hide resolved
shargon
shargon previously approved these changes Jan 25, 2024
View reviewed changes
@Jim8y
update with opcode
7b97d28
@Jim8y
Merge branch 'opcodes-benchmark' of github.com:Jim8y/neo into opcodes…
54e1f15 
…-benchmark

* 'opcodes-benchmark' of github.com:Jim8y/neo:
  Update benchmarks/Neo.VM.Benchmarks/Benchmark.OpCodes.cs
  Clean lines
  Update benchmarks/Neo.VM.Benchmarks/Program.cs

# Conflicts:
#	benchmarks/Neo.VM.Benchmarks/Benchmark.OpCodes.cs
#	benchmarks/Neo.VM.Benchmarks/Program.cs
@Jim8y
Copy link
Contributor Author

Jim8y commented Jan 26, 2024 •
edited
Loading

Method _blockId Mean Error StdDev
RunBlock 288842 171,039.48 us 1,216.548 us 1,078.438 us
RunBlock 288854 172,037.83 us 1,032.248 us 965.565 us
RunBlock 288856 173,519.63 us 1,141.478 us 1,067.739 us
RunBlock 288857 171,658.98 us 986.680 us 922.941 us
RunBlock 288858 84,689.63 us 307.197 us 272.322 us
RunBlock 470619 58,467.03 us 354.543 us 331.639 us
RunBlock 470619 58,846.63 us 219.289 us 183.117 us
RunBlock 715707 59,298.11 us 600.970 us 562.148 us
RunBlock 715707 59,145.30 us 527.556 us 493.476 us
RunBlock 1073113 4,941.38 us 12.328 us 10.928 us
RunBlock 1073120 4,987.47 us 24.125 us 21.386 us
RunBlock 1093317 1,739.68 us 10.469 us 9.280 us
RunBlock 1466600 679.34 us 2.565 us 2.273 us
RunBlock 1545151 3,469.45 us 14.033 us 13.127 us
RunBlock 1551539 40,128.89 us 174.630 us 154.805 us
RunBlock 1551550 36,566.93 us 191.991 us 179.589 us
RunBlock 1551551 36,141.94 us 156.562 us 146.448 us
RunBlock 1551552 72,352.18 us 207.073 us 183.565 us
RunBlock 1570420 3,867.40 us 10.400 us 9.219 us
RunBlock 1584313 79,829.37 us 367.953 us 307.258 us
RunBlock 1667870 43,621.90 us 151.653 us 141.857 us
RunBlock 1667871 42,972.70 us 311.971 us 291.818 us
RunBlock 1683031 691.87 us 3.330 us 2.952 us
RunBlock 1690032 4,421.11 us 23.885 us 21.173 us
RunBlock 1690810 7,866.64 us 33.921 us 31.730 us
RunBlock 1691179 63,170.35 us 255.851 us 213.647 us
RunBlock 1691180 45,363.66 us 150.029 us 125.281 us
RunBlock 1691181 39,268.71 us 194.131 us 181.590 us
RunBlock 1691492 5,146.63 us 23.944 us 19.995 us
RunBlock 1691562 3,365.74 us 9.512 us 7.943 us
RunBlock 1691721 5,426.79 us 30.909 us 28.912 us
RunBlock 1691783 4,920.97 us 22.752 us 20.169 us
RunBlock 1691784 5,769.61 us 27.992 us 26.184 us
RunBlock 1691811 7,598.98 us 14.301 us 11.942 us
RunBlock 1691968 3,388.26 us 21.631 us 18.063 us
RunBlock 1692023 7,089.55 us 48.336 us 45.213 us
RunBlock 1692102 8,562.59 us 37.759 us 35.319 us
RunBlock 1692302 3,921.21 us 18.052 us 16.886 us
RunBlock 1692680 2,970.85 us 10.506 us 9.313 us
RunBlock 1692682 6,107.17 us 49.610 us 46.405 us
RunBlock 1692692 3,699.43 us 15.469 us 14.470 us
RunBlock 1692735 3,570.83 us 20.041 us 17.765 us
RunBlock 1692772 4,870.48 us 23.913 us 22.368 us
RunBlock 1692921 4,416.63 us 28.132 us 26.315 us
RunBlock 1693099 4,292.45 us 22.711 us 20.133 us
RunBlock 1693100 3,004.86 us 17.685 us 14.768 us
RunBlock 1693182 12,398.93 us 27.869 us 24.705 us
RunBlock 1693194 3,905.12 us 13.969 us 13.066 us
RunBlock 1693241 5,878.40 us 32.562 us 28.865 us
RunBlock 1693269 9,201.10 us 36.628 us 34.262 us
RunBlock 1693274 7,814.06 us 27.770 us 23.189 us
RunBlock 1693286 6,689.36 us 32.972 us 29.229 us
RunBlock 1693294 9,787.84 us 47.876 us 44.783 us
RunBlock 1693308 5,245.27 us 35.099 us 32.832 us
RunBlock 1693343 5,817.15 us 43.232 us 40.439 us
RunBlock 1693373 2,653.35 us 13.434 us 12.566 us
RunBlock 1693398 4,197.73 us 20.548 us 19.220 us
RunBlock 1693399 3,349.91 us 14.523 us 12.874 us
RunBlock 1693428 4,196.28 us 25.326 us 23.690 us
RunBlock 1693650 3,897.83 us 28.921 us 25.638 us
RunBlock 1694343 3,339.81 us 10.903 us 10.198 us
RunBlock 1694410 3,368.28 us 12.448 us 11.643 us
RunBlock 1694594 2,107.43 us 13.499 us 11.273 us
RunBlock 1695163 3,762.68 us 18.789 us 16.656 us
RunBlock 1695746 3,311.74 us 7.941 us 7.428 us
RunBlock 1695896 8,356.81 us 45.683 us 40.497 us
RunBlock 1696740 3,258.82 us 24.489 us 22.907 us
RunBlock 1696758 51,517.44 us 167.170 us 156.371 us
RunBlock 1696759 69,864.07 us 252.157 us 235.868 us
RunBlock 1723984 4,685.82 us 33.715 us 29.887 us
RunBlock 1855694 3,535.55 us 16.009 us 14.192 us
RunBlock 1855827 4,199.66 us 26.639 us 24.918 us
RunBlock 1882087 7,930.80 us 48.400 us 42.905 us
RunBlock 1882106 10,390.62 us 41.156 us 38.497 us
RunBlock 1899635 1,720.38 us 6.612 us 5.861 us
RunBlock 1900762 6,973.64 us 22.930 us 20.327 us
RunBlock 1900767 5,228.03 us 21.083 us 18.689 us
RunBlock 1900768 4,910.23 us 10.919 us 9.118 us
RunBlock 1900845 5,879.98 us 35.274 us 32.995 us
RunBlock 1907366 5,671.58 us 28.554 us 25.313 us
RunBlock 1941906 7,306.30 us 23.724 us 19.811 us
RunBlock 1952842 9,365.22 us 60.028 us 53.213 us
RunBlock 1955263 86,541.97 us 550.152 us 514.612 us
RunBlock 1955515 7,022.05 us 17.290 us 14.438 us
RunBlock 1976901 93.24 us 0.244 us 0.217 us
RunBlock 1976901 92.88 us 0.301 us 0.235 us
RunBlock 1991342 2,426.50 us 13.010 us 11.533 us
RunBlock 2027305 486.60 us 2.462 us 2.303 us
RunBlock 2037494 2,436.23 us 16.222 us 15.174 us
RunBlock 2080810 2,783.88 us 4.536 us 4.021 us
RunBlock 2116627 533.59 us 2.332 us 1.820 us
RunBlock 2137993 1,056.17 us 6.795 us 6.356 us
RunBlock 2141425 1,357.55 us 9.609 us 8.989 us
RunBlock 2141551 2,365.64 us 10.608 us 9.923 us
RunBlock 2158835 1,345.29 us 4.930 us 4.117 us
RunBlock 2171359 981.88 us 6.084 us 5.691 us
RunBlock 2173910 153.20 us 0.366 us 0.343 us
RunBlock 2194622 1,265.40 us 9.031 us 8.448 us
RunBlock 2212206 18,358.81 us 104.430 us 92.575 us
RunBlock 2212212 17,438.71 us 69.607 us 61.704 us
RunBlock 2212242 25,124.72 us 195.285 us 173.115 us
RunBlock 2212242 24,865.63 us 99.060 us 92.660 us
RunBlock 2212261 14,760.83 us 105.185 us 93.244 us
RunBlock 2212278 22,889.89 us 116.122 us 102.939 us
RunBlock 2212278 23,033.16 us 128.871 us 120.546 us
RunBlock 2212440 23,656.93 us 132.009 us 117.023 us
RunBlock 2212440 23,627.42 us 197.766 us 175.315 us
RunBlock 2212454 22,416.27 us 112.288 us 99.541 us
RunBlock 2212544 33,278.06 us 215.633 us 191.153 us
RunBlock 2212544 33,285.16 us 235.180 us 219.988 us
RunBlock 2212575 33,824.69 us 333.371 us 311.835 us
RunBlock 2212575 33,998.52 us 148.037 us 138.474 us
RunBlock 2212681 45,675.58 us 489.595 us 434.013 us
RunBlock 2212681 67,040.36 us 426.894 us 399.317 us
RunBlock 2212706 48,598.08 us 580.829 us 543.308 us
RunBlock 2212723 23,430.01 us 117.879 us 104.497 us
RunBlock 2212723 23,748.91 us 146.291 us 136.841 us
RunBlock 2212800 23,682.75 us 170.892 us 142.702 us
RunBlock 2212800 23,751.40 us 131.295 us 122.814 us
RunBlock 2212807 23,975.44 us 478.724 us 686.571 us
RunBlock 2212833 24,229.67 us 263.338 us 246.327 us
RunBlock 2212833 35,471.56 us 307.205 us 287.360 us
RunBlock 2230072 1,333.59 us 6.202 us 5.801 us
RunBlock 2268049 872.60 us 3.547 us 3.318 us
RunBlock 2351614 774.40 us 2.898 us 2.420 us
RunBlock 2363914 1,993.98 us 12.808 us 11.354 us
RunBlock 2398019 35,327.16 us 328.741 us 307.505 us
RunBlock 2398072 41,127.87 us 254.150 us 237.732 us
RunBlock 2501961 566.76 us 1.689 us 1.410 us
RunBlock 2508698 32,638.17 us 173.260 us 153.591 us
RunBlock 2655903 5,862.38 us 12.115 us 10.740 us
RunBlock 2692530 83,346.31 us 426.751 us 399.183 us
RunBlock 3053852 30,989.33 us 181.892 us 161.242 us
RunBlock 4222547 54.36 us 0.233 us 0.195 us
RunBlock 4229846 31,076.64 us 172.004 us 160.893 us

@Jim8y Jim8y mentioned this pull request Jan 26, 2024
Merged
12 tasks
@shargon
Copy link
Member

shargon commented Feb 6, 2024

conflicts

@Jim8y Jim8y closed this Feb 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shargon shargon shargon left review comments

Assignees
No one assigned
Labels
Waiting for Review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Jim8y @cschuchardt88 @shargon