Skip to content
This repository has been archived by the owner on Nov 8, 2021. It is now read-only.

Reproducibility across different platforms #3

Open
tevador opened this issue Jun 2, 2018 · 30 comments
Open

Reproducibility across different platforms #3

tevador opened this issue Jun 2, 2018 · 30 comments

Comments

@tevador
Copy link
Owner

tevador commented Jun 2, 2018

I added a new test project which can be used for testing results on different platforms.

If you want to test it on your platform:

  1. Start the sandbox node sandbox.js
  2. Run command ./Tevador.RandomJS.Test.exe --threads 2 --seed 123 --count 1000

The number of threads can be adjusted according to the number of cores of your processor. The command generates 1000 random programs and calculates a cumulative hash of all the outputs.

So far I have tested:

  • Intel Ivy Bridge, Windows x86_64
  • Intel Sandy Bridge, Ubuntu x86_64
  • AMD Ryzen, Ubuntu x86_64
  • Raspberry PI, Ubuntu armv7l
  • Intel Kaby Lake R, Windows x86_64

In all cases the result is 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc.

(Valid as of commit 79d3c4c)

@ghost
Copy link

ghost commented Jun 3, 2018

  • Kaby Lake Intel Core i7-7500U CPU @ 2.70GHz
  • Ubuntu 17.10, x86_64
  • As of commit: 79d3c4c

./Tevador.RandomJS.Test.exe --threads 2 --seed 123 --count 1000 Collecting statistics from 1000 random program executions (seed = 123) 5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %, Completed in 8.4801327 seconds Runtime histogram: Longest runtimes: Seed = 92117886952e3f3c5010956afba334430a20eaa983b4ee5f6c8647e5010cf6b4, Runtime = 0.05701 s Seed = f767b319d74bdb44329dfeb4072b558bb89bbcd2dadef61b863f06f4c0f84365, Runtime = 0.05645 s Seed = fb0140e24f67a86b8d5da25870f056264d210a1b8a8fa678f86edfe0d17904f4, Runtime = 0.05559 s Seed = 0e1fd5a6d18e5cc2d0fa9cfb781be7f33f4f11a00c0a7b1085142192291d9bfc, Runtime = 0.05529 s Seed = 3a465dcc93206126d4c68d24b8d0854e95fd4d1be6ce3fb18583792997fb0602, Runtime = 0.04514 s Seed = d6e3a869b2518a2e5076cd498b82bb76297779cd9f1bf3121e0f2a74a37fd90c, Runtime = 0.04164 s Seed = beed8f13cc2af4ab76ec0a73c9b286a6c499571d8f554983d5d6fd46bbb654a2, Runtime = 0.04049 s Seed = 39b432618eb3fe0e374033e5bd24e28abc78a6f55fa576fe4a01c03fff5a8ff5, Runtime = 0.03823 s Seed = 46509a1b7306a0929fbfed991ac472499a5fb1c569d4e3ced8863c23eee61eb2, Runtime = 0.03597 s Seed = 73b8ec5cd13343519cee1ef7662121c961f778ed925c8d37665e0b6b2a68f41a, Runtime = 0.03493 s Runtime [s] Min: 0.00167; Max: 0.05701; Avg: 0.00771; Stdev: 0.00640; Runtime [s] 99.99th percentile: 0.0570068010129035 Average entropy of program output (est.): 1242.39 bits Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0; Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0; 0.00167 17 0.00231 60 0.00295 101 0.00359 106 0.00423 97 0.00487 90 0.00551 70 0.00615 62 0.00679 52 0.00743 42 0.00807 44 0.00871 27 0.00935 27 0.00999 20 0.01063 22 0.01127 22 0.01191 15 0.01255 10 0.01319 8 0.01382 12 0.01446 10 0.01510 6 0.01574 3 0.01638 9 0.01702 8 0.01766 4 0.01830 2 0.01894 5 0.01958 5 0.02022 3 0.02086 3 0.02150 4 0.02214 5 0.02278 1 0.02342 1 0.02406 2 0.02470 0 0.02534 0 0.02598 2 0.02662 4 0.02726 0 0.02790 1 0.02854 1 0.02918 0 0.02981 0 0.03045 3 0.03109 1 0.03173 0 0.03237 1 0.03301 0 0.03365 2 0.03429 1 0.03493 0 0.03557 1 0.03621 0 0.03685 0 0.03749 0 0.03813 1 0.03877 0 0.03941 0 0.04005 1 0.04069 0 0.04133 1 0.04197 0 0.04261 0 0.04325 0 0.04389 0 0.04453 1 0.04517 0 0.04580 0 0.04644 0 0.04708 0 0.04772 0 0.04836 0 0.04900 0 0.04964 0 0.05028 0 0.05092 0 0.05156 0 0.05220 0 0.05284 0 0.05348 0 0.05412 0 0.05476 1 0.05540 1 0.05604 1 0.05668 1 Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc

@tevador
Copy link
Owner Author

tevador commented Jun 3, 2018

@wowario Your first result was different. Any idea why?

@ghost
Copy link

ghost commented Jun 3, 2018

Yes, it was. The first test ran on an older version of ProgramOptions.xml with different weight values, I re-cloned the repo and ran the test a second time to get the correct result.

@jwinterm
Copy link

jwinterm commented Jun 4, 2018

Tested with Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz
Ubuntu 18.04 x86_64
commit 79d3c4c

Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc

@tevador
Copy link
Owner Author

tevador commented Jun 4, 2018

We should test an x86 (32bit) platform (not that I'm expecting anyone to mine on such machine). Also more AMD CPUs (they can give different results for transcendental math functions).

@jwinterm
Copy link

jwinterm commented Jun 4, 2018

I have a couple oldish AMD boxes around, don't think I have any x86s though. Will try to test on AMDs in next couple days.

@jwinterm
Copy link

jwinterm commented Jun 5, 2018

Tested on -
AMD Athlon(tm) II X2 220 Processor
Ubuntu 16.04 x86_64
commit: 8923b3b

got:
Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc

@bomb-on
Copy link

bomb-on commented Jun 5, 2018

Tested on:
OS X 10.13.5
Intel Core i5 2.7GHz
commit: 8923b3b

Results:

$ mono Tevador.RandomJS.Test.exe --threads 2 --seed 123 --count 1000

Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %,
Completed in 25.1238964 seconds
Longest runtimes:
Seed = 0e1fd5a6d18e5cc2d0fa9cfb781be7f33f4f11a00c0a7b1085142192291d9bfc, Runtime = 0.38644 s
Seed = 46509a1b7306a0929fbfed991ac472499a5fb1c569d4e3ced8863c23eee61eb2, Runtime = 0.25880 s
Seed = 39b432618eb3fe0e374033e5bd24e28abc78a6f55fa576fe4a01c03fff5a8ff5, Runtime = 0.22791 s
Seed = 2822106c992d76c830b35009fa847da93520c507e5190a48f5357a11a607356b, Runtime = 0.19583 s
Seed = 92117886952e3f3c5010956afba334430a20eaa983b4ee5f6c8647e5010cf6b4, Runtime = 0.16995 s
Seed = d8fe9d7d7855c0d2c6a81564efe881b147cd17d35410adfc609f6ef81d4f9c57, Runtime = 0.16821 s
Seed = fb0140e24f67a86b8d5da25870f056264d210a1b8a8fa678f86edfe0d17904f4, Runtime = 0.15999 s
Seed = e53b00a5e8db6f7295a5de8883ea7ead3600067f8f729882c25bc3f53efa79ac, Runtime = 0.15155 s
Seed = 42f0b1623c4650d47f8348bbeee27edcf9ed5ab50f3d783b2cecc7ffa8be7d8d, Runtime = 0.14507 s
Seed = 7fb4dbf4c3b50b4317176ca78a4d1478084d7e3ff994229ccd373d8ab9e7c54a, Runtime = 0.13481 s
Runtime [s] Min: 0.00323; Max: 0.38644; Avg: 0.02417; Stdev: 0.02796;
Runtime [s] 99.99th percentile: 0.386439353023889
Average entropy of program output (est.): 1242.39 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc

@tevador
Copy link
Owner Author

tevador commented Jun 7, 2018

Note that as of commit d32fd77, the cumulative hash is different because the generator has been slightly modified.

The new hash is:

$ ./Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2
Collecting statistics from 1000 random program executions (seed = 123)
...
Cumulative output hash: 80d77dae86b3b49bfb4cf53281f10cb5db1d1e857d9adb7429bf116a9d7880ba

If you do git pull, you have to manually update your ProgramOptions.xml (copy from src/Tevador.RandomJS/ProgramOptions.xml to bin).

@tevador
Copy link
Owner Author

tevador commented Jun 20, 2018

I decided to remove all transcendental functions (exp, log, sin, cos, cbrt, atan) from the generator due to poor cross-platform compatibility (rounding is not enough).

With these functions enabled, the results don't match between Windows and Linux with the XS engine.

The reason why the results match for the V8 is because V8 has its own software implementation of all these functions using only IEEE-754 safe operations. We could copy this implementation to the XS, but I don't see the point in that because an ASIC could easily include a faster on-chip implementation.

The only function that stays is Math.sqrt which is portable and implemented in hardware in most CPUs.

For testing of floating point math compatibility, please use the following command:

Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0

It generates programs with only math expressions and doesn't depend on ProgramOptions.xml.

Output with commit e136756 should be:

Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c

It is now possible to also test with the XS engine. You have to clone and compile it from here: https://github.com/tevador/moddable

To use this engine, append the following command line option: --runnerUri file://PATH-TO-XST-EXECUTABLE, for example:

--runnerUri file:///home/tevador/moddable/build/bin/lin/release/xst
--runnerUri file://C:/projects/moddable/build/bin/win/release/xst.exe

For the NodeJS sandbox, you can use (it's the default option):

--runnerUri http://localhost:18111

@ghost
Copy link

ghost commented Jun 23, 2018

  • Intel i7 3770
  • NixOS e70194, node v10.3.0, mono 5.8.0.108, RandomJS e136756
mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 ..

..

Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c

@tevador
Copy link
Owner Author

tevador commented Jun 23, 2018

@fuwa0529 b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c is the correct hash for that commit.

I have also tested the latest commit from the dev branch on Intel Kaby Lake (Windows), AMD Ryzen and Raspberry Pi 3 (Ubuntu) and all give the same results.

I think at this point we can conclude that reproducibility is not an issue.

@SChernykh
Copy link
Contributor

I have also tested the latest commit from the dev branch on Intel Kaby Lake (Windows), AMD Ryzen and Raspberry Pi 3 (Ubuntu) and all give the same results.

I'm curious about performance numbers for all these processors. Can even Raspberry Pi 3 be used for mining?

@bomb-on
Copy link

bomb-on commented Jun 28, 2018

Tested on:
OS X 10.13.5
Intel Core i5 2.7GHz
commit: e136756

Results:

$ mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0

Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %,
Completed in 7.6793444 seconds
Longest runtimes:
Seed = 86460416fe8be2c82375ee36c6b48b3c0356f2d9270ff5f856e7202706020a15, Runtime = 0.04320 s
Seed = f74ea47f971bdfe927a67e0eb8bcb0836fc21838dfb69e7ed179defc424e289b, Runtime = 0.03316 s
Seed = 6fa68338a1d4656b071b682aa7828b73dafb4ba8b90f7bc82f384042adb5ff2f, Runtime = 0.03298 s
Seed = fdb87aac7f31faee0aa653df81007761c0cee3f436b3cb0ecb4b2e19932041f5, Runtime = 0.03287 s
Seed = 1a9d352a9af1e6c08bceba71ffe4fbfb7f61c06c8777bdc344f8d010fc74f77f, Runtime = 0.02954 s
Seed = a894dbd527ebf7e1669316b0f7d9b028b33c72c0a341d586d4d32f2e22daad23, Runtime = 0.02395 s
Seed = 3f6524e301d20266e08aca271a4248f6ad025756539a637aaca6bf951a393d96, Runtime = 0.01987 s
Seed = 1fde2adc66eb993b77be43a0df7b313c5908eaa118b78a6edbaea223d47067c1, Runtime = 0.01925 s
Seed = 0021020d8391b1c4a9009b676f61245d7625f4c7fe2b64d06bd304c8f4bf9360, Runtime = 0.01836 s
Seed = c309c0a2c910a6f472259c73169125ccd33b111748ed96c1c75b13a17e21b562, Runtime = 0.01786 s
Runtime [s] Min: 0.00308; Max: 0.04320; Avg: 0.00710; Stdev: 0.00324;
Runtime [s] 99.99th percentile: 0.0431984979659319
Average entropy of program output (est.): 6739.48 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Matches 'SyntaxError' optimization: 0.00 %
'SyntaxError' optimization runtime: 0.00 %
Matches 'XS': 0.00 %
Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c

@tevador
Copy link
Owner Author

tevador commented Jun 28, 2018

@SChernykh

I'm curious about performance numbers for all these processors. Can even Raspberry Pi 3 be used for mining?

Commit 4fc608c, 1 core, XS engine:

  • Intel Kaby Lake R @~3.5 GHz (turbo) - 8 ms per program
  • Intel Sandy Bridge @ 3.7 GHz - 9.3 ms per program
  • Raspberry Pi 3 @ 1.2 GHz - 58 ms per program

So the Raspberry is about 6-7 times slower per core. It consumes ~5 W when mining with all 4 cores, but you need some kind of cooling solution because it will start thermal throttling quickly. The performance per watt seems to be comparable to the most efficient laptop processors, so it's definitely usable for mining.

@jwinterm
Copy link

jwinterm commented Jul 1, 2018

OK, tested again with:
AMD Athlon(tm) II X2 220 Processor
Ubuntu 16.04 x86_64
commit: e136756

Got the same result as others using the options suggested:

jw@couch:~/RandomJS/src/bin$ mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0
Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 20 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %, 
Completed in 10.1174143 seconds
Longest runtimes:
Seed = 69338483f2db38253138ce5df46054706efa4ef560f1c5ff0613988e171b5160, Runtime = 0.02479 s
Seed = 1bee5af3511ebdbe3a67ace3450099f7017a2250ad4ff7f04c1b0a6bd41bcd9b, Runtime = 0.02241 s
Seed = 8debbf497bce946b21d31cb72b3ce444f73c68192896c058520e5fbc8dd1323a, Runtime = 0.02213 s
Seed = c8c7da935221e7b9adf811d532418ad349cea172574975fade635c0bde95718f, Runtime = 0.02132 s
Seed = 3d569937c30d61296a232bc174f24078e9ad4fa5010d9df89714750c2f0d2e2f, Runtime = 0.01967 s
Seed = 05257e9f3a45ba51156bc26681dd628e99ecc990486de7b6926946e43acb3915, Runtime = 0.01910 s
Seed = e7a7b232a38def90fa1abd08d2839ab5640894fd5b78ae8d441c63383c5a488e, Runtime = 0.01903 s
Seed = 6cf3be599f81d43ca2e8cce3c5a0707bad7890048eb8e03864712249f332b073, Runtime = 0.01891 s
Seed = 595a12c78615370757ad264311450d181f1089869cb59d897eaeed082046b481, Runtime = 0.01854 s
Seed = d561beeae65df624e12294d63e7ac4a227a1fc5dfb928c07cbfd37edc9d318af, Runtime = 0.01706 s
Runtime [s] Min: 0.00437; Max: 0.02479; Avg: 0.00749; Stdev: 0.00288;
Runtime [s] 99.99th percentile: 0.0247909612953663
Average entropy of program output (est.): 6739.48 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Matches 'SyntaxError' optimization: 0.00 %
'SyntaxError' optimization runtime: 0.00 %
Matches 'XS': 0.00 %
Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c


@SChernykh
Copy link
Contributor

So the Raspberry is about 6-7 times slower per core. It consumes ~5 W when mining with all 4 cores, but you need some kind of cooling solution because it will start thermal throttling quickly. The performance per watt seems to be comparable to the most efficient laptop processors, so it's definitely usable for mining.

Huh, and considering 40$ price it will be the most profitable option: 8x Raspberry Pi 3 will cost 320$ and will be faster than Kaby Lake. Future mining rigs will look like this: https://www.youtube.com/watch?v=i_r3z1jYHAc

@stoffu
Copy link

stoffu commented Jul 2, 2018

  • OS X 10.12.6
  • Intel Core i7 2.9GHz
  • commit: e136756
RandomJS/src/bin$ mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0
Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 30 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %, 
Completed in 8.1185582 seconds
Longest runtimes:
Seed = 21caf72218e8d1a19652a2679ae35970f84da1cf4eb437cf5bc0c65af08001a3, Runtime = 0.00976 s
Seed = 56d56c8f1a814f493d9f1ee5433de59a18971b2eb59f3a69f90679c866a98749, Runtime = 0.00871 s
Seed = 0b5499ef493be6e4b4851d0c49f203429b9614803f113c9919619c3e619052f2, Runtime = 0.00869 s
Seed = 39f3adaba09054c00e082b417b67930636c7f2b98f83e3481ccd8e27157ec5cb, Runtime = 0.00773 s
Seed = 20b1d24cda3e4257e3d9409e6f55fd983ca277e2d9a3994b48057b88e228c2bc, Runtime = 0.00750 s
Seed = c6b4bcd6199322b38f3dec1bd57a00e149183a283b92fd1a80ec93e7e3059959, Runtime = 0.00745 s
Seed = c645cd0948f1ff0f416ce886d8be805b90b774d4d6304ebd3201d4f986ab9058, Runtime = 0.00735 s
Seed = 816151bf8790a0a75a63de4e9b5e9fe64df33e658215721a6689ea0ca8d0682d, Runtime = 0.00719 s
Seed = e22a901ab382e93965cdc190720a9bd4fedad09d603968f962709cbbe8c1a8e0, Runtime = 0.00718 s
Seed = 0395d8b573caa0ad21a549292cc2e56b0531e530c73d3f975f9e2b2346cff08d, Runtime = 0.00712 s
Runtime [s] Min: 0.00268; Max: 0.00976; Avg: 0.00418; Stdev: 0.00096;
Runtime [s] 99.99th percentile: 0.0097556000109762
Average entropy of program output (est.): 6739.48 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Matches 'SyntaxError' optimization: 0.00 %
'SyntaxError' optimization runtime: 0.00 %
Matches 'XS': 0.00 %
Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c

@antanst
Copy link

antanst commented Jul 6, 2018

  • Macbook Pro 13 (2017) - MacOS 10.13.5
  • Intel(R) Core(TM) i5-7267U CPU @ 3.10GHz
  • commit e136756
$ mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0
Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %,
Completed in 4.7085095 seconds
Longest runtimes:
Seed = 10324c0a8b1c464df6b2cb01dae9bed9d9bf6b5259bf878e699aceb4abf137b7, Runtime = 0.01711 s
Seed = 1ea883a9d1907c1df7e7a34138b0d85077047cb1461739e0e3d8dc283992609c, Runtime = 0.01609 s
Seed = 0e2a96206861eff978d52453296e9e8d7f6bd82e33e1331d1787d407a00e623d, Runtime = 0.01343 s
Seed = 06758e059356767db38bc96bee562e940278fef61aa2270858db9f0a425a232d, Runtime = 0.01181 s
Seed = baec1d3a2b533c4fa2011cdaf4e7089eec3f4877fe59efd093240b0be1cd0d40, Runtime = 0.01073 s
Seed = ab92adcefe0b3c6523bffb5251bd8adc3856c1404dc206ea44611ee806174f0f, Runtime = 0.00992 s
Seed = e22a901ab382e93965cdc190720a9bd4fedad09d603968f962709cbbe8c1a8e0, Runtime = 0.00990 s
Seed = 9709bbcd7b1c158b165bc4e43bdab9a3f0f53a07586fa0b1cceab4c70c19f1d9, Runtime = 0.00896 s
Seed = dcd137ef89f0e5b6813068076884ff6cbaf3ea8b05bda014d1587409981b97ff, Runtime = 0.00855 s
Seed = af9980ddbdee7addcea010a75c55d3d0e221fff6eb6775e816fac3f1818292ea, Runtime = 0.00824 s
Runtime [s] Min: 0.00229; Max: 0.01711; Avg: 0.00385; Stdev: 0.00125;
Runtime [s] 99.99th percentile: 0.0171129430236761
Average entropy of program output (est.): 6739.48 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Matches 'SyntaxError' optimization: 0.00 %
'SyntaxError' optimization runtime: 0.00 %
Matches 'XS': 0.00 %
Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c

@antanst
Copy link

antanst commented Jul 7, 2018

Also tested with Node-Chakracore v10.1.0 official release binaries. Same system as previous post. Results are different.

$ mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0
Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %,
Completed in 8.3624088 seconds
Longest runtimes:
Seed = c98987a1924460048751a7ed8826dedccf35127302ccf677f7d31863d1824547, Runtime = 0.03301 s
Seed = d561beeae65df624e12294d63e7ac4a227a1fc5dfb928c07cbfd37edc9d318af, Runtime = 0.02909 s
Seed = f0347fedcd64695e23d126d4101bb36504979ca52d935c472bcfff348b107da2, Runtime = 0.02673 s
Seed = 4079a249630fe6a1425fa1e04d8d03fcf90001749878c54f5486359888ac554b, Runtime = 0.02255 s
Seed = ccce6bbfd69adaa61606abe6d3a15c8b2ea1dfb77cf9c31a40edbb823befb189, Runtime = 0.01903 s
Seed = 4c193b1c61cb20c405dc687e3854149ac78440e748ea90dfac9a37c143ac7aa9, Runtime = 0.01871 s
Seed = 44474a04ffd6aa03fc4f0bd71d45124276742460fc5a9504a084a9a68cdb3d2d, Runtime = 0.01860 s
Seed = a2c85adc16d97ff5f4bf356034d9a7749c8df41a7f60e57fe4e4ecf69ef15eb7, Runtime = 0.01771 s
Seed = 6d100db791b0bc891a5bf9dd0645b34007b1910828ad1db5da4bf043212fd3ee, Runtime = 0.01537 s
Seed = daf7cf653790ee9a2c74567a5da93d2d89ea4f35aa48bc18edcb87f1fcd26d80, Runtime = 0.01471 s
Runtime [s] Min: 0.00523; Max: 0.03301; Avg: 0.00816; Stdev: 0.00209;
Runtime [s] 99.99th percentile: 0.0330056299862918
Average entropy of program output (est.): 6739.20 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Matches 'SyntaxError' optimization: 0.00 %
'SyntaxError' optimization runtime: 0.00 %
Matches 'XS': 0.00 %
Cumulative output hash: b87b2edee0e6a496f1ec2fcb3dd3f56e0db0d37b25acb92772eb9e0d8109ae36

@tevador
Copy link
Owner Author

tevador commented Jul 7, 2018

@antanst We are no longer aiming for binary compatilibity with all JS engines. The XS engine will be used as reference and some results will not match with other engines. The most important thing is that results match across platforms with the same engine.

Nevertheless, you can try to find what causes the different outputs like this:

  1. Use just one thread --threads 1 (this is needed to make sure programs are always processed in the same order).
  2. Append --debug option to the command line. This will output the hashes of all programs.
  3. Redirect the output to a file: mono Tevador.RandomJS.Test.exe ..... > output1.txt
  4. Run the same command with the other engine and redirect to output2.txt.
  5. Use diff output1.txt output2.txt to find which seeds produce different hashes. Seed is the value before colon :.
  6. Run mono Tevador.RandomJS.exe *seed* > program.js to generate the exact program that causes the mismatch.
  7. Then you can run node fast-eval.js < program.js to compare the outputs. You can also use diff to see which lines differ. From here it's quite easy to find the mathematical operation that causes the mismatch.

@antanst
Copy link

antanst commented Jul 7, 2018

Ok. Apologies for spamming the thread. Since you asked, I've followed your steps and traced the difference to this code output:

Code:

Function.prototype.toString = function() {
    return '[Function]' + this.name;
};
g = 1
g += function() {}
console.log(g)

Outputs of vanilla Node 8 and 10:

$ node fast-eval.js < ./test.js
1[Function]

Output of Chackra Node v10:

$ ./node-v10.1.0-darwin-x64/bin/node fast-eval.js < ./test.js
1[Function]g

Looks like a vanilla Node bug.

@tevador
Copy link
Owner Author

tevador commented Jul 7, 2018

Interesting find. To me it looks like Chackra Node v10 bug, because the anonymous function doesn't have name g (should be empty).

@JohannesLau
Copy link

Ryzen 1700x
Windows 10.0.17134 Build 17134
C:\Users\Johannes\Downloads\RandomJS-master\RandomJS-master\src\Tevador.RandomJS.Test\bin\Debug>Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0 Collecting statistics from 1000 random program executions (seed = 123) 5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %, Completed in 5.3005916 seconds Longest runtimes: Seed = 6eda4048d38469bb1a003a1fa77905ef394e57eb23d030b2e417440382adb38b, Runtime = 0.02108 s Seed = 1a1badef09d9f7ac65a6c798369b9afdb3bac23de7f7cbd98c532340d27c834f, Runtime = 0.02046 s Seed = 46509a1b7306a0929fbfed991ac472499a5fb1c569d4e3ced8863c23eee61eb2, Runtime = 0.02037 s Seed = e635da2a2215b8a98377d1057b9fa17fcfe7238a9de3722a59d80e8bf16aac8f, Runtime = 0.02011 s Seed = f525016ba483981a4faf26f2892f0033c0ca2bb4d1929eabc90aadbd021c9ed1, Runtime = 0.01887 s Seed = 8416a772f22270ab74c50934ca568d1cafed1fafe35d542494633c713ca1bb2a, Runtime = 0.01855 s Seed = 5b58f2d9d91e0638bbb522be955e3700decb7f281736329c0c76cc0a329718a8, Runtime = 0.01846 s Seed = f69690e35ed0063aac349ac68fccceeb0d9937e1cc6ea29e089823b2bbd4c9f2, Runtime = 0.01839 s Seed = 787102c377b98a079e9e522cd453f233fc8919fee7b512e5f1a2863b5701c492, Runtime = 0.01740 s Seed = dedc8ea24911abbed6e14398ad677fda754f679815e894bda3cdc5935d26841e, Runtime = 0.00834 s Runtime [s] Min: 0.00245; Max: 0.02108; Avg: 0.00408; Stdev: 0.00186; Runtime [s] 99.99th percentile: 0.0210800130000735 Average entropy of program output (est.): 6739.48 bits Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0; Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0; Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0; Matches 'SyntaxError' optimization: 0.00 % 'SyntaxError' optimization runtime: 0.00 % Matches 'XS': 0.00 % Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c

@timolson
Copy link

timolson commented Oct 11, 2018

The fact that Rasberry PI is an efficient option should be a huge warning flag that this is not an ASIC resistant PoW. These numbers indicate that the additional complexity of Intel’s branch prediction, cache lookahead, etc is not worth the gates. An ASIC designer will be writing a very simple, small-gate CPU-type thing which runs direct execution from the generator algo with no parsing, interpretation, bytecode, object code, or anything else needed in general purpose CPU’s. An ASIC will put lots of slow but small cores into a chip, and the design will not take long because they can exclude all the fancy stuff Intel and AMD do. ARM cores are easily licensable and customizable for particular applications.
I’ve tried to say this in other ways in other threads, but these numbers back it up.

@SChernykh
Copy link
Contributor

Specialized ARM core with proper core-to-cache ratio will be ~2 times more power efficient. I don't see a big problem with it. Many manufacturers can produce ARM cores because you said yourself they're easily licensable. This ARM-based hardware will be just like tuned Raspberry Pi, it can/will be used for other stuff apart mining.

@timolson
Copy link

timolson commented Oct 11, 2018

2x is huge in mining. As we have seen, hashrate will increase to the point where it’s barely profitable vs electricity. A 2x ASIC means home mining on CPU is not profitable. Everyone will need the ASIC.

@SChernykh
Copy link
Contributor

CPU mining is a different world. First, having just single CPU + motherboard + RAM + power supply is already very inefficient. Buying specialized ARM boards and connecting them to the same power bus will be the way to go: https://www.youtube.com/watch?v=i_r3z1jYHAc&t=284s
These ARM boards will be easy to buy like RPi now.

@SChernykh
Copy link
Contributor

There will also be PCI-e boards with many small ARM cores like Xeon Phi. And we'll be back to making rigs on PCI-e risers 💯

@timolson
Copy link

timolson commented Oct 11, 2018

It’s true. That world would be ok. But I’m not convinced I can’t make an ASIC that’s faster than the ARM for running RandomJS programs only.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants