Benchmark modexp precompiled contract in different clients

[pirapira]

There is an opinion that the gas cost formula of modexp precompiled contract can be improved. To find out the reasonal gas cost formula we need to benchmark modexp contract of different clients up to blockgaslimit.

[timxor]

@pirapira i can take that, looks fun. will work on it tonight

[pirapira]

Great. You can choose

• to prepare and run a BlockchainTest on each client

or

• to call a routine in each client
  • https://github.com/obscuren/go-ethereum/blob/f0f23fc1739fbef5759802faad564e3e65b79395/core/vm/contracts.go#L180
  • https://github.com/ethereum/cpp-ethereum/blob/develop/libethcore/Precompiled.cpp#L123
  • https://github.com/paritytech/parity/pull/4926/files#diff-f01cea79b5ea6391c260249a4cf801a5R234

or

• to join efforts with core/vm: add benchmarks go-ethereum#14641 and facilitate something similar for other clients

[timxor]

@pirapira what option do you think would be better? Go it solo or join forces....maybe solo because then it goes horizontal not vertical :)

[pirapira]

That's what I thought. The second option will be the quickest.

[timxor]

agreed. on it.

[holiman]

I have a geth-benchmark implementation for modexp at https://github.com/holiman/go-ethereum/tree/benchmark_metro , the only thing I'm missing now are some proper values to use in testing. And I guess the other clients ought to use the same. Have either of you located any good inputs? I've found a few from the EIP examples, but I don't think they're very representative.

[timxor]

This is awesome thanks for sharing, can we make a generator for testing all ranges or what do you think the best way to test it? Are there already tests in the test repo that we can use? @holiman

[holiman]

I've added some inputs to my branch the other day

[holiman]

https://gist.github.com/holiman/d2a59ebc9353baffaf21389e296987c1#gistcomment-2128792

[gumb0]

@tcsiwula Hi, is there any update from you on this?

[timxor]

Hey @gumb0, I should have the parity one done by friday this week. I was interviewing for jobs the last two weeks. Is the c++ client still in active development? cc: @pirapira

[pirapira]

@tcsiwula I sent you an email.

[winsvega]

to test benchmark we have to run a scenario with calling a function/opcode on different range of input parameters and eventually build up a graph "exec time / input parameters value"
I think making a static contract in state test is not a best option for determining the reasonable gas cost formula

[timxor]

Hello everyone, here is an update from me. Let me know where I should focus my attention/efforts next. =)

My most recent (August 9th) parity benchmark results for {bn_128_add, bn_128_mul, ...}.

Here are the steps to recreate and run it locally on your computer (linux/ubuntu).

cc: @pirapira @holiman @arkpar @Souptacular

[pirapira]

On my machine, I get 6 times different numbers for ecadd from
go-ethereum

BenchmarkPrecompiledBn256Add/chfast1-Gas=0-4                       20000             90297 ns/op

and Parity

test bn_128_add          ... bench:      14,527 ns/iter (+/- 413)

(will edit with links to steps)

[pirapira]

The original purpose of this issue has been accomplished, as modexp gas formula has been determined.

Since we are here, maybe we can open issues for remaining problems and paste links here.

[timxor]

@pirapira wonderful, yeah that sounds good to me. What other precompile benchmarks would be useful to have that would give us a complete and accurate picture?

[pirapira]

@tcsiwula since ecadd and ecmul are supposed to be of constant gas costs, it would be great if we can come up with some numbers already.

[gcolvin]

@tcsiwula just pointed me at this issue. I've been working with @s-matthew-english on improving the test suite at https://github.com/ethereum/cpp-ethereum/tree/develop/test/unittests/performance so that more opcodes and precompiles and more clients can be easily added to the suite. The more the merrier. The advantage of this suite is that it will work with any client with a command-line program that can interpret EVM binaries.

[winsvega]

@pirapira status?

[pirapira]

No idea.

[winsvega]

I thought we came to the final gas price for this contract. @holiman

[holiman]

Yes, for all precompiles. I think we can close this ticket.. (?)

[pirapira]

OK.