Catch up summary 2018-11-21

[alanshaw]

Apologies I wanted to get this done yesterday. Hopefully this captures our conversation:

Summary

@Elexy gave a presentation demoing the current setup. 3 basic benchmark tests currently.

Infrastructure requirements were ironed out with @eefahy. See conversation here ipfs/infra#452 (comment) for details.

@mcollina identified a perf issue for node initialization and documented here ipfs/js-ipfs#1727

Agreed a weekly catch up at the same time (Wednesday 15:00).

Action items

• Update infra diagram (https://github.com/ipfs/benchmarks/blob/master/docs/images/prod-infrastructure.png) (@Elexy) - updated image #15
• @Elexy to provide @eefahy with public key for access to runner vm
• Infrastructure is set up (@eefahy)
• README to be updated with repo layout, purpose, infra image, URLs for deployed services etc. (@litzenberger @Elexy)
• Add instructions for how to run locally (@litzenberger)
• Add instructions for how to add benchmarks (@litzenberger) Create a guide on how to add new benchmarks #17
• Instruct Nearform on pregenerated key usage (@alanshaw)
• Investigate potential issue running benchmarks with existing repo (@alanshaw)
• Run locally and feedback (@alanshaw, @achingbrain, @hugomrdias)

[alanshaw]

You can specify the bits for the generated key - 512 is really fast but you can elminate it completely using the privateKey param https://github.com/ipfs/js-ipfs#optionsinit

Note that you might want to use this module for spawning nodes https://github.com/ipfs/js-ipfsd-ctl which might come in useful when we start benchmarking against a Go IPFS daemon.

In the catch up we spoke about whether the node initialization time should be part of the benchmarking tests and we decided it should not. Can we add a simple initialization speed benchmark for js/go?

[mcollina]

@alanshaw can you please verify if the info in https://github.com/ipfs/benchmarks#run-tests-locally is sufficient for you to run things locally?

[mcollina]

You can specify the bits for the generated key - 512 is really fast but you can elminate it completely using the privateKey param https://github.com/ipfs/js-ipfs#optionsinit

Is a key of length 512 secure enough? I think the current recommendation is 2048 for RSA. If it is safe, why is not the default? What are the practical limitations of using a pre-generated key in real-world applications? I don't think a user could pre-generate a key, or that could be generated and downloaded from somewhere else, right?

[Elexy]

just a thought; can't you generate the key as part of npm install ?

[alanshaw]

Is a key of length 512 secure enough? I think the current recommendation is 2048 for RSA. If it is safe, why is not the default?

2048 is the default - 512 is not secure enough. In the future we need to restrict the bits parameter to discourage users from generating insecure keys. The Go implementation already does this with the minimum set at 1024.

What are the practical limitations of using a pre-generated key in real-world applications? I don't think a user could pre-generate a key, or that could be generated and downloaded from somewhere else, right?

I'm not 100% sure I understand the question. I suggested these methods as potential solutions for initialising nodes in the benchmarking tests to avoid a long initialization time. I'm not recommending them as best practice for building real-world apps.

just a thought; can't you generate the key as part of npm install ?

I like this idea! It could potentially work if a user is installing ipfs globally (npm i -g ipfs) but as a dependency we can't know the repo location (where the key should be stored) at this time.

[mcollina]

I'm not 100% sure I understand the question. I suggested these methods as potential solutions for initialising nodes in the benchmarking tests to avoid a long initialization time. I'm not recommending them as best practice for building real-world apps.

Effective benchmarks reflect real-world usage. Removing the key generation unblocks us for discovering other issues, but the problem remains. I would like to avoid a case where we generate data that is not valuable because it does not reflect real-world usage.

[alanshaw]

Effective benchmarks reflect real-world usage

Of course! As I understood it, the data for the current benchmarks is only being recorded after a node was initialized, started and ready. Do you think that if we passed a 2048 bit pregenerated key it will be close enough to real-world usage when the benchmarks begin? I think yes, it would be good enough for the time being, and we can make the switch when the real issue is addressed.

This is totally optional - if the initialization time isn't getting in your way then we can just leave it as it is for now and the overall time it takes to run the benchmarks will just get super fast when the real issue is resolved.

I didn't mean to imply that the problem should not be fixed! We do want to address it, the Node.js solution you identified is being tracked here libp2p/js-libp2p-crypto#130. If you'd like to work on that issue then that would be very awesome.

As I suggested above it would be great to have a benchmark for initialization so we can see this perf improvement when it happens!

[mcollina]

I think all of what is described here on our side has been done.

---

Investigate potential issue running benchmarks with existing repo (@alanshaw)

is this ok @alanshaw?

Run locally and feedback (@alanshaw, @achingbrain, @hugomrdias)

Can you please test and verify?

[mcollina]

Closing as it all happened.