Publish results somewhere

[hauleth]

It would be nice to have place to send people from Reddit asking about performance of crypto in Rust.

[briansmith]

Not many applications are bottlenecked on digest functions, and @ctz already did publish some PBKDF2 benchmarks, so I'm not sure we're currently generating useful output yet.

My current plan is to do the following things beforehand:

1. Get the -msvc build working.
2. Add the ECDH benchmark.
3. Add the AEAD benchmarks for ChaCha20-Poly1305, AES-128-GCM, and AES-256-GCM.
4. Add the RSA and ECDSA signature verification benchmarks.
5. Create a tool that converts the cargo bench output into a form that can easily be transformed into SVG bar graphs.

I'm starting at #1 on the list and working down. If people are interested in helping, starting at #5 and working up would be a good strategy.

[hauleth]

About 5. I need to finish rust-lang/rfcs#1284.

About rest of points, I need to implement that in Octavo quickly 😟

[JohnHeitmann]

I've got some rough draft scaffolding up:

1. Clone https://github.com/JohnHeitmann/crypto-bench/tree/bench_viewer
2. cd crypto-bench/bench_viewer
3. cargo run -- --demo
4. Load http://127.0.0.1:3000/ in your browser

rust-crypto's huge results skew the scale. Click on rust-crypto in the legend to turn off its bars and that will work around the scale issues.

To look at real benchmark data:

1. ./cargo_all bench > bench.log // Must use cargo_all from my branch; Won't work on Windows
2. cargo run -- --file bench.log

From front to back the architecture is:

• ChartJS bar chart generated in canvas
• Mostly static site served from Iron
• Data dynamically served, translated live from your bench log
• Translation from bench.log into rust structs with dumb rust code. Translation from structs into json with serde.

The dynamic json looks like a plain old file to the site, so it's easy to swap out for a hardcoded static file if we want to graduate this from a local developer tool to an purely static external site.

I've put very little effort into presentation-- or any part of this for that matter-- so everything is up for grabs if you want to ask to go a different direction. For example, you originally mentioned a transform from bench logs to SVG. I can ditch the dynamic website and go that route if it's ultimately more useful.

If this looks like a good rough direction, then the next thing I'll do (after dusting off some libsodium work) is expand the data model to handle runs across time, and runs across machines/architectures.

[JohnHeitmann]

Here's my plan:

First, I'm going to ditch the small dynamic portion of the prototype site. I'll convert my log -> json parser to run offline.

The web site will have an exposed data/ directory. Inside the data/ directory are directories for each machine we run benchmarks on with human-chosen names like data/heitmann-laptop/. Inside each machine directory are individual json files like my original prototype that track one multi-library benchmark run. These files are named by date, e.g. data/heitmann-laptop/20160610.json.

To aid web queries, each directory will contain a manifest.json. data/manifest.json will list each machine name, and contain optional manually-entered metadata about the machine such as architecture, clock speed, etc. data/<machine>/manifest.json will just include a list of each benchmark file in the directory, and will be auto-generated.

On the visualization side I'll support 2 comparison types: compare all libraries on a specific machine+date, or compare two different machine+date runs for a specific library (which covers the same machine over time, or the most recent results for two different machines).

It should be super low-tech and low-maintenance.

[briansmith]

I like the last version of your plan a lot!

I have a question:

Translation from bench.log into rust structs with dumb rust code. Translation from structs into json with serde.

Why not just use regexes in JS to parse the bench.log? Then, in theory, we one could drag/drop a file containing a bench log onto the web page and the web page could parse it and graph it automatically.

[JohnHeitmann]

I like the drag 'n drop idea. But, I think there also need for offline utilities to create the manifest files to support the comparison queries. I went with rust for that because I'm still new to rust and wanted the exposure, and because I didn't want to force folks to use npm when they already were using cargo.

Since the rust code is already in the bag I'll keep it for the next couple of iterations, then take a second look at making the quick-view use case super easy.

[briansmith]

OK. That sounds great.

[briansmith]

Here are some old results that I published long ago, which show one way of presenting head-to-head numbers in a tabular form. This was done by hand and it was a PITA, but it might inspire some formatting choices: https://docs.google.com/spreadsheets/d/1VP-lL2VQsoY3R4WKKayEknpTARP6OHN-HE73MZOBYzY/edit#gid=0

[JohnHeitmann]

Here's where I'm at: https://johnheitmann.github.io/crypto-bench-demo1/

I just finished a functionality pass, and I think the basics are all there. I spent a bit of time making it generic, so it's 90% of the way to being reusable for other projects.

Next up is some presentation polish.

I'm somewhat happy with the charting library. I don't like that you can't search for benchmark names since it's canvas, and I don't like that I can't get all graphs to have the same bar size, so I might take a quick look at what other chart libraries can do.

[JohnHeitmann]

The presentation is now somewhat polished and viewable at the same demo link. Next up:

1. Drag 'n drop
2. Converting the remaining rusty bits to javascript commands
3. Adding deep linking support (so you can link to a specific comparison)

[briansmith]

Lots of awesome work above! Unfortunately, I just don't have time to deal with this for the foreseeable future, so I'm closing this issue.