Create walkthrough example with data

[marko-polo-cheno]

This PR provides a quick guide with all needed data to get a useful leaderboard on AutoArena. The notebook explains the core principles of why AutoArena uses an Elo leaderboard, and shows how it works.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.92%. Comparing base (a3b7745) to head (13f43fe).
Report is 1 commits behind head on trunk.

Additional details and impacted files

@@           Coverage Diff           @@
##            trunk     #102   +/-   ##
=======================================
  Coverage   96.92%   96.92%           
=======================================
  Files          34       34           
  Lines        1465     1465           
=======================================
  Hits         1420     1420           
  Misses         45       45           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gordonhart]

Very minor, but considering that these are checked into the source tree, smaller is better — would you mind transcoding to JPEG?

[gordonhart]

Based on the number of votes versus the number of responses, it looks like there are many unmatched prompts between these models. Can we pare down the CSVs to only include matched prompts?

[marko-polo-cheno]

I can pare down the CSVs, but unfortunately there are very few prompts shared amongst all models.

[gordonhart]

When I run this myself, I get llama-2-70b ranked above gpt-4-0613 — have you seen this? It's an unexpected result, that might be due to the particular prompts+responses included in the CSVs.

Edit: looks like the culprit is a heavily skewed H2H distribution. If gpt-4-0613 has the lion's share of its H2Hs against the champion, it'll look worse than it really is on the main leaderboard:

[marko-polo-cheno]

When I run this myself, I get llama-2-70b ranked above gpt-4-0613 — have you seen this? It's an unexpected result, that might be due to the particular prompts+responses included in the CSVs.

This is unexpected. Opponents shouldn't be allowed to "farm" Elo off of each other (gpt-4-1106-preview is untouchable).