llama.cpp + Final Jeopardy

[ggerganov]

I was browsing reddit and saw this post:

https://www.reddit.com/r/LocalLLaMA/comments/12xkm9v/alpaca_vs_final_jeopardy/

If anyone is interested, it would be great to add such evaluation as an example to llama.cpp and add instructions for running it with different models: LLaMA, Alpaca, Vicuna, etc. and different quantizations.

Here is the original work by @aigoopy which can be a good starting point:

https://github.com/aigoopy/llm-jeopardy

[CRD716]

Going to attempt this, should it be in a separate folder (ex. examples/jeopardy) or should it be thrown directly in examples?

[ggerganov]

In examples/jeopardy, with a README with instructions, question files, eval scripts, plot scripts, etc. would be nice

[SlyEcho]

I extracted the data from the spreadsheet. I don't know if it is necessary to reverse the Jeopardy "answer in the form of a question". I think LLMs should be able to emulate it.

JSON

[
  {
    "category": "The Oscars",
    "answer": "Born in 1932 & the son of a percussionist in the CBS radio orchestra, he's been nominated for 53 Oscars",
    "question": "Who is John Williams?"
  },
  {
    "category": "English Literature",
    "answer": "It says, \"The mind is its own place, & in itself can make a heaven of hell, a hell of heaven. What matter where, if I be still the same\"",
    "question": "What is Paradise Lost?"
  },
  {
    "category": "Writers' Lesser-Known Works",
    "answer": "Known for more philosophical works, he wrote the play \"La Mandragola\", in which Florentines are rewarded for immoral actions",
    "question": "Who is Niccolò Machiavelli?"
  },
  {
    "category": "Exploration",
    "answer": "James Cook's account of a 1774 visit here records an object \"near 27 feet long, and upwards of 8 feet over the breast or shoulders\"",
    "question": "What is Easter Island (Rapa Nui)?"
  },
  {
    "category": "The Bill of Rights",
    "answer": "England's \"Bloody Assizes\" & a 1685 life sentence for perjury were 2 main origins of this amendment to the U.S. Constitution",
    "question": "What is the Eighth Amendment?"
  },
  {
    "category": "Nobel Peace Prize Winners",
    "answer": "At times they each lived on Vilakazi St. in Soweto, so it claims to be the world's only street home to 2 Nobel Peace Prize winners",
    "question": "Who are Nelson Mandela & Desmond Tutu?"
  },
  {
    "category": "Famous Names",
    "answer": "In 1966, the year of his death, he shared plans for an experimental prototype community in Florida",
    "question": "Who is Walt Disney?"
  },
  {
    "category": "Geography",
    "answer": "Of the 13 nations through which the Equator passes, it's the only one whose coastline borders the Caribbean Sea",
    "question": "What is Colombia?"
  },
  {
    "category": "Fashion History",
    "answer": "These decorative items get their name from their origin in the port city of Strasbourg, on the border of France & Germany",
    "question": "What are rhinestones?"
  },
  {
    "category": "Movies of the '80s",
    "answer": "Based on an off-Broadway play with just 3 characters, it won the Best Picture Oscar & the actors in all 3 roles were nominated",
    "question": "What is Driving Miss Daisy?"
  },
  {
    "category": "Novelists",
    "answer": "A 2012 book review noted subjects that \"sparked his ire\": capital punishment, big tobacco & \"the plight of the unjustly convicted\"",
    "question": "Who is John Grisham?"
  },
  {
    "category": "20th Century Eponyms",
    "answer": "A 1940 headline about this included \"failure\", \"liability when it came to offense\" & \"stout hearts no match for tanks\"",
    "question": "What is the Maginot Line?"
  },
  {
    "category": "City History",
    "answer": "Over 700 years after its traditional 1252 founding date, this port city became associated with a psychological response",
    "question": "What is Stockholm?"
  },
  {
    "category": "Brand Names",
    "answer": "The success of this brand has its roots with a hydrotherapy pump its cofounder created for his son, who had arthritis",
    "question": "What is Jacuzzi?"
  },
  {
    "category": "American Authors",
    "answer": "In a periodical in 1807, he called New York City \"Gotham, Gotham! Most enlightened of cities\"",
    "question": "Who is Washington Irving?"
  },
  {
    "category": "Symbols",
    "answer": "In math, it's a rotated V; in society, it's a feeling of some marginalized or underrepresented people",
    "question": "What is \"less than\"?"
  },
  {
    "category": "Movie Theme Songs",
    "answer": "Monty Norman, the composer of this character's theme, said the staccato riff conveyed sexiness, mystery & ruthlessness",
    "question": "Who is James Bond?"
  },
  {
    "category": "American Novelists",
    "answer": "He served with an airman named Yohannan in World War II & despite what readers might think, he said he enjoyed his service",
    "question": "Who is Joseph Heller?"
  },
  {
    "category": "Medieval Places",
    "answer": "One of the participants in an 1170 event at this place said, \"Let us away, knights; he will rise no more\"",
    "question": "What is Canterbury, England? (Canterbury Cathedral)"
  },
  {
    "category": "Countries of Africa",
    "answer": "At one time a province of the Roman Empire, this kingdom is known to Arabic scholars as Al-Maghrib Al-Aqsa, \"the far west\"",
    "question": "What is Morocco?"
  },
  {
    "category": "Statehood",
    "answer": "Congress relented in 1890 after this prospective state said it would wait 100 years rather than come in without the women",
    "question": "What is Wyoming?"
  },
  {
    "category": "1980s Movies",
    "answer": "A writer & producer of this movie said he wanted it to be like a Western or James Bond film, \"only it takes place in the '30s\"",
    "question": "What is Raiders of the Lost Ark?"
  },
  {
    "category": "Art Exhibitions",
    "answer": "In 1898 what's been called the first blockbuster art show was devoted to him & put on for Queen Wilhelmina's coronation",
    "question": "Who is Rembrandt?"
  },
  {
    "category": "Countries of the World",
    "answer": "Part of the largest contiguous land empire during the 1200s & 1300s, today it's the world's second-largest landlocked country",
    "question": "What is Mongolia?"
  },
  {
    "category": "Literature",
    "answer": "A 2006 book was titled \"The Poem That Changed America:\" this \"Fifty Years Later\"",
    "question": "What is \"Howl\"?"
  },
  {
    "category": "Invasions",
    "answer": "Backed by 14,000 troops, he invaded England to restore, in his words, its \"religion, laws, and liberties\"",
    "question": "Who is William of Orange?"
  },
  {
    "category": "Landmarks",
    "answer": "After its completion in the late 19th c., it was called \"a truly tragic street lamp\" & a \"high & skinny pyramid of iron ladders\"",
    "question": "What is the Eiffel Tower?"
  },
  {
    "category": "Geographic Name's the Same",
    "answer": "The busiest passenger port in the U.K., it shares its name with a capital of one of the original 13 states",
    "question": "What is Dover?"
  },
  {
    "category": "Names in the Bookstore",
    "answer": "This man made lists, perhaps to cope with depression; a set of lists he published in 1852 made his name synonymous with a type of book",
    "question": "Who is Peter Mark Roget?"
  },
  {
    "category": "U.S. History",
    "answer": "An 1869 presidential pardon was granted to this man, due in part to a plea by the Medical Society of Harford County, Maryland",
    "question": "Who is Dr. Samuel Mudd?"
  },
  {
    "category": "American Literature",
    "answer": "Letters, pocket knives, C rations & steel helmets are among the tangible items referred to in the title of this modern war classic",
    "question": "What is The Things They Carried?"
  },
  {
    "category": "Nonfiction",
    "answer": "It has the line, \"The discovery of America…opened up fresh ground for the rising bourgeoisie\"",
    "question": "What is The Communist Manifesto"
  },
  {
    "category": "Laws in U.S. History",
    "answer": "A radical Republican championed this 1875 act but the Supreme Court struck it down in 1883; a new version was passed 81 years later",
    "question": "What is the Civil Rights Act?"
  },
  {
    "category": "Names of Myth",
    "answer": "Her brothers, Castor & Pollux, saved her after Theseus stole her away as a kid; a larger force would seek her later in life",
    "question": "Who is Helen of Troy?"
  },
  {
    "category": "African Countries",
    "answer": "Once Africa's largest country in area, it dropped to third in 2011 when a portion of it declared independence",
    "question": "What is Sudan?"
  },
  {
    "category": "The Ancient World",
    "answer": "The ancient writer Galen said books on ships arriving to this city's port were seized, originals kept & copies returned",
    "question": "What is Alexandria?"
  },
  {
    "category": "Famous Names",
    "answer": "For a special 1970s cookbook, he provided one simple recipe -- a can of Campbell's tomato soup & 2 cans of milk",
    "question": "Who is Andy Warhol?"
  },
  {
    "category": "People & Places",
    "answer": "Thought to descend from people of Southeast Asia, the Chamorro make up this U.S. territory's largest ethnic group",
    "question": "What is Guam?"
  },
  {
    "category": "Current World Leaders",
    "answer": "In office from 2022, the president of this country has taken so many foreign trips a play on his name is \"Ferdinand Magellan Jr.\"",
    "question": "What is the Philippines?"
  },
  {
    "category": "Writers & The South",
    "answer": "In 1939 he lived on Toulouse Street in the French Quarter & chose the professional name that bonded him to the South",
    "question": "Who is Tennessee Williams?"
  },
  {
    "category": "National Parks",
    "answer": "It's named for a river indigenous people called Mi tse a-da-zi, translated by French-speaking trappers as \"Pierre Jaune\"",
    "question": "What is Yellowstone?"
  },
  {
    "category": "Sports",
    "answer": "In 2010 they introduced the 4-point shot, 35 feet from the basket",
    "question": "Who are the Harlem Globetrotters?"
  },
  {
    "category": "The U.S. Military",
    "answer": "Losses over Asia in the 1960s led to the establishment of the program known as this at a San Diego naval base in 1969",
    "question": "What is \"Top Gun\"?"
  },
  {
    "category": "Art & Science",
    "answer": "A craft that visited it was named for Giotto, based on the story that 680 years earlier, the painter depicted it as the Star of Bethlehem",
    "question": "What is Halley's Comet?"
  },
  {
    "category": "Words From World War I",
    "answer": "\"Cistern\" & \"reservoir\" were suggested names for a secret invention, but the British preferred this less clumsy monosyllable",
    "question": "What is \"tank\"?"
  },
  {
    "category": "European History",
    "answer": "Until 1806, some German nobles included among their honors the title of \"Elector\" for their role in selecting this personage",
    "question": "What is Holy Roman Emperor?"
  },
  {
    "category": "Theater History",
    "answer": "In 1904, wearing a harness, actress Nina Boucicault became the first to play this character onstage",
    "question": "Who is Peter Pan?"
  },
  {
    "category": "European Cities",
    "answer": "Alphabetically the first German city in encyclopedias, it was also the first one taken by the Allies in World War II",
    "question": "What is Aachen?"
  },
  {
    "category": "Word Origins",
    "answer": "This Sanskrit word referring to a spoken word or phrase comes from a word for \"to think\"",
    "question": "What is mantra?"
  },
  {
    "category": "Inventions",
    "answer": "1917's \"Elements of Trench Warfare\" said this Old West item was \"difficult to destroy\" & \"difficult to get through\"",
    "question": "What is barbed wire?"
  },
  {
    "category": "World War II",
    "answer": "Mimi Reinhard, who never learned to type using more than 2 fingers, produced this with 1,100 names, including hers",
    "question": "What is Schindler's list?"
  },
  {
    "category": "Mythology",
    "answer": "Poseidon carried off the maiden Theophane & turned her into a ewe; their offspring was the source of this mythical object",
    "question": "What is the Golden Fleece?"
  },
  {
    "category": "Literature",
    "answer": "Published in 2011, P.D. James' final novel, \"Death Comes to Pemberley\", was a sequel to this novel from 200 years earlier",
    "question": "What is Pride and Prejudice?"
  },
  {
    "category": "U.S. State Names",
    "answer": "5 U.S. states have 6-letter names; only these 2 west of the Mississippi River border each other",
    "question": "What are Oregon & Nevada?"
  },
  {
    "category": "Word Origins",
    "answer": "Originally relating to a story of suffering, this word now more commonly refers to strong emotion of any kind",
    "question": "What is passion?"
  },
  {
    "category": "World Cinema",
    "answer": "The 2007 biopic called \"La Môme\" in France, meaning \"The Kid\", was released in the U.S. under this other French title",
    "question": "What is La Vie en Rose?"
  },
  {
    "category": "History",
    "answer": "Returning home in 1493, Columbus stopped in the Azores at an island with this name, also something he'd lost off the Haiti coast",
    "question": "What is Santa Maria?"
  },
  {
    "category": "Landmarks",
    "answer": "Pskov & Nizhny Novgorod are 2 of the cities that have a fortress called this",
    "question": "What is a kremlin?"
  },
  {
    "category": "Foreign-Born Authors",
    "answer": "In the 1950s the New York Times said this author \"is writing about all lust\" & his lecherous narrator \"is all of us\"",
    "question": "Who is Vladimir Nabokov?"
  },
  {
    "category": "Astronomy & Geography",
    "answer": "At the winter solstice, the sun is in Sagittarius; it once appeared in this constellation, giving a geographic feature its name",
    "question": "What is Capricorn?"
  },
  {
    "category": "Television",
    "answer": "Mike Post combined the sound of a slamming jail door, an anvil & 100 men stomping on a floor for this series that debuted in 1990",
    "question": "What is Law & Order?"
  },
  {
    "category": "British Landmarks",
    "answer": "Like Sir Thomas More, 3 16th century English queens are buried at this location",
    "question": "What is the Tower of London?"
  },
  {
    "category": "Early American History",
    "answer": "In 1692 Increase Mather wrote, \"It were better that ten suspected\" these \"escape, than that one innocent person … be condemned\"",
    "question": "What are witches?"
  },
  {
    "category": "Geography Mnemonics",
    "answer": "Mimal, sometimes said to be the silhouette of a chef or elf, stands for Minnesota, Iowa, Missouri, and these 2 states",
    "question": "What are Arkansas and Louisiana?"
  },
  {
    "category": "Business Milestones",
    "answer": "These were first sold in 1908, at a price equivalent to about $27,000 today",
    "question": "What is the Ford Model T?"
  },
  {
    "category": "In The Bookstore",
    "answer": "The name of this author dead since 2013 now appears on books written by a former U.S. marshal & a former Apache helicopter pilot",
    "question": "Who is Tom Clancy?"
  },
  {
    "category": "Historic Art",
    "answer": "The artwork once known in France as \"la tapisserie de la Reine Mathilde\" is better known as this",
    "question": "What is the Bayeux Tapestry?"
  },
  {
    "category": "Pop Stars",
    "answer": "In 2022 she became the first woman to have a Billboard Top 10 album in 5 decades starting with the 1980s",
    "question": "Who is Madonna?"
  },
  {
    "category": "Classic Tale Characters",
    "answer": "In one 19th century translation, she \"perceived the dawn of day and ceased\" speaking nearly 1,000 times",
    "question": "Who is Scheherazade?"
  },
  {
    "category": "USA",
    "answer": "Ironically, though this company founded in the 1860s is Moore County, Tennessee's largest employer, Moore is a dry county",
    "question": "What is Jack Daniel's?"
  },
  {
    "category": "Historic People",
    "answer": "After a 1789 event, he wrote, \"My first determination was to seek a supply of…water at Tofoa, & afterwards to sail for Tongataboo\"",
    "question": "Who was William Bligh?"
  },
  {
    "category": "The Movies",
    "answer": "Laurence Olivier & Ernest Borgnine were considered for the lead role & Sergio Leone to direct for this film that turned 50 in 2022",
    "question": "What is The Godfather?"
  },
  {
    "category": "Continental Geography",
    "answer": "Until a 1903 secession, this country's contiguous territory spanned 2 continents",
    "question": "What is Colombia?"
  },
  {
    "category": "Foreign-Born Authors",
    "answer": "Early in her career she translated romance novels into Spanish, often changing the dialogue to make the heroines smarter",
    "question": "Who is Isabel Allende?"
  },
  {
    "category": "Historic Crimes",
    "answer": "Saying it was stolen by Napoleon, self-styled Italian patriot Vincenzo Peruggia took it in 1911",
    "question": "What is the Mona Lisa?"
  },
  {
    "category": "U.S. Bodies of Water",
    "answer": "Continuing a downward trend, in July 2022 it was at 27% capacity, its lowest level since 1937 when it was first being filled",
    "question": "What is Lake Mead?"
  },
  {
    "category": "Gods & Goddesses",
    "answer": "Each morning she began her ride in her chariot across the sky ahead of her brother Sol, or Helios",
    "question": "Who is Aurora (or Eos)?"
  },
  {
    "category": "America At War",
    "answer": "Until the Civil War, the Jan. 8 date of this battle of dubious military importance but big morale value was a national holiday",
    "question": "What is the Battle of New Orleans?"
  },
  {
    "category": "Children's Books",
    "answer": "Its title character is told \"By the time you are real, most of your hair has been loved off…your eyes drop out & you get…shabby\"",
    "question": "What is The Velveteen Rabbit?"
  },
  {
    "category": "TV Finales",
    "answer": "In a reunion over 40 years in the making, Dolly Parton appeared as an angel named Agnes in the final episode of this comedy in 2022",
    "question": "What is Grace and Frankie?"
  },
  {
    "category": "American Poems",
    "answer": "In an 1847 poem this character sees her town of Grand-Pré burned, but finally reunites with her beau for a kiss before his death",
    "question": "Who is Evangeline?"
  },
  {
    "category": "Famous Names",
    "answer": "In 2001 he published a book called \"Banging Your Head Against a Brick Wall\"; in 2002, \"Existencilism\"",
    "question": "Who is Banksy?"
  },
  {
    "category": "Children's Lit",
    "answer": "The title object of this book \"never looked more beautiful…each strand held dozens of bright drops of early morning dew\"",
    "question": "What is Charlotte's Web?"
  },
  {
    "category": "Classic Songs",
    "answer": "The shouts of excited children at a 1946 holiday parade are said to have inspired this perennial favorite",
    "question": "What is \"Here Comes Santa Claus\"?"
  },
  {
    "category": "Brand Names",
    "answer": "Unable to make these candies perfectly round, the confectioner embraced this flawed name for the product",
    "question": "What are Milk Duds?"
  },
  {
    "category": "Countries of the World",
    "answer": "It's home to 58 UNESCO World Heritage Sites, more than any other country; the sites include a volcano & a lagoon",
    "question": "What is Italy?"
  },
  {
    "category": "Action Movies",
    "answer": "Its last line is \"If this is their idea of Christmas, I gotta be here for New Year's\"",
    "question": "What is Die Hard?"
  },
  {
    "category": "Presidential Facts",
    "answer": "Only 3 presidents have married while in office— John Tyler was the first & he was the last",
    "question": "Who is Woodrow Wilson?"
  },
  {
    "category": "19th Century Americans",
    "answer": "Demonstrating the dignity & humanity of Black Americans, he sat for 160 known photographs, the most of any American in the 19th century",
    "question": "Who is Frederick Douglass?"
  },
  {
    "category": "Latin Phrases",
    "answer": "Originally, this 3-word phrase referred to when a doctor or apothecary substituted one medicine for another",
    "question": "What is \"quid pro quo\"?"
  },
  {
    "category": "1970s Movies",
    "answer": "A 1975 premiere of this comedy advertised free coconuts for the first thousand in the audience",
    "question": "What is Monty Python and the Holy Grail?"
  },
  {
    "category": "Name's The Same",
    "answer": "A cocktail, an island & a WWII venture originally called \"Development of Substitute Materials\" all bear this name",
    "question": "What is Manhattan?"
  },
  {
    "category": "U.S. Presidents",
    "answer": "He was sworn in twice as President within 2 years, first by his father & then later by a former U.S. President",
    "question": "Who is Calvin Coolidge?"
  },
  {
    "category": "Plays",
    "answer": "A 1609 story in which an exiled king of Bulgaria creates a sea palace with his magic may have inspired the plot of this play",
    "question": "What is The Tempest?"
  },
  {
    "category": "Landmarks",
    "answer": "In 2009, during a 20th anniversary celebration, it was called \"an edifice of fear. On Nov. 9, it became a place of joy\"",
    "question": "What is the Berlin Wall?"
  },
  {
    "category": "World Capitals",
    "answer": "Among its nicknames are the \"City of Classical Music\" &, possibly in honor of a famous resident from 1860 to 1938, the \"City of Dreams\"",
    "question": "What is Vienna, Austria?"
  },
  {
    "category": "Language & Its Meanings",
    "answer": "Now meaning someone with nocturnal habits, it catches a sleeping dove in Shakespeare's \"…Lucrece\"",
    "question": "What is a night owl?"
  },
  {
    "category": "Flags of Our Hemisphere",
    "answer": "The stars on this country's flag represent states, 26 of them; unlike the USA's, its \"federal district\" gets its own 27th star",
    "question": "What is Brazil?"
  },
  {
    "category": "Names in U.S. History",
    "answer": "This father was the only man among the 13 plaintiffs in a class-action case filed in 1951",
    "question": "Who is Oliver Brown?"
  },
  {
    "category": "Children's Authors",
    "answer": "Reversing the story of this heroine she created, Patricia Maclachlan was born on the prairie but spent much of her life in New England",
    "question": "Who is Sarah? (from Sarah, Plain and Tall)"
  }
]

[SlyEcho]

I got a notebook here where I use Langchain to create prompts to evaluate the quiz data.

It looks like I was right in that the models can just play Jeopardy without needing to change the data.

https://gist.github.com/SlyEcho/a1e6ac9e44eb48a6769b44b61050b635

[CRD716]

#1168 is in a usable state now.