add_TriviaQA_Benchmark

agential/cog/prompts/react.py

@@ -118,3 +118,60 @@
 Observation 3: (Result 1 / 3) The song peaked at number two on the Billboard Hot 100 in the United States, where it was certified Gold for 500,000 units shipped.
 Thought 4: It only says the song peaked at number two on the Billboard Hot 100, but not if it was in 2003. I am not sure if this claim is true or not.
 Action 4: Finish[NOT ENOUGH INFO]"""
+
+# ======================================================================== TRIVIAQA ======================================================================== #
+
+REACT_INSTRUCTION_TRIVIAQA = """Answer a trivia question with interleaving Thought, Action, Observation steps. Thought can reason about the current question and plan the retrieval steps, and Action can be three types:
+(1) Search[entity], which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
+(2) Lookup[keyword], which returns the next sentence containing keyword in the last passage successfully found by Search.
+(3) Finish[answer], which returns the answer and finishes the task.
+You have a maximum of {max_steps} steps.
+
+Here are some examples:
+{examples}
+(END OF EXAMPLES)
+
+Question: {question}{scratchpad}"""
+
+TRIVIAQA_FEWSHOT_EXAMPLES = """Question: Which innovation for the car was developed by Prince Henry of Prussia in 1911? 
+Thought 1: Let's search the question in google
+Action 1: Search[Which innovation for the car was developed by Prince Henry of Prussia in 1911? site: wikipedia.org]
+Observation 1: [Prince Henry of Prussia (1862–1929) - Wikipedia] Henry was interested in motor cars as well and supposedly invented a windshield wiper and, according to other sources, the car horn.
+Thought 2: The evidence suggests that Prince Henry of Prussia invented a windshield wiper and the car horn. 
+Action 2: Search[When did Prince Henry of Prussia invented a windshield wiper and the car horn?]
+Observation 2: [110 years ago: windscreen wiper patent for Prince Henry of Prussia] Quite apart from a member of the German aristocracy: it was Prince Henry of Prussia (1862-1929) who was granted the first German patent about the windscreen wiper on 24. March 1908. 
+Thought 3: According to the evidence, Prince Henry of Prussia was granted the first German patent for the windshield wiper.
+Action 3: Finish[windshield wiper]
+
+Question: When did the founder of Jehovah's Witnesses say the world would end?
+Thought 1: Let's search the question in google
+Action 1: Search[When did the founder of Jehovah's Witnesses say the world would end? site: wikipedia.org]
+Observation 1: [Eschatology of Jehovah's Witnesses - Wikipedia] In 1911, Russell wrote that October 1914 would witness the "full end" of Babylon, or nominal Christianity, "utterly destroyed as a system". At first, the hopes ... 
+Thought 2: The evidence shows that the founder of Jehovah's Witnesses predicted the world would end in October 1914
+Action 2: Finish[1914]
+
+Question: Three people have featured on the reverse of the £5 note, the two most recent are George Stephenson and Elizabeth Fry, who was the first?
+Thought 1: Let's search the question in google
+Action 1: Search[Three people have featured on the reverse of the £5 note, the two most recent are George Stephenson and Elizabeth Fry, who was the first?]
+Observation 1: [Bank of England £5 note - Wikipedia] The Bank of England £5 note, also known as a fiver, is a sterling banknote . It is the smallest denomination of banknote currently issued by the Bank of England.
+Thought 2: The evidence is about £5 note, it has no information about the featured people we want to find out. 
+Action 2: Search[Who was the first person to feature on the reverse of the £5 note? site: wikipedia.org] 
+Observation 2: [Bank of England £5 note - Wikipedia] The old paper note, first issued in 2002 and bearing the image of prison reformer Elizabeth Fry on the reverse, was phased out and ceased to be legal tender after 5 May 2017. 
+Thought 3: The evidence only mentions Elizabeth Fry, and from the question, we know that she was not the first person to feature on the reverse of the £5 note. 
+Action 3: Search[People have featured on the reverse of the £5 note: George Stephenson and Elizabeth Fry and?]
+Observation 3: [Historical British figures featured on banknotes (1970-2017)] £5 – The Duke of Wellington, George Stephenson, Elizabeth Fry, Sir Winston Churchill £10 – Florence Nightingale, Charles Dickens, Charles Darwin £20 – William Shakespeare, ...
+Thought 4: The evidence shows that the Duke of Wellington and Sir Winston Churchill were also featured on £5.
+Action 4: Finish[Duke of Wellington]
+
+Question: What state had its bi-centenary of joining the Union a year after North Carolina? 
+Thought 1: Let's search the question in google 
+Action 1: Search[What state had its bi-centenary of joining the Union a year after North Carolina? site: wikipedia.org] 
+Observation 1: [List of U.S. states by date of admission to the Union - Wikipedia] 24. Missouri, August 10, 1821 · (admitted) ; 25. Arkansas, June 15, 1836 · (admitted) ... 
+Thought 2: The evidence does not provide information about the state had its bi-centenary of joining the Union a year after North Carolina. To answer the question, we need to find the state joining the Union a year after North Carolina. 
+Action 2: Search[Which state joined the Union a year after North Carolina? site: wikipedia.org] 
+Observation 2: [List of U.S. states by date of admission to the Union - Wikipedia...] ^ This list does not account for the secession of 11 states (Virginia, North Carolina, South Carolina, Georgia, Florida, Alabama, Mississippi, Tennessee, ... 
+Thought 3: Not enough evidence, we need further search. 
+Action 3: Search[What state joined the Union a year after North Carolina?] 
+Observation 3: [States by Order of Entry into Union - Infoplease] Joining the Union | State | Entered Union | Year Settled | | North Carolina | Nov. 21, 1789 | 1660 | | Rhode Island | May 29, 1790 | 1636 | | Vermont | Mar. 4, 1791 | 1724 | 
+Thought 4: The evidence shows North Carolina entered Union in 1789, and Rhode Island entered Union in 1790, which is a year after North Carolina. 
+Action 4: Finish[Rhode Island]"""