Ordering Randomised VersionList

[jjyuhub]

Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access be granted. 🚨

PLEASE READ THIS:

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject it since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. Starting April 10, the minimum eval count is 15 samples, we hope this makes it easier to create and contribute evals.

Also, please note that we're using Git LFS for storing the JSON files, so please make sure that you move the JSON file to Git LFS before submitting a PR. Details on how to use Git LFS are available here.

Eval details 📑

Eval name

Ordering Randomised VersionList

Eval description

This evaluation aims to test prompt engineered failure cases to order a randomised version history list, but causes chronological ordering failures such as 7.5.2 -> 7.4.2 -> 7.5.1 -> 7.4.1 (incorrectly inserted 7.4.2 in between 7.5.2 and 7.5.1 and incorrectly skipping over the major release version 7.5.0 in the Explainable AI chain of thoughts) and 7.5.2 -> 7.5.1 -> 7.5.0 -> 7.4.1 (incorrectly skipped over 7.4.2 in the Explainable AI chain of thoughts).

What makes this a useful eval?

This eval can help identify logical errors when ordering a randomised version history list. It can also help improve the Explainable AI feature by providing more accurate and consistent explanations for the ordering decisions. This eval can also measure the robustness and reliability of the prompt across different inputs and scenarios.

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

• Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
• Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
• Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
• Include at least 15 high-quality examples.

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

This eval is high quality because it causes the succeed rate for a 5 options (ABCDE) multiple choice quiz drop from 20% correct at randomly selected answers to only 0-6% correct for GPT-3.5-Turbo. These are prompt engineered failures, causing bigger failure rates than prior work, as performing so much worse than random is unnatural for such a super easy task.

Eval structure 🏗️

Your eval should

• Check that your data is in evals/registry/data/{name}
• Check that your YAML is registered at evals/registry/evals/{name}.yaml
• Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

• I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the commits on the merged pull request.

• I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgment

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and the high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

• I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access be granted.

Submit eval

• I have filled out all required fields of this form
• I have used Git LFS for the Eval JSON data
• (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input":[{"role":"user","content":"Here's a list of software versions: Version 7.5.0 Release Date: December 02, 2019 Version 7.4.1 Release Date: October 23, 2019 Version 7.5.1 Release Date: December 18, 2019 Version 7.5.2 Release Date: January 21, 2020 Version 7.4.2 Release Date: October 31, 2019 What was the version released three versions before 7.5.2? A. 7.4.2 B. 7.5.2 C. 7.5.1 D. 7.4.1 E. 7.5.0"}],"ideal":"A. 7.4.2"}
{"input":[{"role":"user","content":"Here's a list of software versions: Version 7.5.2 Release Date: January 21, 2020 Version 7.4.1 Release Date: October 23, 2019 Version 7.5.0 Release Date: December 02, 2019 Version 7.4.2 Release Date: October 31, 2019 Version 7.5.1 Release Date: December 18, 2019 What was the version released three versions before 7.5.2? A. 7.5.2 B. 7.5.1 C. 7.4.1 D. 7.4.2 E. 7.5.0"}],"ideal":"D. 7.4.2"}
{"input":[{"role":"user","content":"Here's a list of software versions: Version 7.5.1 Release Date: December 18, 2019 Version 7.5.0 Release Date: December 02, 2019 Version 7.4.1 Release Date: October 23, 2019 Version 7.5.2 Release Date: January 21, 2020 Version 7.4.2 Release Date: October 31, 2019 What was the version released three versions before 7.5.2? A. 7.5.0 B. 7.4.2 C. 7.5.1 D. 7.4.1 E. 7.5.2"}],"ideal":"B. 7.4.2"}
{"input":[{"role":"user","content":"Here's a list of software versions: Version 7.5.0 Release Date: December 02, 2019 Version 7.5.1 Release Date: December 18, 2019 Version 7.4.2 Release Date: October 31, 2019 Version 7.4.1 Release Date: October 23, 2019 Version 7.5.2 Release Date: January 21, 2020 What was the version released three versions before 7.5.2? A. 7.5.1 B. 7.4.1 C. 7.5.2 D. 7.5.0 E. 7.4.2"}],"ideal":"E. 7.4.2"}
{"input":[{"role":"user","content":"Here's a list of software versions: Version 7.4.2 Release Date: October 31, 2019 Version 7.5.1 Release Date: December 18, 2019 Version 7.5.0 Release Date: December 02, 2019 Version 7.5.2 Release Date: January 21, 2020 Version 7.4.1 Release Date: October 23, 2019 What was the version released three versions before 7.5.2? A. 7.4.1 B. 7.5.2 C. 7.4.2 D. 7.5.0 E. 7.5.1"}],"ideal":"C. 7.4.2"}

• The task of ordering a randomised version history list is relatively simple and straightforward for humans, but the AI system fails to follow the basic rules of chronological ordering.
• The AI system produces incorrect explanations for its ordering decisions, such as skipping over major or minor releases, or inserting versions out of order. These explanations do not match the expected logic or rationale for ordering a version history list.
• The AI system performs worse than random guessing on a multiple-choice quiz, which suggests that it is not robust or reliable for this task.

[usama-openai]

Thanks for the contribution. I would like to request the following changes:

1. There are too many repetitions in this dataset. For example, the following two samples are repeated three times:

   {"input":[{"role":"user","content":"Here's a list of software versions: Version 7.5.0 Release Date: December 02, 2019 Version 7.4.1 Release Date: October 23, 2019 Version 7.5.1 Release Date: December 18, 2019 Version 7.5.2 Release Date: January 21, 2020 Version 7.4.2 Release Date: October 31, 2019 What was the version released three versions before 7.5.2? A. 7.4.2 B. 7.5.2 C. 7.5.1 D. 7.4.1 E. 7.5.0"}],"ideal":"A. 7.4.2"}

   {"input":[{"role":"user","content":"Here's a list of software versions: Version 7.5.0 Release Date: December 02, 2019 Version 7.5.1 Release Date: December 18, 2019 Version 7.4.2 Release Date: October 31, 2019 Version 7.4.1 Release Date: October 23, 2019 Version 7.5.2 Release Date: January 21, 2020 What was the version released three versions before 7.5.2? A. 7.5.1 B. 7.4.1 C. 7.5.2 D. 7.5.0 E. 7.4.2"}],"ideal":"E. 7.4.2"}

   Similarly, there are a few samples that are repeated twice in the dataset. Kindly make sure that all the samples are unique. You can do this by diversifying the version numbers.

2. I would recommend asking the model to reason before answering because logical operations and other complex tasks are hard for the model to do zero-shot without a chance to reason through the steps. Asking the model to provide reasoning will give the model a fair chance to solve the question. You can also add instructions to provide output in a specific format and then use that format to write the ideal answer. For example, the model can be asked to enclose the final answer in square brackets, and the ideal answer can be formatted like [A. 1.1.1]. It'll give the model a proper chance to reason before answering, and the proper formatting will help in identifying the final answer using the Includes method.

   You can add instructions to the prompt like this:

   Please reason step by step before answering, and provide your final answer at the end, enclosed in square brackets. For example: [A. 1.1.1].

We would love to review the PR again after the suggested changes.

[jjyuhub]

I'll work on it. It takes some time to get 15 unique permutations that fail with 90-100% certainty when it must reason step by step as shown below.

GPT-3 step-by-step reasoning failure skipping over the correct answer:

Here's a list of software versions: Version 7.5.0 Release Date: December 02, 2019 Version 7.4.1 Release Date: October 23, 2019 Version 7.5.1 Release Date: December 18, 2019 Version 7.5.2 Release Date: January 21, 2020 Version 7.4.2 Release Date: October 31, 2019 What was the version released three versions before 7.5.2? A. 7.4.2 B. 7.5.2 C. 7.5.1 D. 7.4.1 E. 7.5.0 Please reason step by step before answering, and provide your final answer at the end, enclosed in square brackets. For example: [A. 1.1.1].


Answer:  [D. 7.4.1] 

Reasoning: Version 7.5.2 was released on January 21, 2020. In order to find the version released three versions before it, we need to look back at earlier versions sequentially. We can start with Version 7.5.0 released on December 2, 2019, followed by Version 7.4.1 released on October 23, 2019. This is the version released three versions before 7.5.2. Therefore, the answer is [D. 7.4.1].

GPT-4 step-by-step reasoning failure skipping over the correct answer:

Here's a list of software versions: Version 7.5.0 Release Date: December 02, 2019 Version 7.4.1 Release Date: October 23, 2019 Version 7.5.1 Release Date: December 18, 2019 Version 7.5.2 Release Date: January 21, 2020 Version 7.4.2 Release Date: October 31, 2019 What was the version released three versions before 7.5.2? A. 7.4.2 B. 7.5.2 C. 7.5.1 D. 7.4.1 E. 7.5.0 Please reason step by step before answering, and provide your final answer at the end, enclosed in square brackets. For example: [A. 1.1.1].

To answer this question, I need to find the version that was released three versions before 7.5.2. I can do this by looking at the list of software versions and their release dates, and counting backwards from 7.5.2.

The version that was released immediately before 7.5.2 is 7.5.1, which has a release date of December 18, 2019.
The version that was released two versions before 7.5.2 is 7.5.0, which has a release date of December 02, 2019.
The version that was released three versions before 7.5.2 is 7.4.1, which has a release date of October 23, 2019.
Therefore, the correct answer is [D. 7.4.1].

[jjyuhub]

@rlbayes @jwang47 @logankilpatrick @andrew-openai The evals have been updated to reflect chronological reasoning failures such as "The latest version as of 27 February 2020 was Version 3.3.0, since it was released on February 29, 2020 and is the most recent version listed prior to that date." <-- The most recent version came from 2 days into the future?

Plus some other weird Chains of Thoughts:

To answer this question, we need to find the version that was released three versions before 2.5.2. We can do this by looking at the list of software versions and their release dates, and finding the one that comes before 2.5.2 in chronological order.

We can start by eliminating the versions that were released after 2.5.2, since they cannot be three versions before it. According to the list, 2.5.2 was released on January 21, 2020, so any version that was released after that date is not a possible answer. This means we can eliminate option C. 2.5.1, which was released on December 18, 2019.

[NOTE: December 18, 2019 does not come after January 21, 2020!]

Next, we can eliminate the versions that are not in the same major or minor version as 2.5.2, since they are not considered to be in the same sequence of versions. A major version is the first number in the version name, such as 2 in 2.5.2, and a minor version is the second number, such as 5 in 2.5.2. A patch version is the third number, such as 2 in 2.5.2, and it indicates a small update or bug fix within the same minor version. According to the list, there are two major versions: 2 and 3, and two minor versions within 2: 4 and 5. Since we are looking for a version that is three versions before 2.5.2, we need to find a version that has the same major and minor version as 2.5.2, which is 2 and 5 respectively. This means we can eliminate option B. 2.4.1 and option D. 2.4.2, which have a different minor version than 2.5.2.

[NOTE: Incorrect generalisation! Version 2.4.X can precede Version 2.5.X]

Finally, we can compare the remaining options based on their patch versions and their release dates, and find the one that is three versions before 2.5.2 in chronological order. According to the list, there are three patch versions within 2.5: 0, 1 and 2. Since we are looking for a version that is three versions before 2.5.2, we need to find a version that has a patch version that is three less than 2 in 2.5.2, which is -1 in this case. However, there is no such version in the list, so we need to look at the release dates instead and find the earliest one among the remaining options.

[NOTE: Incorrect reasoning! Version Numbers don't go to -1]

The only remaining option is A. 2.5.0, which was released on December 02, 2019, which is earlier than any other option in the list.

Therefore, the answer is [A. 2.5.0], which was the version released three versions before 2.5.2.

[NOTE: Incorrect final answer! Chain Of Thoughts FAIL]

[usama-openai]

This PR looks in good shape now. I'm approving this PR.

[usama-openai]

You should see GPT-4 API access enabled in your account in the next few days.

[jjyuhub]

You should see GPT-4 API access enabled in your account in the next few days.

Thanks! I can see gpt-4, gpt-4-0613 and gpt-4-0314 showing up on the Playground Chat Mode, but can I also have gpt-4-32k-0314, gpt-4-32k-0613 and if it exists also text-davinci-004 on the Playground Complete Mode?

[usama-openai]

You should see GPT-4 API access enabled in your account in the next few days.

Thanks! I can see gpt-4, gpt-4-0613 and gpt-4-0314 showing up on the Playground Chat Mode, but can I also have gpt-4-32k-0314, gpt-4-32k-0613 and if it exists also text-davinci-004 on the Playground Complete Mode?

When the time comes, OpenAI will make an announcement if these models are available via a waitlist or other means. If not, you will be able to access these models once they are made available to the general public.