Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Erro in training dataset #3

Open
danbst opened this issue Sep 23, 2019 · 2 comments
Open

Erro in training dataset #3

danbst opened this issue Sep 23, 2019 · 2 comments

Comments

@danbst
Copy link

danbst commented Sep 23, 2019

This is an excerpt from train.json:

{
  "question": "did abraham lincoln write the letter in saving private ryan",
  "title": "Bixby letter",
  "answer": true,
  "passage": "In the 1998 war film Saving Private Ryan, General George Marshall (played by Harve Presnell) reads the Bixby letter to his officers before giving the order
 to find and send home Private James Francis Ryan after Ryan's three brothers died in battle."
}

Should the correct answer be false? I can't see how "abraham lincoln" is mentioned in this passage.

@hannamw
Copy link

hannamw commented Feb 23, 2020

The Bixby Letter is a letter that was in fact written by Abraham Lincoln, though that isn't mentioned in the passage. I'm not sure how a model would know this - I sure didn't.

@alenas
Copy link

alenas commented Feb 20, 2024

Seems like a lot of passages contain not enough contextual information to be able to answer these questions without prior knowledge. These tests are only suited for LLM's as my AGI model does not need to read the whole internet to function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants