Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hyperparameter of the number of distractor documents and the ratio of golden documents in training RAFT #325

Closed
yifan1130 opened this issue Apr 7, 2024 · 1 comment · Fixed by #353
Closed

Hyperparameter of the number of distractor documents and the ratio of golden documents in training RAFT #325

yifan1130 opened this issue Apr 7, 2024 · 1 comment · Fixed by #353

Comments

@yifan1130
Copy link

Hi, I just wonder what the number of distractor documents and the ratio of golden documents in training RAFT are in producing the results in the paper. I saw the default ratio in the script the ratio is 1.0, which means no distractor documents are used. I wonder whether the default number in the scripts represents the hyperparameter used to produce the results in the paper?
@kaiwen129
Copy link
Contributor

Hi, the optimal value for the P% hyperparameter may vary from dataset to dataset. To answer your question, the paper used various values of P%, not just the default value of 1.0 in the script.

@ShishirPatil ShishirPatil mentioned this issue Apr 14, 2024
Merged
ShishirPatil added a commit that referenced this issue Apr 14, 2024
@ShishirPatil
Update raft.py with default p to match paper (#353)
7032c0e 
Change `p` which dictates the fraction of dataset with golden documents
in them (vs) no golden documents.

So, p = 0.8 means, for 80% of the train data set, `A* = Q + D* + D1 ..
Dn` and for 20% of the train data set `A* = Q + D1 .. Dn` where `D*`
are/is the golden document with the answer `A*`.

Close #325
devanshamin pushed a commit to devanshamin/gorilla that referenced this issue Jul 9, 2024
@ShishirPatil
Update raft.py with default p to match paper (ShishirPatil#353)
717abe4 
Change `p` which dictates the fraction of dataset with golden documents
in them (vs) no golden documents.

So, p = 0.8 means, for 80% of the train data set, `A* = Q + D* + D1 ..
Dn` and for 20% of the train data set `A* = Q + D1 .. Dn` where `D*`
are/is the golden document with the answer `A*`.

Close ShishirPatil#325
aw632 pushed a commit to vinaybagade/gorilla that referenced this issue Aug 22, 2024
@ShishirPatil
Update raft.py with default p to match paper (ShishirPatil#353)
3058927 
Change `p` which dictates the fraction of dataset with golden documents
in them (vs) no golden documents.

So, p = 0.8 means, for 80% of the train data set, `A* = Q + D* + D1 ..
Dn` and for 20% of the train data set `A* = Q + D1 .. Dn` where `D*`
are/is the golden document with the answer `A*`.

Close ShishirPatil#325
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update raft.py with default p to match paper ShishirPatil/gorilla
2 participants
@kaiwen129 @yifan1130