About the Results in the paper.

[littlefive5]

I rerun the code in the IR repo. However, I just get the result 22%, which is far lower than the 34% in the paper for USMLE. Is there any other setting for the IR method?

[jind11]

I just now updated the data in the github repo by adding the 4 options version of data I previously used. Let me whether you can replicate the numbers in my paper using this version of data. Thanks!

[zyngielg]

I tried running the solvers/textsearch.py on the 4_options/dev.jsonl:

• for topn=10 the accuracy was 28.7%
• for topn=5 the accuracy was 27.7%

[littlefive5]

I tried running the solvers/textsearch.py on the 4_options/dev.jsonl:

• for topn=10 the accuracy was 28.7%
• for topn=5 the accuracy was 27.7%

Me too. I also tried TW dataset and I got about 31% when the topn is 15.

[jind11]

hmmm, this code was written a year ago and I tried my best to release the old code here without running and verifying. Thanks for helping me find out the issue here. I am gonna check on my side the source of performance discrepancy but it may be finished after 5/17 which is the conference ddl I am now busy with. I am so sorry for the slow process. Thank you for your great patience.

[MotzWanted]

@jind11, did you get a chance to solve the issue?

And is it possible for you to release the reader model as well?

[vlievin]

@jind11 What is the status on reproducing the results? Are you 100% the dataset is correct? Great work by the way, that's a very nice dataset you have built here!