Reproducibility Issues (Bias Score and Accuracy)

[rajneesh407]

I have been working on reproducing the results using the code from your repository and implementing it in Python. I successfully converted the provided R code into Python, and the outputs from both versions match. However, the results from your repository code do not align with the figures presented in your research paper.

Code used: [BBQ_calculate_bias_score.R] (https://github.com/nyu-mll/BBQ/blob/main/analysis_scripts/BBQ_calculate_bias_score.R)
Research paper link: [QA Bias Benchmark] (https://github.com/nyu-mll/BBQ/blob/main/QA_bias_benchmark.pdf)

Here is the output I obtained using the R code from your repository:

Comparing Dberta V3 Base (For Disambiguous):
Comparison With Paper :
- Age : Match
- Disability : Match
- Gender Identity : 13.9 (R-Code) instead of 15 (paper)
- Gender Identity Names : 12.7 instead of 14
- Nationality : Match
- Physical Appearance : 41.8 instead of 41
- Race and Ethnicity : 4.7 instead of 4.6
- Race and Ethnicity Names : Match
- Religion : Match
- Sexual Orientation : Match
- SES : Match

Same patter can be seen across other models also.

Any help/clarifications would be appreciated here. Thanks