Why is my F1 so low？

[Arima-Kisho]

I'm a newbie and I'm having some issues running your code, hoping to get your help.
In the case of Adaptive Retrieval, I use TinyLlama and I can't get the F1 in the article：

{'data_source': 'retrievalqa', 'total_data_count': 2785, 'retrieval_frequency': 1137, 'retrieval_rate': 40.8, 'match_score': 54.0, 'f1_score': 12.5, 'em_score': 0.1, 'accuracy_score': 27.6, 'match_total': 1503, 'f1_total': 348.9463775190812, 'em_total': 3.0, 'accuracy_total': 769.0, 'total_q_tokens': 35779, 'total_context_tokens': 715530, 'total_no_retrieval_tokens': 35779, 'total_always_retrieval_tokens': 715530, 'estimate_no_retrieval_cost': 0.017889500000000003, 'estimate_always_retrieval_cost': 0.3756545, 'saved_cost_rate': 0.9523777833088649, 'args': {'openai_config_path': './openai_config.txt', 'data_source': 'retrievalqa', 'retrieval_mode': 'adaptive_retrieval', 'input_data_path': './data/retrievalqa.jsonl', 'output_score_path': './results/adaptive_retrieval/TinyLlama/TinyLlama-1.1B-Chat-v1.0/m=vanilla/t=0.0/score_retrievalqa_seed20.json', 'output_prediction_path': './results/adaptive_retrieval/TinyLlama/TinyLlama-1.1B-Chat-v1.0/m=vanilla/t=0.0/predict_retrievalqa_seed20.jsonl', 'model_name': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0', 'max_tokens': 100, 'batch_size': 1, 'doc_top_n': 5, 'limit_input': 0, 'prompt_method': 'vanilla', 'seed': 20, 'temperature': 0.0, 'top_p': 1.0, 'world_size': 1}}
./results/adaptive_retrieval/TinyLlama/TinyLlama-1.1B-Chat-v1.0/m=vanilla/t=0.0
./results/adaptive_retrieval/TinyLlama/TinyLlama-1.1B-Chat-v1.0/m=vanilla/t=0.0

My F1 is only 12.5, but the F1 in Table 10 in the article shows 48.5, is there a problem with my settings somewhere？
What data correspond to Retrieval Acc and Precision and Recall? I can't find the corresponding fractional value？

[ZhangzihanGit]

Hi Arima,

I'm sorry for not getting back to you sooner. I just saw this message.

The 'f1_score': 12.5 is the token-level F1 score for QA, as calculated here, which is different from the F1 score in Table 10. Table 10 presents the retrieval macro precision, recall, and F1, where we only focus on the questions that require retrievals. We use the following code to calculate these values:

from sklearn.metrics import accuracy_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score

def calc_precision_recall_f1(preds, gts):
    # accuracy: (tp + tn) / (p + n)
    accuracy = accuracy_score(preds, gts)
    print('Accuracy: %f' % accuracy)
    # precision tp / (tp + fp)
    precision = precision_score(preds, gts, average="macro")
    print('Precision: %f' % precision)
    # recall: tp / (tp + fn)
    recall = recall_score(preds, gts, average="macro")
    print('Recall: %f' % recall)
    # f1: 2 tp / (2 tp + fp + fn)
    f1 = (2 * precision * recall) / (precision + recall)
    print('F1 score: %f' % f1)

    return accuracy, precision, recall, f1

do_retrieval_scores = [item["do_retrieval"] for item in input_data]
do_retrieval_scores = [int(i == True)  for i in do_retrieval_scores]
retrieval_gts = [item["param_knowledge_answerable"] for item in input_data]

retrieval_accuracy, precision, recall, f1 = calc_precision_recall_f1(do_retrieval_scores, retrieval_gts)

Thanks.