LLM-as-Meta-Reviewer Assistants'

Folder Organization

• Annotation Files
  • Contains human annotation for micro and macro evaluation.
• LLM Responses
  • This folder contains responses generated by three LLMs (i.e., PalM2, GPT-3.5, and Llama2) for four prompt levels. Here, LLM1, LLM2, and LLM3 represents PalM2, GPT-3.5, and Llama2. The responses are in the columns res_prompt1,res_prompt2, res_prompt3, and res_prompt4.
• GPT Ratings
  • Contains the ratings given by GPT-4o on the LLM generated responses. The human and GPT ratings are merged together in the files. GPT_rating_LLMx and Macro_GPT_score contains micro and macro evaluation ratings of both human and GPT-4o respectively.
  • corr_average_likert_ceil file contains the correlations between human and GPT-4o in micro-evaluation for each LLMs.
• Meta-Review Data
  • The curated dataset used for the meta-review generation. Review1, Review2, and Review3 columns have the peer reviews and Meta_Review column contain the meta-reviews written by humans.

Citation