RR

This is the code repository for the Arxiv paper Rethinking with Retrieval: Faithful Large Language Model Inference. If you use this code for your work, please cite

@article{he2022rethinking,
  title={Rethinking with Retrieval: Faithful Large Language Model Inference},
  author={He, Hangfeng and Zhang, Hongming and Roth, Dan},
  journal={arXiv preprint arXiv:2301.00303},
  year={2022}
}

Installing dependencies

Use virtual environment tools (e.g miniconda) to install packages and run experiments
python==3.7.10
pip install -r requirements.txt

If you couldn't install all dependencies in one virtual environment, you may need three different virtual environments:

• pyserini (pyserini==0.19.2): for BM25 in commonsense_wikipedia.py
• transformers (transformers==4.23.1, sentence_transformers==2.2.2): for huggingface transformer models in commonsense_evidence.py, temporal_evidence.py, tabular_evidence.py
• reasoning: default environment dependencies in the remaining files

Code organization

The code is organized as follows:

• commonsense (commonsense reasoning)
  • commonsense_gpt3.py (chain-of-thought prompting with GPT-3)
  • commonsense_wikipedia.py (retrieve knowledge from wikipedia)
  • commonsense_evidence.py (supporting evidence from retrieved knowledge for inference)
  • commonsense_inference.py (faithful inference)
  • commonsense_utils.py (utility functions)
• temporal (temporal reasoning)
  • temporal_preprocessing.py (data preprocessing)
  • temporal_gpt3.py (chain-of-thought prompting with GPT-3)
  • temporal_wikidata.py (retrieve knowledge from wikidata)
  • temporal_evidence.py (supporting evidence from retrieved knowledge for inference)
  • temporal_inference.py (faithful inference)
  • temporal_utils.py (utility functions)
• tabular (tabular reasoning)
  • tabular_preprocessing.py (data preprocessing)
  • tabular_gpt3.py (chain-of-thought prompting with GPT-3)
  • tabular_evidence.py (supporting evidence from retrieved knowledge for inference)
  • tabular_inference.py (faithful inference)
  • tabular_utils.py (utility functions)
  • relation_templates.json (relation templates; needs to be put under /path/to/working/dir/Tabular/)

Change the working path

Change the /path/to/working/dir to the path to your working directory.

Export OPENAI API KEY

You need to export your own OpenAI API key before running experiments with OpenAI API, i.e., export OPENAI_API_KEY=$YOUR_OPENAI_API_KEY

Data preparation

Download and put StrategyQA dataset under /path/to/working/dir/Commonsense/
Download and put TempQeustions datset (link: http://qa.mpi-inf.mpg.de/TempQuestions.zip) under /path/to/working/dir/Temporal/
Download and put INFOTABS dataset under /path/to/working/dir/Tabular/

Reproducing experiments

To reproduce the experiments for commonsense reasoning:

python commonsense_gpt3.py
python commonsense_wikipedia.py
python commonsense_evidence.py
python commonsense_inference.py

To reproduce the experiments for temporal reasoning:

python temporal_preprocessing.py
python temporal_gpt3.py
python temporal_wikidata.py
python temporal_evidence.py
python temporal_inference.py

To reproduce the experiments for tabular reasoning:

python tabular_preprocessing.py
python tabular_gpt3.py
python tabular_evidence.py
python tabular_inference.py