LLM Cultural Assumption Bias

Requirements

Install Python 3.11.x from here

Get an API key from openai with Model-Write rights. Create an .env file inside this directory and put your key next to OPENAI_API_KEY:

OPENAI_API_KEY="<your key>"

Create an environment

Create the environment inside this directory.

python -m venv .venv

activate it like so:

& .\.venv\Scripts\activate

or like this on Linux/Max:

. ./.venv/bin/activate

Usage

Make sure your environment is activated, it should say the name of the environment in you terminal: (.venv) yourpath>

Install the required libraries with:

pip install -r requirements.txt

To query GPT3.5 use the CA_gpt3_query.py script, it takes just one argument, which should be a file containing all queries, the breakfast_prompts.csv is an example.

python CA_gpt3_query.py breakfast_prompts.tsv

The script will then output the queries and their most probably completions in a new file called <query_file>_results.tsv.

To score these results you can use the CA_gpt3_scoring.py script, which will output the distances of all queries from the neutral query. (less is closer)

python CA_gpt3_scoring.py breakfast_prompts_results.tsv