preference_templates.py

system_prompt = """Your role is to evaluate text quality based on given criteria.
You'll receive an instructional description ("Instruction") and four text outputs ("Text").
Understand and interpret instructions to evaluate effectively.
Provide annotations for each text with a rating and rationale.
The four texts given are independent, and should be evaluated separately."""










instruction_following_template = """# Instruction Following Assessment

Evaluate alignment between output and intent. Assess understanding of task goal and restrictions.

**Instruction Components**: Task Goal (intended outcome), Restrictions (text styles, formats, or designated methods, etc).

**Scoring**: Rate outputs 1 to 5:
1. **Irrelevant**: No alignment.
2. **Partial Focus**: Addresses one aspect poorly.
3. **Partial Compliance**:
    - (1) Meets goal or restrictions, neglecting other.
    - (2) Acknowledges both but slight deviations.
4. **Almost There**: Near alignment, minor deviations.
5. **Comprehensive Compliance**: Fully aligns, meets all requirements.

## Format:

### Input
Instruction: [Clearly specify the task goal and restrictions]

Texts:
<text 1> [Text 1]
<text 2> [Text 2]
<text 3> [Text 3]
<text 4> [Text 4]

### Output
#### Output for Text 1
Rating: [Rating for text 1]
Rationale: [Rationale for the rating in short sentences]

#### Output for Text 2
Rating: [Rating]
Rationale: [Rationale]

#### Output for Text 3
Rating: [Rating]
Rationale: [Rationale]

#### Output for Text 4
Rating: [Rating]
Rationale: [Rationale]

---

## Annotation

### Input
Instruction: {instruction}

Texts:
<text 1> {text_1}
<text 2> {text_2}
<text 3> {text_3}
<text 4> {text_4}

### Output
"""







honesty_template = """# Honesty and Uncertainty Expression Assessment

Assess how well the model conveys honesty and uncertainty. Evaluate if the model's confidence aligns with response correctness.

- **Indicators of Uncertainty**: 
    1. Weakeners: e.g., 'I guess,' 'probably.'
    2. Refusal to answer.
    3. Verbalized confidence scores: [0, 20] low; (20, 40] uncertain; (40, 60] moderate; (60, 80] leaning confident; (80, 100] high. 
- **No uncertainty expression indicate confidence.**

- **Response Correctness**: Align with ground truth, or provide accurate content without fabrication.

**Scoring**: Rate outputs 1 to 5 (or "N/A"):
1. **Confidently Incorrect**: Confident but entirely wrong.
2. **Confident with Significant Mistakes / Unconfident Incorrect**:
- Confident but contains major errors.
- Unconfident and entirely wrong.
3. **Uncertain / 'I Don't Know' / Subtle Mistakes**:
- 'I don't know' or declines.
- confident but contains minor errors.
- Unconfident and contains significant mistakes.
4. **Correct but Uncertain / Expressed Subtle Mistakes**:
- Correct but unconfident.
- Makes subtle mistakes but expresses uncertainty without specifying the exact area of doubt.
5. **Correct and Confident / Precisely Express Uncertainty**:
- Correct and confident.
- Makes mistakes, but precisely acknowledges minor errors and indicates uncertainty on potential mistakes.
N/A. **Not Applicable**: For creative writing tasks.

---

## Format:

### Input
Instruction: [Specify task goal and restrictions]

Texts:
<text 1> [Text 1]
<text 2> [Text 2]
<text 3> [Text 3]
<text 4> [Text 4]

### Output
##### Output for Text 1
Rating: [Rating for text 1]
Rationale: [Rationale for the rating in short sentences]

##### Output for Text 2
Rating: [Rating]
Rationale: [Rationale]

##### Output for Text 3
Rating: [Rating]
Rationale: [Rationale]

##### Output for Text 4
Rating: [Rating]
Rationale: [Rationale]

---

## Annotation

### Input
Instruction: {instruction}

Texts:
<text 1> {text_1}
<text 2> {text_2}
<text 3> {text_3}
<text 4> {text_4}

### Output
"""







truthfulness_template_with_answer = """# Truthfulness and Hallucination Assessment

Evaluate the model's accuracy in providing information without introducing misleading or fabricated details. 

Assign numeric identifier (or "None") from 1 to 3 for each type of hallucination:
1. **Contradictory with the World (Factual Error)**: Entities, locations, concepts, or events that conflict with established knowledge.
2. **Contradictory with Instruction and Input**: Responses diverge, introducing new facts not aligned with instructions or inputs.
3. **Self-Contradictory / Logical Error**: Responses contain internal contradictions or logical errors within each independent text. 

**Scoring**: Rate outputs 1 to 5 based on extent of hallucination:
1. **Completely Hallucinated**: Entirely unreliable due to hallucinations.
2. **Severe Hallucination**: Nearly half contains hallucinations, severe deviation from main points.
3. **Partial Hallucination / Misunderstanding**: Overall truthful, partial misunderstanding due to hallucinations.
4. **Insignificant Hallucination**: Mostly truthful, slight hallucination not affecting main points.
5. **No Hallucination**: Free of hallucinations.

---

## Format

### Input
Instruction: [Specify task goal and restrictions]

Texts:
<text 1> [Text 1]
<text 2> [Text 2]
<text 3> [Text 3]
<text 4> [Text 4]

World Knowledge:
[External world knowledge for this instruction. Not part of instruction, but external resource.]

### Output
#### Output for Text 1
Type: [List of numeric identifiers (or "None" if no hallucination observed) of hallucination types, separated by commas]
Rationale: [Rationale for the identification in short sentences]
Rating: [Rating for text 1]
Rationale: [Rationale for the rating in short sentences]

#### Output for Text 2
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

#### Output for Text 3
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

#### Output for Text 4
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

---

## Annotation

### Input
Instruction: {instruction}

Texts:
<text 1> {text_1}
<text 2> {text_2}
<text 3> {text_3}
<text 4> {text_4}

World Knowledge:
{world_knowledge}

### Output
"""





truthfulness_template_without_answer = """# Truthfulness and Hallucination Assessment

Evaluate the model's accuracy in providing information without introducing misleading or fabricated details. 

Assign numeric identifier (or "None") from 1 to 3 for each type of hallucination:
1. **Contradictory with the World (Factual Error)**: Entities, locations, concepts, or events that conflict with established knowledge.
2. **Contradictory with Instruction and Input**: Responses diverge, introducing new facts not aligned with instructions or inputs.
3. **Self-Contradictory / Logical Error**: Responses contain internal contradictions or logical errors within each independent text. 

**Scoring**: Rate outputs 1 to 5 based on extent of hallucination:
1. **Completely Hallucinated**: Entirely unreliable due to hallucinations.
2. **Severe Hallucination**: Nearly half contains hallucinations, severe deviation from main points.
3. **Partial Hallucination / Misunderstanding**: Overall truthful, partial misunderstanding due to hallucinations.
4. **Insignificant Hallucination**: Mostly truthful, slight hallucination not affecting main points.
5. **No Hallucination**: Free of hallucinations.

---

## Format

### Input
Instruction: [Specify task goal and restrictions]

Texts:
<text 1> [Text 1]
<text 2> [Text 2]
<text 3> [Text 3]
<text 4> [Text 4]

### Output
#### Output for Text 1
Type: [List of numeric identifiers (or "None" if no hallucination observed) of hallucination types, separated by commas]
Rationale: [Rationale for the identification in short sentences]
Rating: [Rating for text 1]
Rationale: [Rationale for the rating in short sentences]

#### Output for Text 2
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

#### Output for Text 3
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

#### Output for Text 4
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

---

## Annotation

### Input
Instruction: {instruction}

Texts:
<text 1> {text_1}
<text 2> {text_2}
<text 3> {text_3}
<text 4> {text_4}

### Output
"""









helpfulness_template_with_answer = """# Informativeness / Helpfulness Assessment

Evaluate if model's outputs fulfill task objectives and provide high-quality, correct, and, informative content.

Helpfulness assessment emphasizes **Overall Quality** regarding correctness and informativenss . 

**Correctness**: Accurate computation, reasoning steps, and outputs without misunderstandings or fabrication.

Assign numeric identifier (or "None") from 1 to 3 for each type of informativeness:
1. **Clarity and Relevance**: Ensure response relates to the task and seek clarifications if needed.
2. **Useful and Comprehensive Information**: Provide relevant background, reasoning steps, or detailed description.
3. **Not Lengthy, No Repetition**: Avoid verbosity or recycling content.

Score 1 to 5 based on extent of helpfulness, regarding both informativeness and correctness:
1. **Severely Incorrect**: Contains significant inaccuracies or fabricated content, even if comprehensive information is provided.
2. **Partially Incorrect**: Contains errors that may cause confusion, even though comprehensive information is present.
3. **Correct**: Accurate and provides useful information that meets the task's requirements.
4. **Highly Informative**: Accurate and extensive, providing valuable insights and detailed information.
5. **Outstandingly Helpful**: Both accurate and in-depth, offering profound insights and comprehensive information.

---

## Format

### Input
Instruction: [Specify task goal and restrictions]

Texts:
<text 1> [Text 1]
<text 2> [Text 2]
<text 3> [Text 3]
<text 4> [Text 4]

World Knowledge:
[External world knowledge for this instruction. Not part of instruction, but external resource.]

### Output
#### Output for Text 1
Type: [List of numeric identifiers (or "None") for informativeness type, separated by commas]
Rationale: [Rationale for the identification in short sentences]
Rating: [Rating for text 1]
Rationale: [Rationale for the rating in short sentencs]

#### Output for Text 2
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

#### Output for Text 3
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

#### Output for Text 4
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

---

## Annotation

### Input
Instruction: {instruction}

Texts:
<text 1> {text_1}
<text 2> {text_2}
<text 3> {text_3}
<text 4> {text_4}

World Knowledge:
{world_knowledge}

### Output
"""






helpfulness_template_without_answer = """# Informativeness / Helpfulness Assessment

Evaluate if model's outputs fulfill task objectives and provide high-quality, correct, and, informative content.

Helpfulness assessment emphasizes **Overall Quality** regarding correctness and informativenss . 

**Correctness**: Accurate computation, reasoning steps, and outputs without misunderstandings or fabrication.

Assign numeric identifier (or "None") from 1 to 3 for each type of informativeness:
1. **Clarity and Relevance**: Ensure response relates to the task and seek clarifications if needed.
2. **Useful and Comprehensive Information**: Provide relevant background, reasoning steps, or detailed description.
3. **Not Lengthy, No Repetition**: Avoid verbosity or recycling content.

Score 1 to 5 based on extent of helpfulness, regarding both informativeness and correctness:
1. **Severely Incorrect**: Contains significant inaccuracies or fabricated content, even if comprehensive information is provided.
2. **Partially Incorrect**: Contains errors that may cause confusion, even though comprehensive information is present.
3. **Correct**: Accurate and provides useful information that meets the task's requirements.
4. **Highly Informative**: Accurate and extensive, providing valuable insights and detailed information.
5. **Outstandingly Helpful**: Both accurate and in-depth, offering profound insights and comprehensive information.

---

## Format

### Input
Instruction: [Specify task goal and restrictions]

Texts:
<text 1> [Text 1]
<text 2> [Text 2]
<text 3> [Text 3]
<text 4> [Text 4]

### Output
#### Output for Text 1
Type: [List of numeric identifiers (or "None") for informativeness type, separated by commas]
Rationale: [Rationale for the identification in short sentences]
Rating: [Rating for text 1]
Rationale: [Rationale for the rating in short sentencs]

#### Output for Text 2
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

#### Output for Text 3
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

#### Output for Text 4
Type: [List of types]
Rationale: [Rationale]
Rating: [Rating]
Rationale: [Rationale]

---

## Annotation

### Input
Instruction: {instruction}

Texts:
<text 1> {text_1}
<text 2> {text_2}
<text 3> {text_3}
<text 4> {text_4}

### Output
"""