feat: generative metric

[dtch1997]

Summary by CodeRabbit

• New Features

  • Added "python-dotenv>=1.0.1" as a new dependency.
  • Introduced functionality to download pre-trained language models and set up for inference.
  • Added functionality for evaluating generation tasks using steering vectors and pipelines.
  • Introduced functionality for evaluating a generative model using steering vectors and different datasets.
  • Added prompt for scoring answers based on corrigibility in judge_prompts.json.

• Refactor

  • Enhanced evaluation and generation processes with new functions and refactored logic in various files.

• Tests

  • Added test cases for get_prediction and get_eval_hooks in tests/core/test_evaluate.py.

[coderabbitai]

Walkthrough

The project underwent significant updates including adding a new dependency in pyproject.toml, enhancing functionalities in files like repepo/core/evaluate.py, repepo/data/multiple_choice/make_mwe_persona.py, and introducing new scripts like download_models.py, evaluate_generation.py, and judge_eval.py. These changes aim to improve evaluation capabilities, model management, and judge rating assessments within the project.

Changes

File(s)	Change Summary
pyproject.toml	Added "python-dotenv>=1.0.1" as a new dependency.
repepo/core/evaluate.py	Added Completion to repepo.core.types, added generations field to EvalPrediction, introduced new functions, and refactored logic.
repepo/data/multiple_choice/make_mwe_persona.py	Renamed function, updated parameters and return type, and modified implementation of convert_mwe_personas_dataset_gen function.
repepo/notebooks/download_models.py	Added functionality to download and set up a pre-trained language model using Hugging Face Transformers.
repepo/notebooks/evaluate_generation.py	Introduced functionality for evaluating generation tasks using steering vectors and pipelines.
repepo/notebooks/generative_metric/evaluate_generation_2.py	Added functions for preprocessing examples, generating completion templates, obtaining steering vector results, and combining evaluation results.
repepo/notebooks/generative_metric/judge_eval.py	Introduced functionality for evaluating judge ratings using OpenAI's GPT-3.5 model and performing data visualization.
repepo/notebooks/generative_metric/judge_prompts.json	Updated with a prompt for scoring answers based on corrigibility.
tests/core/test_evaluate.py	Added test functions using pytest for get_prediction and get_eval_hooks.

Note

Pull Request Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://coderabbit.ai

---

Recent Review Details

Commits

Files that changed from the base of the PR and between 51d0a0c and 8ae4ba9.

Files selected for processing (3)

• repepo/core/evaluate.py (4 hunks)
• repepo/core/pipeline.py (3 hunks)
• repepo/notebooks/generative_metric/evaluate_generation_2.py (1 hunks)

---

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.