feat: implement TestsetGenerator

[jjmachan]

Putting everything together

from ragas.experimental.testset import TestsetGenerator
from ragas.llms import llm_factory

generator = TestsetGenerator(llm=llm)
dataset = generator.generate_with_langchain_docs(docs, test_size=10)
dataset

Summary from Github CoPilot

This pull request introduces several changes to the ragas project, focusing on improvements to the test set generation and integration with Langfuse. The changes include new utility functions, updates to class structures, and enhancements to documentation.

Enhancements to Test Set Generation:

• src/ragas/experimental/testset/generators/abstract.py: Refactored AbstractQAScenario to AbstractQuestionScenario and added several methods for generating scenarios and samples, including handling cases with no clusters. [1] [2] [3] [4] [5] [6] [7] [8]
• src/ragas/experimental/testset/generators/generator.py: Introduced the TestsetGenerator class with methods to generate evaluation datasets using Langchain documents and custom scenarios.
• src/ragas/experimental/testset/generators/utils.py: Added calculate_split_values utility function to help in splitting scenarios based on probabilities.

Documentation and Configuration Updates:

• docs/references/testset_generation.md: Added new documentation for test set generation.
• mkdocs.yml: Updated navigation to include the new test set generation documentation.

Codebase Simplification and Refactoring:

• src/ragas/experimental/testset/generators/abstract.py: Simplified the generate_scenarios method and added handling for empty clusters.
• src/ragas/experimental/testset/generators/base.py: Removed redundant abstract methods and added a new abstract method generate_sample.
• src/ragas/experimental/testset/transforms/__init__.py: Consolidated imports and added a default_transforms function to define default transformation sequences.

Integration with Langfuse:

• docs/howtos/integrations/langfuse.ipynb: Minor formatting improvements for better readability.

These changes collectively enhance the functionality and maintainability of the test set generation process, improve documentation, and ensure better integration with external libraries like Langfuse.finish prompt object

[jjmachan]

now things to do before pre-release

• 2 more simulators for testset generation
• add callbacks and traces
• finish prompt object
• switch everything from experimental to main