Evaluation techniques for text generation LLMs

[hemajv]

We should look into different techniques to evaluate the API documentation generated by the LLMs.

• Look into evaluation techniques for text generation LLMs and compare them against human analysis
• Have guardrails in place to ensure that the LLM generated output is in alignment with the product documentation and styling
• Generate responses that meet the evaluation metrics
• Quantitative analysis of the different evaluation criteria by conducting experiments with different prompts as inputs to the LLM

[suppathak]

Hi Hema,
A way to check how well a model responds is by using certain criteria tailored to individual needs. This is shown here using the OpenAI API, where they judge the model's response using specific criteria. Please take a look.

https://python.langchain.com/docs/guides/evaluation/

I have also added a demo notebook related to it: ptal
https://github.com/suppathak/foundation-models-for-documentation/blob/langeval/notebooks/langchain-evaluation.ipynb