Prompt Experimentation and UI updates #36
Conversation
Co-authored-by: Aakanksha Duggal <aduggal@redhat.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
| "**LangChain evaluation on grammar, descriptiveness and helpfulness:**", | ||
| help="Use Langchain to evaluate on cutsom criteria (this list can be updated based on what we are looking to see from the generated docs" | ||
| "**LLM based evaluation on logic, correctness and helpfulness:**", | ||
| help="Use Langchain Criteria based Eval to evaluate on cutsom criteria (this list can be updated based on what we are looking to see from the generated docs). Note this is language mo0del based evaluation and not always a true indication of the quality of the output that is generatged." |
There was a problem hiding this comment.
small nit fix: Note this is language mo0del.... should be Note this is language model.... and ...output this is generatged should be ....output that is generated.
| @@ -0,0 +1,3141 @@ | |||
| ,model,prompt,code_file,part,response,langchain_helpfulness,langchain_correctness,langchain_logical,instruction,total_langchain_score | |||
There was a problem hiding this comment.
repo structuring change: We should perhaps move all the CSV files and the eval_df.pkl file into a separate folder: /evaluation/data
There was a problem hiding this comment.
I couldn't review this notebook on Reviewnb and the file diff was too large to leave comments line by line, so summarizing my comments here:
- Include a small description at the beginning of the notebook to explain what the notebook is about
- Since the
get_response()andappend_row_to_dataframe()functions are already defined in thehelper_functions.ipynb, maybe you can invoke the function from there rather than defining the function again in this notebook? - Small nit fix: in the
Conclusionsection, I think point 3 needs to be rephrased from :When he granite model it fails, it fails becuase half....toWhen the granite model fails, it fails because half of the time....
|
As discussed, we will be addressing the comments in a separate PR. Merging this for now. /lgtm |
No description provided.