Add evaluation notebook

[hemajv]

This addresses #24

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[aakankshaduggal]

Thanks @hemajv for adding the notebook. Presents good metrics for evaluation. Would we want all these evaluation metrics to be added to the pipeline and UI or we want to select which ones we want to ship further.

[oindrillac]

Thanks @hemajv for adding the notebook. Presents good metrics for evaluation. Would we want all these evaluation metrics to be added to the pipeline and UI or we want to select which ones we want to ship further.

We have already added some of these to the UI but they may not be actually relevant or well performing. What we could look into from here is

A quantitative eval of whether how often these scores are valid.

 - We can look at cases where we know the generated output is deliberately wrong and see how the allotted scores perform
 - Do this over a number of outputs (say 10 outputs) for each relevant default criteria and some custom criteria

That will help us determine which out of the langchain eval criteria are doing well and also whether or not this is working most of the time.

We can do the same evaluation also of the custom gpt 3 prompt evaluation and see how well that is performing.

[oindrillac]

Thanks for the notebook @hemajv 🎉 This is a great start

I added a comment on how we can follow this up with some quantitative eval to get a sense of which criteria we should include and whether this is actually working most of the time.

[hemajv]

Thanks @hemajv for adding the notebook. Presents good metrics for evaluation. Would we want all these evaluation metrics to be added to the pipeline and UI or we want to select which ones we want to ship further.

We have already added some of these to the UI but they may not be actually relevant or well performing. What we could look into from here is

A quantitative eval of whether how often these scores are valid.

 - We can look at cases where we know the generated output is deliberately wrong and see how the allotted scores perform
 - Do this over a number of outputs (say 10 outputs) for each relevant default criteria and some custom criteria

That will help us determine which out of the langchain eval criteria are doing well and also whether or not this is working most of the time.

We can do the same evaluation also of the custom gpt 3 prompt evaluation and see how well that is performing.

Yes, definitely agree on this 👍 Seems like some of the Langchain criteria are doing a pretty decent job in evaluating the outputs accurately. As you mentioned, we need to try this out with a certain set of cases to get a better understanding on how well its doing.

[oindrillac]

Yes, definitely agree on this 👍 Seems like some of the Langchain criteria are doing a pretty decent job in evaluating the outputs accurately. As you mentioned, we need to try this out with a certain set of cases to get a better understanding on how well its doing.

I started creating a framework to quantitatively evaluate this as an extension to this notebook. Will also add that in a separate PR

[hemajv]

@oindrillac addressed the comments and pushed new commits, ready for your review

[oindrillac]

Thanks for the changes @hemajv LGTM 👍