Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add evaluation pipeline(s) #100

Merged
merged 68 commits into from
May 6, 2024
Merged

Add evaluation pipeline(s) #100

merged 68 commits into from
May 6, 2024
+1,708 −99

Conversation

strickvl
Copy link
Contributor

Adds options for evaluation pipelines as well as generation to support those pipelines.

strickvl added 30 commits April 9, 2024 13:56
@strickvl
add validation for DB connection
46aa98d
@strickvl
add document class
5baab90
@strickvl
add better data struct for document
81b0b05
@strickvl
add tests
9eef4b8
@strickvl
add helper function go extract parent section from url
6996d3e
@strickvl
remove excess kw arg
c12352e
@strickvl
pre-populate the document at url loader step
570dd19
@strickvl
formatting
5233338
@strickvl
update pipeline
a3f47e1
@strickvl
add tiktoken for token counts
de99928
@strickvl
add materializers folder
0e9c476
@strickvl
add token_count attribute + populate the db
3c56a1d
@strickvl
remove breakpoint
4366dc4
@strickvl
embedding visualisation notebook
f2ded7a
@strickvl
add custom materializer
f554370
@strickvl
docs materializer uses lists now
af1d97b
@strickvl
update materializer
514afc7
@strickvl
custom materializer
7664c00
@strickvl
increase chunk size and use default openai models
f42b2c7
@strickvl
add rag retrieval unit test
9a4b73e
@strickvl
format and add logging for retrieval test
29a2ef6
@strickvl
add good responses eval
6e0fe5d
@strickvl
allow db password as env var
0e90d2f
@strickvl
update positive responses
26f32c5
@strickvl
formatting
13b080c
@strickvl
use different concurrent executor
4d2113d
@strickvl
tests become steps
c5c0959
@strickvl
aws s3 requirements (optional)
6752748
@strickvl
add visualization of eval results
215fe08
@strickvl
WIP fix for materializer
066175d
@strickvl strickvl added enhancement New feature or request internal labels Apr 16, 2024
@review-notebook-app Review Notebook App
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@strickvl strickvl requested a review from AlexejPenner April 16, 2024 13:12
@strickvl strickvl marked this pull request as ready for review April 16, 2024 15:07
@strickvl
Add reranking functionality to retrieval process (#101)
22bea49 
* add reranking to inference

* make reranking optional
AlexejPenner
AlexejPenner reviewed Apr 17, 2024
View reviewed changes
Copy link
Collaborator

@AlexejPenner AlexejPenner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looked at all files except the notebook so far, found two things that looked suspicious. Else looking good :)

@strickvl strickvl merged commit 1c48b28 into main May 6, 2024
1 of 3 checks passed
@strickvl strickvl deleted the feature/evaluation-llm-complete-guide branch May 6, 2024 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AlexejPenner AlexejPenner AlexejPenner left review comments

Assignees
No one assigned
Labels
enhancement New feature or request internal
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@strickvl @AlexejPenner