Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Evals documentation missing some crucial context / information. #1391

Closed
ssbushi opened this issue Nov 25, 2024 · 0 comments · Fixed by #1394
Closed

[Docs] Evals documentation missing some crucial context / information. #1391

ssbushi opened this issue Nov 25, 2024 · 0 comments · Fixed by #1394
Assignees
Labels
docs Improvements or additions to documentation

Comments

@ssbushi
Copy link
Contributor

ssbushi commented Nov 25, 2024

Is your report related to a problem? Please describe.
There are a few things that are glossed over the evaluation documentation (Evaluation and Writing an Evaluator Plugin)

Evaluator:

  • How to pass auth to the eval:flow command.
  • The format of the input.json file for eval:flow.

Writing an Evaluator Plugin:

  • Add more context to code snippets (eg: it is unclear if regexMatchScore and fillScores are provided in Genkit or if the user should implement)
  • Provide a simple explanation of the nature of EvaluatorFn.

Additional context
https://discord.com/channels/1255578482214305893/1309828408141807657

Some Genkit imports were missing / not updated to 0.9. This was addressed in this PR by @DenisVCode: #1388

@ssbushi ssbushi added the docs Improvements or additions to documentation label Nov 25, 2024
@ssbushi ssbushi self-assigned this Nov 25, 2024
@ssbushi ssbushi moved this to In Progress in Genkit Backlog Nov 25, 2024
@ssbushi ssbushi mentioned this issue Nov 25, 2024
1 task
@github-project-automation github-project-automation bot moved this from In Progress to Done in Genkit Backlog Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Improvements or additions to documentation
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant