Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Eval] Relay evaluation metrics/schema from the Impact Metrics Garden #47

Open
bitbeckers opened this issue Apr 22, 2024 · 4 comments
Open

Comments

@bitbeckers
Copy link
Collaborator

No description provided.

@bitbeckers
Copy link
Collaborator Author

Blocked on having a SSOT for metrics garden

@holkexyz
Copy link
Member

@bitbeckers would this be a full evaluation schema in addition to the attestation that we currently have in the evaluator app?

@bitbeckers
Copy link
Collaborator Author

bitbeckers commented May 1, 2024

Yes, potentially even lists of criteria or concattenated attestaions (c-atts). I thinks it's reasonable to note have this in scope now, as the repository of schema's is being rebuilt at the moment. But the Metrics Garden is a good source of metrics and schemas to pull from.

Edit: we'd go from the <3 schemas we know now, to > 50

@holkexyz
Copy link
Member

holkexyz commented May 1, 2024

OK, but starting with a low number of schemas can also be useful for user not to get lost in them. I want to be very intentional about it. I think it is more a user question than a technical question.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants