adding documentation #282
adding documentation #282
Conversation
There was a problem hiding this comment.
Overall looks pretty good. Some notes:
- the saving results section is quite outdated and this is probably one of the most important sections (see my suggestions)
- missing an explanation on what each metric does, how things are normalized etc
- arguments for the commands, such as pretrained=... and task list (and task groups) should likely also be better explained, as this is the entrypoint to run evals
docs/source/index.md
Outdated
| Note that it is still very much early so don't expect 100% stability ^^' In | ||
| case of problems or questions, feel free to open an issue! |
There was a problem hiding this comment.
This is still the same text as from the original release I believe. Maybe at this point it's ok to rephrase a bit into something less "it's very alpha" and more along the lines of "Even though lighteval has successfully been used in a variety of different research projects both internally and externally, keep in mind that parts of it are liable to change, and things may break. In case of problems or questions, feel free to open an issue!"
| --output_dir="./evals/" | ||
| ``` | ||
|
|
||
| Here, `--tasks` refers to either a comma-separated list of supported tasks from |
There was a problem hiding this comment.
maybe also mention task groups #308
There was a problem hiding this comment.
Once the PR is merged yeah I will add it
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
Co-authored-by: Guilherme Penedo <nostrumg@gmail.com>
|
Do you want to keep this one open since we now have the wiki? |
|
nope we can close |
No description provided.