Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better error message for lm_eval script #444

Merged
merged 2 commits into from
Jul 3, 2024
Merged

Conversation

drisspg
Copy link
Contributor

@drisspg drisspg commented Jun 26, 2024

Summary

Add install instructions and make output printing better

+----------+--------------------------------------------+
| Task     | Metrics                                    |
+==========+============================================+
| wikitext | +-----------------------------+----------+ |
|          | | bits_per_byte,none          | 0.538825 | |
|          | +-----------------------------+----------+ |
|          | | bits_per_byte_stderr,none   | N/A      | |
|          | +-----------------------------+----------+ |
|          | | byte_perplexity,none        | 1.452788 | |
|          | +-----------------------------+----------+ |
|          | | byte_perplexity_stderr,none | N/A      | |
|          | +-----------------------------+----------+ |
|          | | word_perplexity,none        | 7.368333 | |
|          | +-----------------------------+----------+ |
|          | | word_perplexity_stderr,none | N/A      | |
|          | +-----------------------------+----------+ |
+----------+--------------------------------------------+

Copy link

pytorch-bot bot commented Jun 26, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/444

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 5ffe391 with merge base 0725f0b (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 26, 2024
scripts/hf_eval.py Outdated Show resolved Hide resolved
@msaroufim msaroufim self-requested a review June 26, 2024 19:13
@msaroufim msaroufim merged commit e5548b7 into main Jul 3, 2024
13 checks passed
@msaroufim msaroufim deleted the better-error-message branch July 4, 2024 00:38
dbyoung18 pushed a commit to dbyoung18/ao that referenced this pull request Jul 31, 2024
* Better error message

* change error message and make output prettier
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants