Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trace output distributions to a log file #246

Closed
wants to merge 5 commits into from
Closed

Trace output distributions to a log file #246

wants to merge 5 commits into from

Conversation

Piezoid
Copy link
Contributor

@Piezoid Piezoid commented Mar 17, 2023

I do not expect this to be merged, but I figured it might help others. Although, I don't know if this is the right place.

This logs information to a ./out.log (hard-coded) file. I wrote this throwaway code before the switch to stderr, which is why it uses a global file handle.
The refactoring of the sampler code should produce the same results than the master branch.

For each predicted token, it logs:

in:' because' n_past=14, remaining_tokens=62, embd.size()=1, embd_inp.size()=13
soft_max: top_sact=25.503617 mean_sact=19.826111 top_p=0.357196 entropy=1.664120
top_p: n: 15 sum: 0.990421
->0: ' they' p=0.357196 act=17.853 temp=0.70
  1: ' I' p=0.231013 act=20.643 temp=0.82
  2: ' of' p=0.228527 act=17.540 temp=0.70
[...]
  15: ' the' p=0.000876 act=13.645 temp=0.70
  • The soft_max: lines reports statistics of the top k tokens' logits (divided by temp) and entropy (in nats, not bits),
  • top_p: line gives the number of retained tokens after the top p filtering, and the sum of their probabilities,
  • Last, a list of the top 16 tokens, along with their respective probabilities, original logits, and the product of their temperature and eventual repetition penalty.. The drawn token is indicated by an ->.

I will close this either when it becomes obsolete or when it can no longer be rebased.

@gjmulder gjmulder added the enhancement New feature or request label Mar 20, 2023
@Piezoid Piezoid closed this Mar 24, 2023
AAbushady pushed a commit to AAbushady/llama.cpp that referenced this pull request Jan 27, 2024
* add tokens per second output

* Update gpttype_adapter.cpp

simplify

---------

Co-authored-by: LostRuins <39025047+LostRuins@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants