Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement] reseed random number when loading from cache and --seed provided #1439

Closed
d-takemori opened this issue May 13, 2023 · 1 comment · Fixed by #1550
Closed

[enhancement] reseed random number when loading from cache and --seed provided #1439

d-takemori opened this issue May 13, 2023 · 1 comment · Fixed by #1550

Comments

@d-takemori
Copy link

Current Behavior

When loading from a saved --prompt-cache file, llama.cpp appears to initialize response generation with the same state that was saved with the cache-file. This yields the same output every run.

Initial run

% ./main --model ../../Models/LLaMA/7B/ggml-model-q5_1.bin --prompt-cache test-cachefile -n 36 --color -p 'What happens with a test prompt?'
main: build = 537 (6456a4e)
main: seed = 1684011989
[snip]
main: attempting to load saved session from 'test-cachefile'
main: session file does not exist, will create
[snip]
What happens with a test prompt?
A test prompt is an assignment that helps you find out whether something works.
This can be anything from finding out if your code does what it should do or if the data

next run

% ./main --model ../../Models/LLaMA/7B/ggml-model-q5_1.bin --prompt-cache test-cachefile -n 36 --color -p 'What happens with a test prompt?'
main: build = 537 (6456a4e)
main: seed = 1684012061
[snip]
main: attempting to load saved session from 'test-cachefile'
main: loaded a session with prompt size of 8 tokens
main: session file has exact match for prompt!
[snip]
What happens with a test prompt?
A test prompt is an assignment that helps you find out whether something works.
This can be anything from finding out if your code does what it should do or if the data

another run, but with explicit --seed

% ./main --model ../../Models/LLaMA/7B/ggml-model-q5_1.bin --prompt-cache test-cachefile -n 36 --color -p 'What happens with a test prompt?' --seed 1234
main: build = 537 (6456a4e)
main: seed = 1234
[snip]
main: attempting to load saved session from 'test-cachefile'
main: loaded a session with prompt size of 8 tokens
main: session file has exact match for prompt!
[snip]
What happens with a test prompt?
A test prompt is an assignment that helps you find out whether something works.
This can be anything from finding out if your code does what it should do or if the data

Desired Behavior

If loading from a cache-file AND "--seed" # is given, llama.cpp should initialize the random number generator before response generation, so that different responses are generated.

If loading from a cache-file and "--seed #" not given, preserve current behavior.

@aleksusklim
Copy link

aleksusklim commented May 24, 2023

I posted a comment about other strange behavior of prompt caching:
#1550 (comment)

UPD: reposted it as a separate issue: #1585

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants