Skip to content
This repository has been archived by the owner on Jan 15, 2024. It is now read-only.

[Bug] Bug of top-p sampling #1481

Open
sxjscience opened this issue Jan 15, 2021 · 0 comments
Open

[Bug] Bug of top-p sampling #1481

sxjscience opened this issue Jan 15, 2021 · 0 comments
Labels
bug Something isn't working

Comments

@sxjscience
Copy link
Member

Description

The top-p sampling here is not correct.

probs = mx.np.where(
probs > self._sampling_topp,
probs,
mx.np.zeros_like(probs)
)
We should sort the probabilities in descending order and then do a cumsum of the probabilities. After that, we set the threshold based on the cumsum values.

@hymzoque

@sxjscience sxjscience added the bug Something isn't working label Jan 15, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant