Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exclude Top Choices (XTC): A sampler that boosts creativity, breaks writing clichés, and inhibits non-verbatim repetition #6335

Merged
merged 7 commits into from
Sep 28, 2024
+78 −3
Merged

Exclude Top Choices (XTC): A sampler that boosts creativity, breaks writing clichés, and inhibits non-verbatim repetition #6335

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f1232b1
Add Exclude Top Choices (XTC) sampler
p-e-w Aug 18, 2024
5176eaa
Change maximum of `xtc_threshold` slider to 0.5
p-e-w Aug 20, 2024
37680db
Merge branch 'dev' into p-e-w-xtc
oobabooga Aug 21, 2024
af7b57c
Update the descriptions
oobabooga Aug 21, 2024
0f62744
Check for EOS and \n
oobabooga Sep 3, 2024
15daf36
Merge branch 'dev' into p-e-w-xtc
oobabooga Sep 28, 2024
29d38a7
Add missing :
oobabooga Sep 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions modules/sampler_hijack.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,12 @@ def __init__(self, threshold: float, probability: float, filter_value: float = -
self.threshold = threshold
self.probability = probability
self.filter_value = filter_value
self.special_token_ids = [
shared.tokenizer.encode("\n")[-1],
]

if shared.tokenizer.eos_token_id is not None:
self.special_token_ids.append(shared.tokenizer.eos_token_id)

def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
# `random` returns values in the half-open range [0, 1), so setting `probability`
Expand All @@ -221,7 +227,14 @@ def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> to
# of all tokens that meet the threshold, *except* the least probable one.
sorted_indices_to_remove[..., :-1] = probs[..., 1:] >= self.threshold

# Convert sorted_indices_to_remove to the original indices
indices_to_remove = sorted_indices_to_remove.scatter(1, sorted_indices, sorted_indices_to_remove)

# If newline or EOS tokens would be removed, return the original scores
if indices_to_remove[:, self.special_token_ids].any()
return scores

# Otherwise, remove tokens with the mask
scores = scores.masked_fill(indices_to_remove, self.filter_value)
return scores

Expand Down