Skip to content

Commit

Permalink
don't strip the prompt for check since we don't strip to tokenize any…
Browse files Browse the repository at this point in the history
…more (#650)
  • Loading branch information
winglian authored Sep 28, 2023
1 parent b2edaae commit 8662e8f
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/axolotl/prompt_tokenizers.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ def _tokenize(
self, prompt: str, add_eos_token: bool = True, strip_bos_token: bool = False
) -> BatchEncoding:
result: BatchEncoding
if not prompt.strip():
if not prompt:
LOG.warning("Empty text requested for tokenization.")
result = BatchEncoding(data={"input_ids": [], "attention_mask": []})
else:
Expand Down

0 comments on commit 8662e8f

Please sign in to comment.