Skip to content

Commit

Permalink
Revert "[docs] Fix wrong newline in the middle of a paragraph (huggin…
Browse files Browse the repository at this point in the history
…gface#6573)"

This reverts commit 1e5524a.
  • Loading branch information
fabiocapsouza authored Nov 15, 2020
1 parent d3f7763 commit c27e99f
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions docs/source/preprocessing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ The tokenizer can decode a list of token ids in a proper sentence:
As you can see, the tokenizer automatically added some special tokens that the model expect. Not all model need special
tokens; for instance, if we had used` gtp2-medium` instead of `bert-base-cased` to create our tokenizer, we would have

seen the same sentence as the original one here. You can disable this behavior (which is only advised if you have added
those special tokens yourself) by passing ``add_special_tokens=False``.

Expand Down

0 comments on commit c27e99f

Please sign in to comment.