Add template postprocess module for faster_tokenizer

[joey12300]

PR types

New features

PR changes

Models

Description

Add template postprocessor for faster_tokenizer

Usage

# define a postprocessor 
tokenizer.postprocessor = postprocessors.TemplatePostProcessor(
    single=f"{cls_token}:0 $A:0 {sep_token}:0",
    pair=f"{cls_token}:0 $A:0 {sep_token}:0 $B:1 {sep_token}:1",
    special_tokens=[
        (cls_token, cls_token_id),
        (sep_token, sep_token_id),
    ],
)