You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Not sure if this already covered, but it appears LitGPTTokenizer._tokenize_do_it also needs to be updated in order to report a 'tokens' key in the dictionary. Specifically, CachingTokenizer calls that method here and the in the next call expects the response dictionary to have an 'tokens' key here
As an alternative, it looks like LitGPTTokenizer could override _tokenization_raw_response_to_tokens.
The text was updated successfully, but these errors were encountered: