Token-Tail

A CLI tool for processing STDIN or a file with the tiktoken tokenizer.

Options:
  --help  Show this message and exit.

Commands:
  count  Prints the number of tokens in the input.
  dump   Dumps the tokens and their text representations.
  head   Prints the first N tokens from the input.
  split  Splits the input text into multiple files.
  tail   Prints the last N tokens from the input.

Installation

pip install git+https://github.com/urbandictionary/token-tail