Add a tokenizing visualizer utility

[shhlife]

It's very helpful to get an understanding of how text is tokenized. A visualizer would be super useful.

We have a method on the OpenAiTokenizer that can count tokens, but nothing that visualizes how they look to the user.

Here's a script that can do that:

import tiktoken
from rich.console import Console
from rich.text import Text


def visualize_tokens(text, model="gpt-3.5-turbo"):
    """
    Tokenize the input text and display the tokens with unique colors using the rich library.

    Args:
        text (str): The input text to tokenize.
        model (str): The name of the OpenAI model (default: gpt-3.5-turbo).
    """
    # Load the tokenizer for the specified model
    encoding = tiktoken.encoding_for_model(model)

    # Encode the text into tokens
    tokens = encoding.encode(text)

    # Decode tokens back to text for clarity
    token_texts = [encoding.decode([token]) for token in tokens]

    # Create a Rich console instance
    console = Console()

    # Create a Text object to hold the colored tokens
    rich_text = Text()

    # Define a pool of colors to choose from
    colors = [
        "light_sea_green",
        "deep_sky_blue1",
        "slate_blue3",
        "pale_turquoise4",
        "grey53",
        "medium_purple",
        "chartreuse2",
        "medium_orchid3",
        "dark_khaki",
        "light_steel_blue",
    ]

    color_iterator = iter(colors)

    for token_text in token_texts:
        try:
            color = next(color_iterator)
        except StopIteration:
            # Restart the iterator when we reach the end of the color list
            color_iterator = iter(colors)
            color = next(color_iterator)

        rich_text.append(token_text, style=color)

    # Print the colored tokens
    console.print(rich_text)

To use it, you can do something like:

visualize_tokens("This is a sentence. I'll count to three: one, two, three.")

The result looks something like:

[collindutter]

Very cool! Griptape's Tokenizers don't have a detokenize method so I don't think this can live in the framework without some changes (that may not be possible).

Going to move this to show-and-tell so others can see what you've built.

[shhlife]

ah, so it would only work for certain tokenizers, like openAi?

[collindutter]

Correct, not all tokenizers support de-tokenization.

[shhlife]

gotcha!