Skip to content

refactor(tiktoken): using openai/tiktoken's #331 approach #603

@illusory0x0

Description

@illusory0x0

openai/tiktoken#331's approach is more simple and elegant.

This PR drops a lookahead part of the pattern, the part for catching the whitespaces, and handles the whitespaces with scripting instead, with mathematically provable exactly same output.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions