feat: add o1 support

[monilpat]

Relates to:

#1185

Risks

Low.

• Potential impacts include:
  • Compatibility with existing functionality in js-tiktoken.
  • Possible edge cases in handling new O1 models.

Background

What does this PR do?

This PR updates the js-tiktoken package to include support for the o1 model. It modifies the tokenizer logic to handle the specific tokenization patterns and configurations required for the o1 model.

What kind of change is this?

• Features (non-breaking change which adds functionality)

Why are we doing this? Any context or related work?

Including support for the o1 model aligns with expanding the capabilities of js-tiktoken to handle the latest OpenAI models. This ensures broader usability and aligns with ongoing support for new model releases.

Documentation changes needed?

• My changes require a change to the project documentation.
• Updated README to include details on o1 model support.
• Added an entry to the changelog to document the feature addition.

Testing

Where should a reviewer start?

Begin by reviewing changes in the tokenizer logic in src/tokenizer.js and the associated test cases in tests/tokenizer.test.js.

Detailed testing steps

1. Clone the repository and check out the branch containing this PR.
2. Run npm install to ensure dependencies are up to date.
3. Execute tests with npm test and verify that all test cases pass.
4. Manually test by tokenizing examples specific to the o1 model and comparing output against expected results.

Example manual test:

const { encode } = require('js-tiktoken');
const model = 'o1';
const input = 'Hello, this is a test.';
console.log(encode(input, model));