fix: Switch from tiktoken to js-tiktoken for worker compatibility

[antpb]

Relates to:

Eliza currently uses tiktoken which relies on WASM bindings. This causes build failures in Cloudflare Workers and other edge runtime environments where WASM support is limited or unavailable. This PR migrates the token truncation functionality from tiktoken to js-tiktoken, which is a pure JavaScript implementation of the same tokenization logic. This change removes the WASM dependency while maintaining the same tokenization functionality.

Risks

Seems the collaborator on both packages are the same person so this should be a trusted package. (verify pls)
https://www.npmjs.com/package/js-tiktoken
https://www.npmjs.com/package/tiktoken

Background

What does this PR do?

This PR migrates the token truncation functionality from tiktoken to js-tiktoken, which is a pure JavaScript implementation of the same tokenization logic. This change removes the WASM dependency while maintaining the same tokenization functionality.

What kind of change is this?

Pretty beefy package change and maybe opinionated but id be willing to defend the change if anyone objects. :)

Key changes:

• Replace tiktoken import with js-tiktoken
• Remove unnecessary TextDecoder usage since js-tiktoken.decode() returns strings directly
• Remove encoding.free() call as JavaScript handles memory management
• Update types to match js-tiktoken interfaces

Why are we doing this? Any context or related work?

Edge compat is gud.

• Enables deployment to edge runtimes like Cloudflare Workers
• Maintains same tokenization functionality and accuracy
• Reduces bundle size by removing WASM dependency
• Simplifies deployment by removing need for WASM file handling

Documentation changes needed?

Testing

Where should a reviewer start?

Take a look at the included test file for instructions on how to use it.

The automated tests have been done to ensure it:

• Correctly truncates text to specified token limits
• Handles edge cases (empty strings, invalid inputs)
• Maintains proper error handling
• Works in worker environments

Discord username

antpb