TypeError: bpe(...).split is not a function

[bramses]

if token == toString the encoder errors out with this error.

https://github.com/latitudegames/GPT-3-Encoder/blob/master/Encoder.js#L163

Suggested fix: Cast token to string:

(toString).split(' ')
VM236:1 Uncaught TypeError: toString.split is not a function
    at <anonymous>:1:12

('toString').split(' ')
['toString']

EDIT:

same issue occured with constructor. The encoder may struggle with all reserved JS keywords :/

[NickHeiner]

I have the same issue with this:

var gpt3Encoder = require("gpt-3-encoder")

gpt3Encoder.encode(`foo.valueOf()`);

With some console logging, I can see that the bpe function returns [Function: valueOf] rather than a string.

[NickHeiner]

This is fixed in my fork: https://www.npmjs.com/package/@nick.heiner/gpt-3-encoder.

[jdan]

Fixed with an upgrade to 1.1.4