Invalid use of free in tokenizer.cpp

[fairydreaming]

In tokenizer.cpp you have: char* str_buffer = new char[str_buffer_size];, but later there is: free(str_buffer). You should replace it with delete[] str_buffer.

There is also probindex = new ProbIndex[vocab_size]; freed with free(probindex);. You should replace it with delete[] probindex.

[b4rtaz]

@fairydreaming Thanks for the reporting a problem. I fixed this in this PR.