v2.0.5 Bug fixes and safety checks

Natooz released this 04 May 16:32

· 216 commits to main since this release

Changes

f9f63d0 (related to #37) adding a compatibility check to learn_bpe method
f1af66a fixing an issue when loading tokens in learn_bpe with unique_track compatible tokenizer (REMIPlus) causing no BPE learning
f1af66a in learn_bpe: checking that the total number of unique base tokens (chars) is inferior to the target vocabulary size
47b6166 handling multi-voc indexing with tokens present in all vocabs eg special

Compatibility

All good 🙌

Assets 2