0.20.4 Partial release?

[tazarov]

Looking at the pypi release for 0.20.4 I can only see part of the wheels - https://pypi.org/project/tokenizers/0.20.4/#files

Seems this new release was pushed to pypi about an hour ago (26-Nov-2024 9am PT)

[kopyl]

Thank you, this led me to the solution for the problem of not being able to install transformers I've described here

[ArthurZucker]

In transformers we no longer "support" python 3.8, but yeah this does break a bit.
The release is just not out for python3.7 or 3.8 anymore

[ArthurZucker]

The partial wheel are there because of #1674

[ArthurZucker]

We are gonna yank the release, to have 0.21.0 instead, sorry !

[tazarov]

Thanks @ArthurZucker.

Just out of curiosity what was the reasoning for moving away from py38? EOL and/or other considerations? I see a fair bit of usage of tokenizers with py38 (https://pypistats.org/packages/tokenizers)

[ArthurZucker]

It has been deprecated and we want to make sure it still compatible with transformers which also removed official support for 3.8!
Also a little bit for convenience, pypi refuses to give use more storage so we had to remove all our RC to get space for the 110 wheels ...