cchardet requirement, Python 3.11 compat

[pansen]

Hi,

first: thank you for your great library. I much enjoy using that :)

With the release of Python 3.11 I was checking my projects upgrade capabilities and it turns out, that cchardet is a difficult candidate. There is already an open issue for Python 3.10 (how is this working for Python 3.10 then anyway ...) and the project seems not maintained anymore.

In the sourcecode of cryptofeed I was looking for places where cchardet is actually used, as I wanted to see if it can be replaced with charset-normalizer (via SO) but couldn't find the location.

So my questions are:

• is cchardet used
• if so, where (maybe I can put some work in it to replace it)

Best, Andi

[pansen]

Stupid question, sorry. I was using ripgrep and not grep, which skipped the relevant files.

[peedrr]

@pansen - Notwithstanding your "stupid question" 😜 I think this issue should be re-opened.

Cryptofeed does not build out-of-the-box on 3.11.0 and it does indeed seems that cchardet is the culprit.

      src/cchardet/_cchardet.cpp:196:12: fatal error: 'longintrepr.h' file not found
        #include "longintrepr.h"
                 ^~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> cchardet

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Potential solutions:

1. Wait for fix on cchardet (NB @pansen's comment re. no longer maintained; latest release is > 2 years old)
2. Find a suitable replacement to cchardet

Regarding (2), there is a note in setup.py that cchardet is optional. Cryptofeed does indeed build (and run) when using the Python chardet. @bmoscon: thoughts on abandoning cchardet in favour of Python chardet? (In terms of speed gained using Python 3.11 plus speed lost by not using Cchardet?) Is there perhaps some clever way to conditionally install Python chardet if Python version ≥ 3.11 else Cchardet?

Temporary solutions:
I propose we Update README.md (and probably INSTALL.md as well) to state that "Cryptofeed requires Python ≥ 3.8 < 3.11", or give explicit instructions how to opt for Python chardet instead of Cchardet.

[bmoscon]

I was able to get it working with 3.11, what error do you get?

[peedrr]

I pasted the error above

[peedrr]

My setup was Python 3.11 within Conda env, running on Mac. It wouldn't build from either source or pip

[bmoscon]

resolved with latest commit