Support Python 3.11

[potiuk]

Currently when you try to install sasl on Python 3.11, the compilation fails with:

#56 513.0   × Running setup.py install for sasl did not run successfully.
#56 513.0   │ exit code: 1
#56 513.0   ╰─> [31 lines of output]
#56 513.0       running install
#56 513.0       /usr/local/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
#56 513.0         warnings.warn(
#56 513.0       running build
#56 513.0       running build_py
#56 513.0       creating build
#56 513.0       creating build/lib.linux-x86_64-cpython-311
#56 513.0       creating build/lib.linux-x86_64-cpython-311/sasl
#56 513.0       copying sasl/__init__.py -> build/lib.linux-x86_64-cpython-311/sasl
#56 513.0       running egg_info
#56 513.0       writing sasl.egg-info/PKG-INFO
#56 513.0       writing dependency_links to sasl.egg-info/dependency_links.txt
#56 513.0       writing requirements to sasl.egg-info/requires.txt
#56 513.0       writing top-level names to sasl.egg-info/top_level.txt
#56 513.0       reading manifest file 'sasl.egg-info/SOURCES.txt'
#56 513.0       reading manifest template 'MANIFEST.in'
#56 513.0       adding license file 'LICENSE.txt'
#56 513.0       writing manifest file 'sasl.egg-info/SOURCES.txt'
#56 513.0       copying sasl/saslwrapper.cpp -> build/lib.linux-x86_64-cpython-311/sasl
#56 513.0       copying sasl/saslwrapper.h -> build/lib.linux-x86_64-cpython-311/sasl
#56 513.0       copying sasl/saslwrapper.pyx -> build/lib.linux-x86_64-cpython-311/sasl
#56 513.0       running build_ext
#56 513.0       building 'sasl.saslwrapper' extension
#56 513.0       creating build/temp.linux-x86_64-cpython-311
#56 513.0       creating build/temp.linux-x86_64-cpython-311/sasl
#56 513.0       gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Isasl -I/usr/local/include/python3.11 -c sasl/saslwrapper.cpp -o build/temp.linux-x86_64-cpython-311/sasl/saslwrapper.o
#56 513.0       sasl/saslwrapper.cpp:196:12: fatal error: longintrepr.h: No such file or directory
#56 513.0         196 |   #include "longintrepr.h"
#56 513.0             |            ^~~~~~~~~~~~~~~
#56 513.0       compilation terminated.
#56 513.0       error: command '/usr/bin/gcc' failed with exit code 1
#56 513.0       [end of output]
#56 513.0   

I know it is eaarly (Python 3.11 has just been released yesterday) but we are hoping in Apache Airflow to a much faster cycle of adding new Python releases - especially that Pyhon 3.11 introduces huge performance improvements (25% is the average number claimed) due to a very focused effort to increase single-threaded Python performance (Specialized interpreter being the core of it but also many other improvements) without actually changing any of the Python code.

I'd appreciate if someone in the cloudera team attempted to fix it. Otherwise we might want to simply skip hive provider from Python 3.11 compatible version of Airflow.

I just opened a PR in Apache Airflow yesterday and plan to keep it open until it gets green :). So far I have to exclude hive provider.

apache/airflow#27264

I think it would be fantastic if we could as the open source community migrate to the new Python much faster.

Looking forward to cooperation on that one :)

[gughy8]

@potiuk: I added a #31 for python3.11.

[MarechJ]

bumping this up. I would review the MR myself but unfortunately it's a bit beyond my depth.

[potiuk]

I hope someone from cloudera (@attilajeges ? ) might actually review and merge the PR #31 and release python-sasl. I am forced to remove hive provider because of this problem from our Python 3.11 releases.

[Hansz]

Bump ... running into the same problem with "longintrepr.h" ... Would appreciate this fix.

[mdeshmu]

Ping, Please look into PR 31

[guo-steve]

I am installing localstack[runtime] and it failed because of this error, not sure whether there is a workaround?

[potiuk]

Just a bit of a warning here for Python-sasl maintainers. We are very close to have Python 3.11 support in Apache Airflow - we are just about to merge the Google Provider upgrading ~20 client libraries which was the biggest blocker, and ApacheBeam released 2.47.0 version for Python 3.11. Similarly as it happened alredy with yandex provider - we are going to suspend the hive provider from our releases if Papermill will be holding us back.

We have a process for that described in our process: https://github.com/apache/airflow/blob/main/PROVIDERS.rst#suspending-releases-for-providers - you can also learn there what are the consequences of being suspended (in short - no new releases of the provider until the problem is removed, papermill will be removed from "airflow" extras in the next minor relese of Airflow).

The first step of the process is to let the maintainers of the provider library that holds us back, which is happending via this comment.

Apparently there is a PR already in your repo to prevent it and add 3.11 support, and there is about a week to before we attempt to merge 3.11 charge with suspended the Apache Hive provider so in case you would like to avoid the suspension, there is a about a week to get the python-sasl release that will support Python 3.11.

[potiuk]

Also announced in airflow devlist https://lists.apache.org/thread/0dcvjj0f6bnjg3mk4zn32stjbxtprb5j so in case you have something to add, comment etc - feel free.

[potiuk]

FYI . We just merged 3.11 support. The Apache.Hive provider has been excluded from Python 3.11 until this issue is fixed.

[mdeshmu]

@potiuk
Does Airflow Hive provider use PyHive?
FYI, I have raised a PR for PyHive to support another working library pure-sasl.

[eladkal]

Does Airflow Hive provider use PyHive?

Yes but Pyhive is discontinued
https://github.com/dropbox/PyHive#project-is-currently-unsupported

We likely drop hive integration eventually as there is no other Python SDK for it

[mdeshmu]

@potiuk @eladkal

I made a couple of contributions to PyHive which were accepted and released in 0.7.1.dev0. You are requested to test with the dev version and report any bugs in PyHive github repository before 0.7.1 is released in a month or so.

1. PyHive also supports pure-sasl via additional extras 'pyhive[hive_pure_sasl]' which supports Python 3.11 in addition to previous Python versions. See Use pure-sasl in python 3.11 dropbox/PyHive#454
2. PyHive is now compatible with SQLAlchemy 2.0. See Adding compatibility with SQLAlchemy 2.0 dropbox/PyHive#457

[potiuk]

Fantastic! Thanks ! That will be great to have it released and add Hive back to be supported in 3.11
I just run created a draft to test it -> PR apache/airflow#32607

If all goes right, this will both, test installability, cross-dependencies of PyHive and pip resolvabilty on 3.11 as well as run unit tests for it, which should give us good confidence that it works.

Fingers crossed it wil be green

Re:

• PyHive also supports pure-sasl via additional extras 'pyhive[hive_pure_sasl]' which supports Python 3.11 in addition to previous Python versions. See Use pure-sasl in python 3.11 dropbox/PyHive#454

is pure-sasl workig with whole range Py 3.8 - 3.11 ? Because if so, I would gladly switch to pure-sasl instead for Airflow we already use pure-sasl so . I will run another PR to check it after. We had our dose of problems with sasl (see below comment):

in case of Python 3.9 sasl library needs to be installed with version higher or equal than
0.3.1 because only that version supports Python 3.9. For other Python version pyhive[hive] pulls
the sasl library anyway (and there sasl library version is not relevant)

• sasl>=0.3.1; python_version>="3.9"

• PyHive is now compatible with SQLAlchemy 2.0. See Adding compatibility with SQLAlchemy 2.0 dropbox/PyHive#457

That's a good timing. We are completling rewriting all our queries to SQLAlchemy 2.0 too, so having Hive supporting it too, would remove a blocker for our migration.

[mdeshmu]

@potiuk
Yes, pure-sasl officially supports Py 3.8 - 3.10.
I have personally tested with Py 3.9 - 3.11 and it works.

[potiuk]

Cool. Updated the PR with switching to pure-sasl as well. Let's see :)

[eladkal]

@potiuk @eladkal

I made a couple of contributions to PyHive which were accepted and released in 0.7.1.dev0.

This is awesome!
And surprising giving Pyhive mention in thier repo that the project is discontinued
https://github.com/dropbox/PyHive#project-is-currently-unsupported

[potiuk]

Looks like we are close enough - we had some intermittent issues on apache/airflow#32607 - re-run it to see but images for 3.11 were built, and tests passed, so looks like things are good from Airlflow side (I re-run the failing jobs tests to verify).

[mdeshmu]

Awesome! Thanks for testing @potiuk.

[potiuk]

Yep. it looks good.

The constraint job is failing because we have dev dependency and == there, once the package is released and we switch it to regular >=, it will start working.

[potiuk]

Awesome! Thanks for testing @potiuk.

Are there any expectations on 0.7.1 to be released :) ? (A month or so was 3 weeks ago so, just checking what to expect :) )

[mdeshmu]

Are there any expectations on 0.7.1 to be released :) ? (A month or so was 3 weeks ago so, just checking what to expect :) )

The PyHive repo maintainer with whom I am co-ordinating, is on vacation. He will release 0.7.1 once he is back from vacation. It will take a couple of weeks more.

[potiuk]

OK. Cool. Thanks. We just release new version of the Hive provider in two days, so I was just checking :)

[mdeshmu]

PyHive 0.7.0 is released which includes Python 3.11 and SQLAlchemy 2.0 support.

[potiuk]

Oh... Fantastic. Thank you for letting me know. I kept on checking from time to time but being notified is even cooler :)