ci(ingest): test with python 3.11

[hsheth2]

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[github-actions]

Unit Test Results (build & test)

621 tests  ±0   617 ✔️ ±0   15m 53s ⏱️ +10s
157 suites ±0       4 💤 ±0 
157 files   ±0       0 ❌ ±0 

Results for commit 8cd620c. ± Comparison against base commit b7c0373.

[github-actions]

Unit Test Results (metadata ingestion)

    4 files   -     4      4 suites   - 4   28m 6s ⏱️ - 29m 36s
759 tests ±    0  755 ✔️  -     1  4 💤 +1  0 ❌ ±0 
760 runs   - 760  756 ✔️  - 757  4 💤  - 3  0 ❌ ±0 

Results for commit 8cd620c. ± Comparison against base commit b7c0373.

This pull request skips 1 test.

tests.integration.feast.test_feast_repository ‑ test_feast_repository_ingest

[hsheth2]

Currently blocked on googleapis/python-bigquery-sqlalchemy#500.

Now blocked on googleapis/python-bigquery-sqlalchemy#543

[anshbansal]

LGTM once CI is green

[anshbansal]

Looks all tests are failing now.

[hsheth2]

Looks like we're somehow pinning pyarrow to an old version, which doesn't have a pre-built binary for python 3.11. Ideally we should loosen our deps, but we can also add the necessary requirements so that it can build from source for 3.11 instead.

[hsheth2]

Blocked because of feast-dev/feast#3510

[hsheth2]

No longer blocked on feast.

Now we're blocked on cloudera/python-sasl#30 (we actually use sasl3 and not sasl but they face the same issue). We only depend on sasl3 from acryl-pyhive, so we probably could upgrade it to use pure-sasl instead. There seems to be an effort around that already dropbox/PyHive#454.

[hsheth2]

Just need to merge acryldata/PyHive#7 and update here, and then we should be good to go.

[mdeshmu]

@hsheth2

I made a couple of contributions to PyHive which were accepted and released in 0.7.1.dev0. You are requested to test with the dev version and report any bugs in the PyHive GitHub repository before 0.7.1 is released in a month or so.

1. PyHive also supports pure-sasl via additional extras 'pyhive[hive_pure_sasl]' which supports Python 3.11 in addition to previous Python versions. See Use pure-sasl in python 3.11 dropbox/PyHive#454
2. PyHive is now compatible with SQLAlchemy 2.0. See Adding compatibility with SQLAlchemy 2.0 dropbox/PyHive#457

[hsheth2]

Looks like there's two errors now:

1. PowerBI using a mutable type for a dataclass default
2. Something related to typing (with snowflake?), similar to types.CodeType regression in Python 3.11 python/cpython#100316

[hsheth2]

• The powerbi issue will be fixed by fix(ingest): avoid mutable defaults in powerbi dataclass #8609
• Looks like the CodeType error is from pyspark and fixed in recent versions, so we just need to upgrade: blocked on chore(ingest/s3) Bump Deequ and Pyspark version  #8638
• There's also an issue with the ratelimiter package - we'll need to find an alternative because the project looks dead: needs update for python3.11: module 'asyncio' has no attribute 'coroutine' RazerM/ratelimiter#18

[hsheth2]

Unblocked now that #9008 was merged

[hsheth2]

One more thing here: To be compatible with Python 3.11, we need to be on pyspark 3.4+ (see apache/spark#38987)

However, pydeequ still depends on pyspark 3.3 right now (awslabs/python-deequ#168). The underlying deequ issue was recently fixed (awslabs/deequ#505), so pydeequ will hopefully get updated quickly.

[hsheth2]

Moving the StrEnum changes to #11270

[hsheth2]

Closing in favor of #11280