Update Python dependencies to eliminate version conflicts

[justinclift]

What type of PR is this?

• Other

Description

This PR gets all of the Python backend dependencies to work together without version mismatch errors during their pip install.

How is this tested?

• Manually

Only tested using the basic "docker build" step, as it's a work in progress. It'll be useful to see if the CI tests are happy with it though, especially after the dql code is gone (not yet done).

[justinclift]

Error: No such command 'list_types'.

That'll probably go away after I rip out the dql code. 😄

---

Nope, it wasn't dql directly causing that. Got it fixed anyway though.

[justinclift]

k, that looks like an initial minimum set of updates to get the dependencies working together without problems.

Next step is to clean it up, and see if any of the temporarily removed dependencies can be re-added. Firebolt is pretty clearly out (for now) though, due to direct incompatibility with other things.

[justinclift]

@wlach Would you be ok to cast an eye over this review-wise to ensure it seems sane to you?

[justinclift]

Ugh. Hold off on this. I've just done a final test of the dependencies using a different process, and some further conflicts are showing up:

google-auth 2.21.0 requires urllib3<2.0, but you'll have urllib3 2.0.3 which is incompatible.
botocore 1.13.50 requires urllib3<1.26,>=1.20; python_version >= "3.4", but you'll have urllib3 2.0.3 which is incompatible.
snowflake-connector-python 3.0.4 requires urllib3<1.27,>=1.21.1, but you'll have urllib3 2.0.3 which is incompatible.
nzalchemy 11.0.2 requires SQLAlchemy<=1.3.24, but you'll have sqlalchemy 2.0.17 which is incompatible.

Dammit. k, will fix them...

---

It turns out those are false warnings. They're just due to the different process not actually looking at all the requirements*.txt files properly. So, this is fine to review and potentially merge after all. 😄

[justinclift]

@getredash/maintainers Anyone up for reviewing this PR?

It only touches the Python backend part of things though, and doesn't have any front end changes.

[junnplus]

LGTM.

[justinclift]

Thanks heaps @junnplus. 😄

[justinclift]

Now that we've largely updated our Python dependencies, we could probably check if that Firebolt code we removed in f7aa5b4 could be added back.

[masayuki038]

@justinclift I noticed that redash/query_runner/dynamodb_sql.py and some query runners were removed in this PR. However, the issue regarding DynamoDB was added this week 😅

Will these query_runners be unsupported from the next release? Or are there any plans to revive them?

[justinclift]

Will these query_runners be unsupported from the next release?

From my perspective, I reckon it depends on what state their Python libaries are in.

Looking at the dynamodb library info on PyPi just now... its one and only release was over three years ago:

https://pypi.org/project/dynamodb/#history

The source code repository shows the same lack of continued development:

https://github.com/AllanKT/python-dynamo

That GitHub repository also shows no forks at all. 😦

So, for that one specific Python library I'd personally be ok with not including it with the next release. Which of course is going to be lousy for anyone currently using it. ☹️

---

There are several of other results when searching for "dynamodb" on PyPi:

https://pypi.org/search/?q=dynamodb

So, maybe it'll be possible to switch to a different Python library instead without too much effort?

It'd need someone to investigate things a bit better, and potentially do some experimenting / updating of the old DynamoDB query runner code to get it working again.

[masayuki038]

@justinclift Thank you for your reply.

Indeed, this python-dynamo has not been updated. In addition, Python version is only listed up to 3.7, so it is certainly not a good idea to keep using this library.

Is the fundamental problem dependency hell? I'm concerned that Redash is loading libraries for all datasources. I think it is very hard to keep dependencies. Introducing new datasources can also be difficult. Are there any plans to address this issue?

[justinclift]

Are there any plans to address this issue?

Some rough ideas for fixing the problem have been talked about before, but we've not really investigated and committed to a clearly defined plan yet.