Switch from 'cloudant' to 'ibmcloudant'

[uranusjr]

Apache Airflow Provider(s)

cloudant

Apache Airflow version

main (development)

What happened

The IBM Cloudant API client we’re currently using, cloudant (aka python-cloudant), reached EOL last December. The authors recommend switching to ibmcloudant. The switch itself doesn’t seem too complicated (disclaimer: I have zero knowledge on IBM Cloudant), but ibmcloudant is depending on PyJWT<3.0.0,>=2.0.1, which conflicts to Flask-Appbuilder’s PyJWT<2.0.0,>=1.7.1. So we need to unblock either pacakge somehow to make this work.

See also dpgaspar/Flask-AppBuilder#1691

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[pritampbonde]

Could you please let us know when can this issue be resloved? In which release of airflow

[uranusjr]

You need to ask Flask-AppBuilder. They cause the conflict, and if they don’t resolve it, Airflow cannot fix the issue.

[vincentkoc]

dpgaspar/Flask-AppBuilder#1691 Seems to be resolved now

[potiuk]

Yes, We need to upgrade FAB to 4.0.0 - but that will take time and effort #23899

[eladkal]

Now that #24399 is merged I believe this task is unblocked :)
We will need however to set the provider with min version of Airflow 2.4.0

[potiuk]

Yep. fully justified. Marked it as good first issue.

I think however, that we might want to decide to release FAB 4.1.* upgrade in 2.3.3 so this might be 2.3.3+

The FAB 4.1.1* was surprisingly smooth so far and did not bring any really breaking changes. Bumping required dependencies is NOT a breaking change on its own even if we are bumping major versions of Flask/PyJWt - (especially that they bring a lot of security-related upgrade). And the upgrade itself did not introduce any incompatibilities on the "airflow" API or usage side.

I am going to discuss it in the #releasemanagement channel on our slack

[potiuk]

Yeah FAB 4.1.2 is relased in 2.3.3. so the min-version should be >= 2.3.3 when released.

[topherinternational]

@uranusjr @potiuk I took a look at this as a good-first-issue, and I think there's a PMC-level question to be answered before I want to start writing any code.

The CloudantHook returns a cloudant object from the now-discontinued cloudant library, a context manager object that yields a logged-in client session object (of type Cloudant, with a capital C).

ibmcloudant is the replacement for cloudant, but it discontinues the context manager patterns, and its client object also has a different set of functions that can be called against the Cloudant server.

The mapping of cloudant calls to ibmcloudant calls can be found here: https://github.com/cloudant/python-cloudant/blob/master/MIGRATION.md#reference-table; you can see that calls have changed names and been moved across objects/interfaces.

So my question is, if cloudant were swapped out for ibmcloudant, how should we handle the breaking change? Is it enough to bump the cloudant provider version to 4.x.x, or does this need to break across Airflow 2->3 or 3->4?

Thanks for any insight you can provide, It's hard to proceed intelligently not knowing the provider breaking change policy nor understanding at all how users might be using this hook.

[eladkal]

So my question is, if cloudant were swapped out for ibmcloudant, how should we handle the breaking change? Is it enough to bump the cloudant provider version to 4.x.x, or does this need to break across Airflow 2->3 or 3->4?

Its not related to Airflow version.
We simply introduce version 4 of the cloudant provider. The provider is compatible with Airflow 2 and will be compatible with Airflow 3.

[topherinternational]

As per #41555 (comment), my PR for this issue is blocked on Snowflake upgrading its urllib3 dependency, so we're back to waiting for a third-party change 😬 .