Migrate HiveToDynamoDBOperator and SqlToS3Operator to use get_df

[guan404ming]

Related Issue

• Migrate get_pandas_df to get_df in providers #49334
  cc: @potiuk @eladkal

Why

get_pandas_df deprecated in #48875, which would be replaced by get_df. Thus, we need to migrate them in providers.

How

• This PR is focus on migration of Amazon/transfer
• Migrate unit test
• Refine the common-sql type

Note

There is some workaround for type here. It should be removed after the common-sql bundle the refined type definition. I've added the TODO in the comment and would get back here after the release.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[guan404ming]

Failed since the hive also need the overload like something I added in common-sql. I would open PR to solve it later.

[guan404ming]

Here is the PR #50211

[guan404ming]

Hi, I’ve tried to figure out why mypy passes locally but fails in CI: the CI pipeline seems not import (or install ?) the dependency (common-sql/sql.pyi) mypy needs for its checks. If you compare the local and CI error logs, you’ll see that with our new type overload the error disappears locally, but CI still reports it. I'm not sure that should we update the CI configuration to force import that file, or just add a # type: ignore here or is there any better way to solve this in Slack/transfer and this one?

Thanks in advance!

Local:

CI:

[guan404ming]

When I keep searching the answer I find the type def did not really following the newest python typing spec, thus I update them in #50229. I think maybe it could solve the error here since it didn't rely on the pyi file but to directly write the overload in the .py which I think it must be imported here.

[guan404ming]

Mypy seems passed and even the Hive related error is passed. It seems like related to the change here #50198 right?

[guan404ming]

Pushed again to ensure it is stable.

[guan404ming]

Mypy works great and push again for unrelated ci error

[guan404ming]

Thanks for the suggestions and review!

[github-actions]

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test