Integrate polars in get_df, get_df_by_chunks

[guan404ming]

Related Issue and PR

closes: #33911

• Add get_polars_df() method to BigQueryHook. #33911
• Feature add get polars df to dbapihook and bigqueryhook #34679
  cc: @eladkal @Taragolis @bfeif

Why

This PR addresses the ongoing discussion regarding the introduction of a generic get_df function, which would abstract the retrieval of dataframes from various sources, supporting both pandas and polars. The goal is to provide a more user-friendly and consistent API.

As discussed, there are arguments against a direct implementation, primarily concerning:

• Complexity and Maintenance: Concerns that a single implementation could become overly complex, especially considering the syntactic differences between pandas and polars querying APIs.
• Potential Breaking Changes: Concerns about future breaking changes if existing methods were removed.
• Current Pandas Issues: Existing issues with the pandas implementation should be addressed first.
• Over-engineering: The feeling that this change is an over-engineering.

There is also support for a more abstract approach:

• Abstraction and Simplification: A generic get_df could simplify user interaction and provide a consistent interface.
• Gradual Transition: Deprecation warnings can be used to facilitate a smooth transition, allowing users time to adapt.

How

This PR implements get_df while maintaining a flexible approach to address the community's concerns.

• get_df is introduced as a new, generic function that accepts a parameter specifying the desired dataframe type (pandas or polars).
• This implementation introduces the _get_polars_df and _get_pandas_df methods that are used within the get_df method. This allows the new generic method to function, while allowing the option to expose or deprecate the underlying methods in the future.
• Adding deprecation for original get_pandas_df
• This approach allows for a gradual transition and addresses the concerns about immediate breaking changes.
• The implementation is designed to be easily modified based on community feedback.

This PR aims to provide a flexible solution that can be adapted based on community consensus. The implementation can be refined or reverted as needed, ensuring that the changes align with the project's long-term goals.

Note

This PR is considered a starting point for discussion and may require further triage by the community. I plan to implement the corresponding changes within the Google provider after this PR is reviewed and a consensus is reached, as it is dependent on the outcome of this discussion. This separation allows for focused review and iteration on the core functionality before extending it to specific providers.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[guan404ming]

@eladkal Hi, I have updated PR with your suggestion thanks for you reviewing~

[eladkal]

LGTM