Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Alibaba Cloud AnalyticDB Spark Support #31787

Merged
merged 9 commits into from
Jun 27, 2023
Merged

Add Alibaba Cloud AnalyticDB Spark Support #31787

merged 9 commits into from
Jun 27, 2023

Conversation

dcoliversun
Copy link
Contributor

This PR aims to add Alibaba Cloud AnalyticDB Spark1 support.

  1. implement custom operator/hook/sensor for AnalyticDB Spark.
  2. enable users to schedule AnalyticDB Spark Batch/SQL applications by DAGs.
  3. AnalyticDB Spark hooks provides log/Web UI url/application status for users.

closes: #31738

@dcoliversun dcoliversun requested review from kaxil and potiuk as code owners June 8, 2023 02:07
@boring-cyborg
Copy link

boring-cyborg bot commented Jun 8, 2023

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@dcoliversun dcoliversun marked this pull request as draft June 8, 2023 02:09
@dcoliversun dcoliversun marked this pull request as ready for review June 8, 2023 02:15
@dcoliversun
Copy link
Contributor Author

@potiuk It would be good if you have time to review this PR

@potiuk
Copy link
Member

potiuk commented Jun 19, 2023

I am less available over the next 2 days (at conference). But maybe others can take a look before @eladkal starts preparing new wave of providers ? Sounds like cool addition to the Alibaba provider.

@potiuk
Copy link
Member

potiuk commented Jun 19, 2023

(first glance - it looks rather complete :) and good quality one).

@utkarsharma2
Copy link
Contributor

@dcoliversun Thanks for the contribution :)

Just added a few code suggestions, please check.

sunank200
sunank200 reviewed Jun 19, 2023
View reviewed changes
Copy link
Collaborator

@sunank200 sunank200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution

airflow/providers/alibaba/cloud/hooks/analyticdb_spark.py Show resolved Hide resolved
airflow/providers/alibaba/cloud/sensors/analyticdb_spark.py Show resolved Hide resolved
eladkal
eladkal requested changes Jun 20, 2023
View reviewed changes
airflow/providers/alibaba/CHANGELOG.rst Outdated Show resolved Hide resolved
airflow/providers/alibaba/provider.yaml Outdated Show resolved Hide resolved
@dcoliversun
Add Alibaba Cloud AnalyticDB Spark Support
63d772a 
Apply suggestions from code review

Co-authored-by: Utkarsh Sharma <utkarsharma2@gmail.com>

replace Exception with ValueError

fix ci deps failure

add version

remove return in comment

fix indent error

address comments

address comments
eladkal
eladkal reviewed Jun 20, 2023
View reviewed changes
airflow/utils/db.py Outdated Show resolved Hide resolved
phanikumv
phanikumv reviewed Jun 20, 2023
View reviewed changes
airflow/providers/alibaba/cloud/hooks/analyticdb_spark.py Outdated Show resolved Hide resolved
airflow/providers/alibaba/cloud/hooks/analyticdb_spark.py Outdated Show resolved Hide resolved
airflow/providers/alibaba/cloud/operators/analyticdb_spark.py Outdated Show resolved Hide resolved
tests/providers/alibaba/cloud/hooks/test_analyticdb_spark.py Show resolved Hide resolved
tests/providers/alibaba/cloud/hooks/test_analyticdb_spark.py Show resolved Hide resolved
tests/providers/alibaba/cloud/hooks/test_analyticdb_spark.py Show resolved Hide resolved
tests/providers/alibaba/cloud/hooks/test_analyticdb_spark.py Show resolved Hide resolved
tests/providers/alibaba/cloud/hooks/test_analyticdb_spark.py Show resolved Hide resolved
tests/providers/alibaba/cloud/hooks/test_analyticdb_spark.py Show resolved Hide resolved
tests/providers/alibaba/cloud/hooks/test_analyticdb_spark.py Show resolved Hide resolved
@eladkal eladkal mentioned this pull request Jun 21, 2023
Open
1 task
@potiuk potiuk requested a review from eladkal June 23, 2023 15:03
@eladkal eladkal merged commit f05beb7 into apache:main Jun 27, 2023
@boring-cyborg
Copy link

boring-cyborg bot commented Jun 27, 2023

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@dcoliversun dcoliversun deleted the AIRFLOW-31738 branch June 27, 2023 09:36
@dcoliversun
Copy link
Contributor Author

Thanks all :) @potiuk @eladkal @sunank200 @utkarsharma2 @phanikumv

ferruzzi pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jun 27, 2023
@dcoliversun @ferruzzi
Add Alibaba Cloud AnalyticDB Spark Support (apache#31787)
81ac390 
* Add Alibaba Cloud AnalyticDB Spark Support
@eladkal eladkal mentioned this pull request Jul 6, 2023
Closed
69 tasks
@ephraimbuddy ephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jul 6, 2023
@eladkal
Copy link
Contributor

eladkal commented Nov 16, 2023

@dcoliversun can you please assist with solving AnalyticDBSparkHook item in #28790

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ephraimbuddy ephraimbuddy ephraimbuddy left review comments

@eladkal eladkal eladkal approved these changes

@kaxil kaxil Awaiting requested review from kaxil

@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner

@pankajkoti pankajkoti Awaiting requested review from pankajkoti

@pankajastro pankajastro Awaiting requested review from pankajastro

@sunank200 sunank200 Awaiting requested review from sunank200

@utkarsharma2 utkarsharma2 Awaiting requested review from utkarsharma2

@phanikumv phanikumv Awaiting requested review from phanikumv

Assignees
No one assigned
Labels
area:providers area:system-tests changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) kind:documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add Alibaba Cloud AnalyticDB Spark support
7 participants
@dcoliversun @potiuk @utkarsharma2 @eladkal @ephraimbuddy @sunank200 @phanikumv