Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate kubernetes async provider #1418

Closed
1 task
Lee-W opened this issue Jan 2, 2024 · 5 comments · Fixed by #1480
Closed
1 task

Deprecate kubernetes async provider #1418

Lee-W opened this issue Jan 2, 2024 · 5 comments · Fixed by #1480
Assignees
Labels
feature New feature or request
Milestone

Comments

@Lee-W
Copy link
Contributor

Lee-W commented Jan 2, 2024

Is your feature request related to a problem? Please describe.

as most of the functionality has been contributed back to OSS airflow, we're going to leverage the code there and deprecate this project

Describe the solution you'd like

  1. Contribute the delta to the OSS provider
  2. Point the astronomer provider to the OSS provider
  3. Do the release and wait for a month or so before we remove the astronomer provider in a major release

Describe alternatives you've considered

Additional context


Current Progress

  • Number of operator changes on OSS airflow needed: 1
  • Number of operators to deprecate: 1
  1. KubernetesPodOperatorAsync
    1. Modify KPO to log container log periodically  apache/airflow#37279
    2. KPO Maintain backward compatibility for execute_complete and trigger run method apache/airflow#37363
    3. Fix KPO task hanging when pod fails to start within specified timeout apache/airflow#37514
    4. KPO Maintain backward compatibility for execute_complete and trigger run method apache/airflow#37454
    5. Fix KeyError when KPO exits too soon apache/airflow#37508
    • Minimum Provider Version:
@Lee-W Lee-W added the feature New feature or request label Jan 2, 2024
@Lee-W Lee-W self-assigned this Jan 2, 2024
@Lee-W Lee-W changed the title Drepcreate kubernetes provider Deprecate kubernetes async provider Jan 2, 2024
@Lee-W Lee-W added this to the 1.19.0 milestone Jan 19, 2024
@phanikumv phanikumv assigned pankajastro and unassigned Lee-W Jan 31, 2024
@pankajastro
Copy link
Contributor

OSS KPO prints the container log at the end of execution, specifically when the execute_complete method runs. I am considering logging the container log in the trigger, but doing so would result in the log being printed twice: once when the trigger runs and again from the execute_complete method.

I have a rough implementation in progress, which still needs testing and the addition of the parameter periodic_container_log. This parameter, when set to True, will also print the log from the trigger and keep it False by default.

https://github.com/apache/airflow/compare/main...astronomer:airflow:kpo_logging?expand=1

@pankajkoti
Copy link
Collaborator

We verified that PR apache/airflow#37279 is able to fetch the container logs periodically.

Screenshot 2024-02-12 at 12 49 29 PM

@pankajkoti
Copy link
Collaborator

Waiting for OSS provider release

pankajkoti added a commit that referenced this issue Feb 18, 2024
Deperecate KubernetesPodOperatorAsync and proxy it to its OSS counterpart.

related: #1418
---------

Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Pankaj Koti <pankaj.koti@astronomer.io>
@pankajkoti
Copy link
Collaborator

I have linked the above PRs in the description above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants