Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[async] Emit datasets when using ExecutionMode.AIRFLOW_ASYNC #1264

Open
1 task
tatiana opened this issue Oct 21, 2024 · 2 comments
Open
1 task

[async] Emit datasets when using ExecutionMode.AIRFLOW_ASYNC #1264

tatiana opened this issue Oct 21, 2024 · 2 comments
Assignees
@pankajkoti
Labels
area:datasets Related to the Airflow datasets feature/module execution:async Related to the Async execution mode priority:high High priority issues are blocking or critical issues without a workaround and large impact stretch goal
Milestone
Cosmos 1.9.0

Comments

@tatiana
Copy link
Collaborator

tatiana commented Oct 21, 2024 •
edited
Loading

Context

As of Cosmos 1.7, we introduced ExecutionMode.AIRFLOW_ASYNC. However, when using this mode, users are not benefiting from Cosmos automatic Airflow dataset emission both from an OpenLineage perspective and also from an Airflow dataset perspective.

Acceptance criteria

  • When using ExecutionMode.AIRFLOW_ASYNC with BQ, Cosmos should emit Datasets the same way it does when using ExecutionMode.LOCAL
@tatiana tatiana added the execution:async Related to the Async execution mode label Oct 21, 2024
@dosubot dosubot bot added the area:datasets Related to the Airflow datasets feature/module label Oct 21, 2024
@tatiana tatiana mentioned this issue Oct 21, 2024
Merged
2 tasks
@tatiana tatiana added this to the Cosmos 1.9.0 milestone Oct 30, 2024
@github-actions GitHub Actions
Copy link

This issue is stale because it has been open for 30 days with no activity.

@github-actions github-actions bot added the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Nov 30, 2024
@tatiana tatiana added stretch goal and removed stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed labels Jan 13, 2025
@pankajkoti pankajkoti self-assigned this Jan 24, 2025
@pankajkoti
Copy link
Contributor

Since the async operator now extends the DbtRunLocalOperator as part of the work in PR #1474, the functionality is already available through DbtRunLocalOperator. Therefore, if we proceed with this approach, no additional work may be required to support this. We will monitor this issue alongside PR #1474 and validate once the work in that PR is completed.

@tatiana tatiana added the priority:high High priority issues are blocking or critical issues without a workaround and large impact label Jan 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pankajkoti pankajkoti

Labels
area:datasets Related to the Airflow datasets feature/module execution:async Related to the Async execution mode priority:high High priority issues are blocking or critical issues without a workaround and large impact stretch goal
Projects
None yet
Milestone
Cosmos 1.9.0
Development

No branches or pull requests
3 participants
@tatiana @pankajkoti @pankajastro