Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filter datasets by dag_id in rest API #37423

Closed
1 of 2 tasks
bbovenzi opened this issue Feb 14, 2024 · 3 comments · Fixed by #37512
Closed
1 of 2 tasks

Filter datasets by dag_id in rest API #37423

bbovenzi opened this issue Feb 14, 2024 · 3 comments · Fixed by #37512
Assignees
@Satoshi-Sh
Labels
area:API Airflow's REST/HTTP API area:webserver Webserver related Issues kind:feature Feature Requests

Comments

@bbovenzi
Copy link
Contributor

bbovenzi commented Feb 14, 2024 •
edited
Loading

Description

We have a list datasets endpoint in the API. Right now we can only filter by searching uri string. It would be useful to be able to pass one or multiple dag_ids and only return datasets that include that dag_id in consuming_dags or producing_tasks

Use case/motivation

Make it easy to see what datasets are connected to a dag.

Right now our webserver endpoint to get dataset dependencies returns all datasets+dags and then the UI needs to filter it down.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@bbovenzi bbovenzi added kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet area:API Airflow's REST/HTTP API and removed needs-triage label for new issues that we didn't triage yet labels Feb 14, 2024
@bbovenzi
Copy link
Contributor Author

Some questions I have I'd like to get people's thoughts on:

  1. Should it be a single dag_id or an array of multiple dag_ids?
  2. Should we specify only consuming or only producing or both?
  3. Maybe we need a separate, perhaps not in the public REST API, endpoint to return the entire dataset-dag graph based on either a dataset_uri or a dag_id?

@bbovenzi bbovenzi added the area:webserver Webserver related Issues label Feb 15, 2024
@bbovenzi
Copy link
Contributor Author

Actually this could just be in datasets_summary endpoint in the webserver.

@bbovenzi bbovenzi mentioned this issue Feb 15, 2024
Merged
@Satoshi-Sh
Copy link
Contributor

I can take on this task.

@Satoshi-Sh Satoshi-Sh mentioned this issue Feb 18, 2024
Merged
Satoshi-Sh added a commit to Satoshi-Sh/airflow that referenced this issue Feb 18, 2024
@Satoshi-Sh
Merge branch 'main' into feat/apache#37423/filter_datasets_by_dag_id
90448d7
Satoshi-Sh added a commit to Satoshi-Sh/airflow that referenced this issue Feb 20, 2024
@Satoshi-Sh
merged main into feat/apache#37423/filter_datasets_by_dag_id
6cc53b8
Satoshi-Sh added a commit to Satoshi-Sh/airflow that referenced this issue Feb 20, 2024
@Satoshi-Sh
Merge branch 'main' into feat/apache#37423/filter_datasets_by_dag_id
5369085
Satoshi-Sh added a commit to Satoshi-Sh/airflow that referenced this issue Feb 20, 2024
@Satoshi-Sh
Merge branch 'feat/apache#37423/filter_datasets_by_dag_id' of https:/…
dc2e052 
…/github.com/Satoshi-Sh/airflow into feat/apache#37423/filter_datasets_by_dag_id
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Satoshi-Sh Satoshi-Sh

Labels
area:API Airflow's REST/HTTP API area:webserver Webserver related Issues kind:feature Feature Requests
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Filter Datasets by associated dag_ids (GET /datasets) Satoshi-Sh/airflow
2 participants
@bbovenzi @Satoshi-Sh