Add dag re-parsing request endpoint #39138
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
utkarsharma2
requested review from
ryanahamilton,
ashb,
bbovenzi,
pierrejeambrun,
ephraimbuddy,
potiuk,
jedcunningham,
kaxil and
XD-DENG
as code owners
boring-cyborg
bot
added
area:API
utkarsharma2
requested review from
bolkedebruin,
dstandish,
eladkal,
o-nikolas,
mobuchowski,
Fokko,
mik-laj,
hussein-awala,
josh-fell,
uranusjr and
jhtimmins
as code owners
uranusjr
reviewed
May 10, 2024
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
utkarsharma2
added a commit
to astronomer/airflow
that referenced
this pull request
May 15, 2024
utkarsharma2
added a commit
that referenced
this pull request
May 22, 2024
* UI changes for DAG Reparsing feature releated: #39138 * Add correct permission for UI * Fix JS * Update airflow/www/static/js/main.js * Add a wrapper arounf REST DAG Reparsing API * Move file_token from dag object to a separate field * Fix CI issue * Change icons
romsharon98
pushed a commit
to romsharon98/airflow
that referenced
this pull request
Jul 26, 2024
* Add dag re-parsing request endpoint * Fix migration files * Fix test_file_paths_in_queue_sorted_by_priority test * Fix database cleanup issue * Change http method to post * Fix static checks * Remove created object from response * Fix testcases * Fix testcases * Change http response status for duplicate request * Fix tests * Changed the airflow version in migration file * Update airflow/api_connexion/openapi/v1.yaml Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Update airflow/api_connexion/openapi/v1.yaml Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Update airflow/models/dagbag.py Co-authored-by: Wei Lee <weilee.rx@gmail.com> * Update airflow/dag_processing/manager.py Co-authored-by: Wei Lee <weilee.rx@gmail.com> * Better exception handling * Addressed PR comments * Fix tests * Update airflow/api_connexion/openapi/v1.yaml Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * Update airflow/api_connexion/endpoints/dag_parsing.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * Update airflow/api_connexion/openapi/v1.yaml Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * Update airflow/models/dagbag.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * Apply the suggestion from PR comments * Remove __eq__ magic method * Remove get_requests() method * Optimize database request * Update airflow/dag_processing/manager.py Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Fix test * Change return status code * Fix test * Fix static checks --------- Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Co-authored-by: Wei Lee <weilee.rx@gmail.com> Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
romsharon98
pushed a commit
to romsharon98/airflow
that referenced
this pull request
Jul 26, 2024
* UI changes for DAG Reparsing feature releated: apache#39138 * Add correct permission for UI * Fix JS * Update airflow/www/static/js/main.js * Add a wrapper arounf REST DAG Reparsing API * Move file_token from dag object to a separate field * Fix CI issue * Change icons
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Nov 9, 2024
* UI changes for DAG Reparsing feature releated: apache/airflow#39138 * Add correct permission for UI * Fix JS * Update airflow/www/static/js/main.js * Add a wrapper arounf REST DAG Reparsing API * Move file_token from dag object to a separate field * Fix CI issue * Change icons GitOrigin-RevId: 2250a1a228f30d13ae3c481b059e05bbd29d3839
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
API endpoint to request reparsing of DAG.
The API is useful in cases when users have lots of dags and want to prioritize the parsing of a particular dag. We record the requests in the DB table and rearrange the
file_path_queuevariable such that the requested DAG is prioritized over other dags in the queue.