Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature airflow dag #3321

Merged
merged 4 commits into from
May 8, 2022
Merged

Feature airflow dag #3321

merged 4 commits into from
May 8, 2022

Conversation

nopcoder
Copy link
Contributor

@nopcoder nopcoder commented May 8, 2022 •
edited
Loading

Close #3257

  • Airflow hook: add wait_for_dag property, boolean (Default: false) to indicate if we like to wait for DAG to complete.
  • Airflow hook: timeout (Default: 1m). Duration to wait for DAG to complete.
  • DAG Run ID updated to use hook run ID and not timestamp

Testing was done manually - using a successful and unsuccessful DAG.

@nopcoder nopcoder added area/hooks improvements or additions to the hooks subsystem include-changelog PR description should be included in next release changelog labels May 8, 2022
@nopcoder nopcoder self-assigned this May 8, 2022
@nopcoder nopcoder marked this pull request as ready for review May 8, 2022 08:19
@nopcoder nopcoder requested a review from itaiad200 May 8, 2022 08:19
itaiad200
itaiad200 approved these changes May 8, 2022
View reviewed changes
Copy link
Contributor

@itaiad200 itaiad200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! minor comments

pkg/actions/webhook.go Outdated
@@ -128,7 +128,7 @@ func (w *Webhook) Run(ctx context.Context, record graveler.HookRecord, buf *byte
return nil
}

func executeAndLogResponse(ctx context.Context, req *http.Request, buf *bytes.Buffer, timeout time.Duration) (n int, err error) {
func executeAndLogResponse(ctx context.Context, req *http.Request, buf *bytes.Buffer, timeout time.Duration, respMsg interface{}) (n int, err error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would create a separate function with shared logic with this one, instead of forcing most callers to pass a nil for a value they don't use (at the very least add documentation for this arg).

@nopcoder nopcoder requested a review from itaiad200 May 8, 2022 09:27
itaiad200
itaiad200 approved these changes May 8, 2022
View reviewed changes
Copy link
Contributor

@itaiad200 itaiad200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for applying the changes!

@nopcoder nopcoder merged commit 3f96351 into master May 8, 2022
@nopcoder nopcoder deleted the feature/airflow-dag branch May 8, 2022 10:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@itaiad200 itaiad200 itaiad200 approved these changes

Assignees

@nopcoder nopcoder

Labels
area/hooks improvements or additions to the hooks subsystem include-changelog PR description should be included in next release changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Airflow hook wait for dag to finish
2 participants
@nopcoder @itaiad200