Add a check in BigQueryInsertJobOperator to verify the Job state before marking it as success #40863
Conversation
boring-cyborg
bot
added
area:providers
provider:google
| if job.state in ("PENDING", "RUNNING"): | ||
| import time | ||
|
|
||
| time.sleep(5) |
There was a problem hiding this comment.
I don't think we need to add a sleep here. Why not do...
while job.state in ("PENDING", "RUNNING"):
...
There was a problem hiding this comment.
@RNHTTR @kandharvishnu In a second thought, maybe it's better to have a time.sleep() - otherwise, if it is pending or running for long time - requests will be sent immedately one after another, which might cause exceeding API rate limit.
Example from another Google providers:
What do you guys think?
There was a problem hiding this comment.
@pankajastro , fyi for #44279
There was a problem hiding this comment.
Yes, if we are looping, then sleep makes sense.
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
kandharvishnu
force-pushed
the
AIRFLOW-40839
branch
from
76e680a to
1abb8a6
Compare
github-actions
bot
removed
the
stale
|
Suppressed by #44279 |
Closes: 40839
Occasionally, Airflow marks the BigQueryInsertJobOperator task as successful before it has actually completed in BigQuery. To resolve this issue, we will add a condition to check the job's state and wait until it is successfully completed in BigQuery.