Add exception handling for recent data scraping airflow operators #3607
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
charlie-costanzo
added
the
data-pipeline-ingestion-and-modeling
charlie-costanzo
requested review from
evansiroky and
vevetron
as code owners
charlie-costanzo
requested review from
mjumbewu and
erikamov
and removed request for
evansiroky and
vevetron
erikamov
reviewed
Dec 20, 2024
| req = requests.get(url) | ||
| soup = BeautifulSoup(req.text, "html.parser") | ||
| for key, url in xlsx_urls.items(): | ||
| try: |
There was a problem hiding this comment.
Since there are others exception handling on line 44 and 55, could those line be unnested and this other "try" moved to line 60?
Comment on lines
+188
to
+190
| except Exception as e: | ||
| logging.error(f"Error in XLSX operator execution: {e}") | ||
| raise AirflowException(f"XLSX operator execution failed: {e}") |
There was a problem hiding this comment.
For the nested exceptions here, this last one is handling just lines 138 to 154? Maybe could be moved up?
Comment on lines
+128
to
+130
| except Exception as e: | ||
| logging.error(f"Error in geoportal data fetch: {e}") | ||
| raise AirflowException(f"Failed to fetch geoportal data: {e}") |
Comment on lines
+252
to
+254
| except Exception as e: | ||
| logging.error(f"Error in geoportal operator execution: {e}") | ||
| raise AirflowException(f"Geoportal operator execution failed: {e}") |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We recently introduced MVP data syncs of the DOT NTD data portal and the state geodata portal. This PR goes back and introduces the appropriate exception handling to catch unwanted behavior.
Operators impacted:
Type of change
How has this been tested?
Post-merge follow-ups
ensure these operators run as expected