Skip to content

Commit

Permalink
Make check_exists cache negative results as well as positive ones
Browse files Browse the repository at this point in the history
When loading a workflow with URL references that can't be resolved,
don't keep trying them over and over again.

Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@curii.com>
  • Loading branch information
tetron authored and mr-c committed Jan 25, 2023
1 parent 097061d commit 31d0117
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion schema_salad/fetcher.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,10 @@ def fetch_text(self, url: str, content_types: Optional[List[str]] = None) -> str
raise ValidationException(f"Unsupported scheme in url: {url}")

def check_exists(self, url: str) -> bool:
if url in self.cache:
entry = self.cache.get(url)
if entry is False:
return False
elif entry is not None:
return True

split = urllib.parse.urlsplit(url)
Expand All @@ -121,6 +124,7 @@ def check_exists(self, url: str) -> bool:
resp = self.session.head(url, allow_redirects=True)
resp.raise_for_status()
except Exception:
self.cache[url] = False
return False
self.cache[url] = True
return True
Expand Down

0 comments on commit 31d0117

Please sign in to comment.