Skip to content

Commit

Permalink
Add in a retry for urls that do not succeed
Browse files Browse the repository at this point in the history
Adding an open timeout and upping the read timeout
  • Loading branch information
carolyncole committed Oct 16, 2024
1 parent d5147d1 commit f4b7d39
Showing 1 changed file with 17 additions and 4 deletions.
21 changes: 17 additions & 4 deletions app/lib/describe_indexer.rb
Original file line number Diff line number Diff line change
Expand Up @@ -82,14 +82,27 @@ def rss_url_list
##
# Parse the rss_url, get a JSON resource url for each item, convert it to XML, and pass it to traject
def perform_indexing
urls_to_retry = []
rss_url_list.each do |url|
resource_json = URI.open(url, read_timeout: 30).read
resource_xml = prep_for_indexing(resource_json)
traject_indexer.process(resource_xml)
Rails.logger.info "Successfully imported record from #{url}."
process_url(url)
rescue
urls_to_retry << url
end

# retry an errored urls a second time and send error only if they don't work a second time
urls_to_retry.each do |url|
process_url(url)
rescue => ex
Rails.logger.warn "Error importing record from #{url}. Exception: #{ex.message}"
Honeybadger.notify "Error importing record from #{url}. Exception: #{ex.message}"
end
end

def process_url(url)
uri = URI.open(url, open_timeout: 30, read_timeout: 30)
resource_json = uri.read
resource_xml = prep_for_indexing(resource_json)
traject_indexer.process(resource_xml)
Rails.logger.info "Successfully imported record from #{url}."
end
end

0 comments on commit f4b7d39

Please sign in to comment.