Most elegant and terse way of adding wrappers around the library

[forthrin]

Consider this simplified example for adding caching and error handling to HTTP GET requests:

require 'fileutils'
require 'httpx'

def cache(url, headers:, expire: 3600)
  uri = URI.parse(url)
  local = "/tmp/cache/#{uri.host}#{uri.request_uri}"
  if !File.exist?(local) || Time.now - File.mtime(local) > expire
    response = HTTPX.get(url, headers:)
    pp [url, headers, response] unless response.status == 200
    FileUtils.mkdir_p(File.dirname(local))
    File.write(local, response.body)
  end
  File.read(local)
end

cache('http://localhost:8080/foo/bar.html', expire: 3600)

What would be the most elegant and terse way to extend this to all HTTP methods and arguments?

1. Inheritance is discouraged by the author (for unspecified reasons).

class HTTPY < HTTPX::Session
  # Not sure what goes here
end

HTTPY.update('http://localhost:8080/', json:)
HTTPY.delete('http://localhost:8080/', params:)

2. Refinements bar direct access to the original.

module Cache
  refine HTTPX::Chainable do
    def post(*uri, **options)
      puts 'Refined!'
      super
    end
  end
end

using Cache

HTTPX.post('http://localhost:8080/')

3. Lambdas don't have access to the URL (to make the local path), and they look quirky.

def cache(lame, expire: nil)
  response = lame.call unless File.exists?(local = 'no.clue')
  response.body
end

cache(-> { HTTPX.post('http://localhost:8080/', form:) }, expire: 3600)

[HoneyryderChuck]

The recommended way is building a custom plugin.

Intercepting a request can be done like this:

module MyCachePlugin
  module InstanceMethods
    def send_requests(*requests)
       cached_responses = get_cached_requests(requests)
       # and for the remainder, call super, and guarantee ordered responses based on requests input
    end
  end
end

You'll need to get familiar with httpx internal API, so I suggest reading the documentation, and studying other similar plugins, such as the response_cache plugin.

Inheritance is discouraged by the author (for unspecified reasons).

There's no documentation, nor wiki page, suggesting inheritance as a gateway to customize functionality. On the other hand, alternatives to do so are documented (as linked above). While I could go into detail about specifying why you shouldn't (but can) do option A or B (it may even work), not being a documented alternative should send a clear message.

[forthrin]

This became a cakewalk when I found out how HTTPX.request and **options can support any type of request.

Below is a incomplete illustration of how I ended up. You're right that the best thing would be to study your plugin system and write something that works with that. That way other people could benefit from having a user-controlled cache/mirror.

I wrapped the calls in classless methods because I didn't want to write new for every call, and I haven't figured out how to do this (Cache.get) in Ruby without making everything static. Please do share the trick if you have the time.

$http = HTTPX.plugin(:follow_redirects)

def get(url, expire: nil, ext: 'html', **options)
  Cache.new(url, ext).cache(expire:, method: 'GET', **options)
end

def post(url, expire: nil, ext: 'html', **options)
  Cache.new(url, ext).cache(expire:, method: 'POST', **options)
end

class Cache
  def initialize(url, ext)
    @local = url.convert_to_local_path
    @ext = @local.extension || ext
  end

  def cache(expire: nil, method: nil, **options)
    if local_file_missing? || local_file_expired?(expire)
      response = $http.request(method, @url, max_redirects: 2, **options)
      if response.error
        warn response.error.class
      elsif response.status != 200 || response.body.empty?
        warn "HTTP/#{response.version} #{response.status}"
        response.headers.each { |k, v| warn "#{k.gsub(/\b./, &:capitalize)}: #{v}" }
      else
        File.write(@local, response.body)
      end
    end

    body = File.read(@local)
    case @ext
    when 'html' then Nokogiri.HTML5(body)
    when 'json' then JSON.parse(body)
    else body
    end
  end
end

[HoneyryderChuck]

Cool to hear it's working out for you 👍

Please do share the trick if you have the time.

I don't know what you mean by "everything static", but there's nothing wrong with having class-level methods (like Cache.get) and not have to deal with ivars (your use case above doesn't seem to benefit from being built with objects).

A few tips:

• instead of $http.request(method, @url, max_redirects: 2,..., consider initializing the client with the common options to apply to all requests, i.e. $http = HTTPX.plugin(:follow_redirects, max_redirects: 2).
• instead of having that if/else logic, and if using a ruby version supporting pattern matching, consider using it:

# instead of
if response.error
    warn response.error.class
elsif response.status != 200 || response.body.empty?
   warn "HTTP/#{response.version} #{response.status}"
   response.headers.each { |k, v| warn "#{k.gsub(/\b./, &:capitalize)}: #{v}" }
 else
   File.write(@local, response.body)
end

# consider
case response
in { error: error } 
  # handle exception
in { status: 200 }
  # success case
else
  # everything else
end

(I know it's not exactly the same, but you gett the point).

• File.write(@local, response.body) should be te other way round (File.write( response.body, @local)); still, there's something better, response.body.copy_to(@local).
• Instead of Nokogiri.HTML5(body) or JSON.parse(body), you can do response.xml or response.json, same outcome.

[forthrin]

Thanks for your friendly advice and recommendations. Some comments/follow-up questions.

1. I plan to read up on pattern matching (the major new feature in Ruby 3). But won't the need for checking for empty response break this and revert to if statements? (HTTP 200 and Content-Length: 0 does happen!)
2. Your comment about File.write doesn't comply with Ruby documentation (and all my other code).
3. By static I mean all methods in the class are def self.method and no instantiation. Or is there a design pattern that lets you create new instances via class methods? (Without quirky hacks and dummy classes)
4. Is there a way to get human readable status codes? (eg. "404 Not Found") Should be easy to implement.
5. If I remove if response.error, I get occasional deprecation warnings for response.status. Maybe because certain errors (ie. failure before getting a server response) don't have a "status" per se?

[HoneyryderChuck]

1. Indeed, just wanted to describe "yet another way", if it makes sense, and that's why I wrote "I know it's not exactly the same".
2. You're right, the ordering of this API always confuses me (I always think (src, dest)).
3. You can technically implement "method_missing" in the class, and instantiate-then-call-method. Not sure if that would be of great benefit though, YMMV.
4. I don't find much value in implementing that in httpx. However, it's easy for you to get it, considering that you can require "net/http/status" from stdlib and then access it via status code to Net::HTTP::STATUS_CODES[status].
5. response.status on error objects is a deprecated API, and will be removed in v1 release. if you want to test for errors, you can response.raise_for_status, use pattern matching as described above, of test for response.error like you do already.

[forthrin]

A follow-up on error handling.

1. Does response.error happen only for problems occurring before the server has had a chance to respond? Or can it trigger after too, ie. for HTTP status errors? (In any case, explain the logic and connection here.)
2. For errors occurring before, is there any more information to be had for the caller than the exception code itself? If so, how does one print this information?
3. Where can one find a code example on how to elegantly weave the case of the server returning an empty document into the terse methods suggested in the Wiki? (As an empty document usually constitutes an anomality.)
4. How does one handle exceptions generated by raise_for_status? The exception code seems to include both the HTTP error code, the response header, and the response body.HTTP Error: 403 {...} <html>...</html>.

[HoneyryderChuck]

1. response.error willl either return the exception which caused a request to fail, or a HTTPX::HTTPError, in case it's a 4xx or 5xx response.
2. you have the exception object. you can use whatever the exception object defines as usable parameters, including its backtrace.
3. what do you mean by "empty document"? If the server advertises a content-length which does not match the received payload, this will already yield an error (if it doesn't, please provide a reproduction).
4. raise_for_status will yield the error object described in 1. The case you are describing is from a HTTPX::HTTPError object, and you can refer to the docs to know what's available.

[forthrin]

Let's get practical.

The goal is to create an error handler that, when something goes wrong, the user gets information that is 1) precise and 2) as detailed as necessary to understand and address the situation.

Thus it would have be something like the pseudo-code below. How would this be written in actual code?

if request_failed_before_server_responded
  case reason
  when certificate_failure
    print_certificate_etc # like a verbose openssl/curl request
  when no_internet_connection # eg. !File.read('/etc/resolv.conf').match?(/^[^#]/)
    warn "no internet connection"
  when other_relevant_tests
    ...
  end
elsif ![103, 200].include?(response.status) || response.body.empty?
  warn "HTTP/#{response.version} #{response.status}"
  response.headers.each { |k, v| warn "\033[2m#{k.gsub(/\b./, &:capitalize)}: #{v}\033[m"
else
  # OK
end

Notes:

1. Reason for considering 103 "OK" is a server returning this as the final document.
2. A server returning 200 and empty document constitutes an anomaly in the use case, so one wants to see header for hints.

[HoneyryderChuck]

begin
  response = HTTPX.get...
  response.raise_for_status
rescue => e
  handle_error(e)
end

def handle_error(e)
  case e
  when OpenSSLErrorForCertificateFailure
    # 
  when NoInternetError
    # 
  when HTTPX::HTTPError
     status = e.status
     res = e.response
     # 
  # when other errors
  else
    # general handling
  end

[forthrin]

def x
  url = 'http://127.0.0.1:4567/status'
  response = HTTPX.get(url)
  response.raise_for_status
rescue => e
  case e
  when HTTPX::HTTPError
    puts 'HTTP 400-599 (Server Error)'
    pp e.class
    pp response.status # AKA e.status
    pp response.headers

  # All below this point are errors occuring before server response

  when OpenSSL::SSL::SSLError
    puts 'Could not establish secure connection (Certificate problem, etc.)'
    pp e
  # when OpenSSLErrorForCertificateFailure
  #   puts 'uninitialized constant OpenSSLErrorForCertificateFailure (NameError)'

  when IPAddr::AddressFamilyError
    puts 'Could not connect to the Internet'
    pp e # #<IPAddr::AddressFamilyError: address family must be specified>
  # when NoInternetError
  #   puts 'uninitialized constant NoInternetError (NameError)'

  else
    puts 'Other errors occuring before server response'
    pp e
  end

else
  puts 'HTTP 100-399 (Server OK)'
  puts response.status
  puts response.body.to_s.length
end

1. This is what I have at the moment, based on your snippet. Comments?
2. Where are OpenSSLErrorForCertificateFailure and NoInternetError defined? They cause NameError. The only ones I can find in the source code are the ones below (some subclasses omitted for brevity):

class ConnectionError < Error
class GoawayError < Error
class HTTPError < Error
class HTTPProxyError < Error
class MisdirectedRequestError < HTTPError
class ResolveError < Error
class Socks4Error < Error
class Socks5Error < Error
class TimeoutError < Error
class UnsupportedSchemeError < Error

[HoneyryderChuck]

1. LGTM 👍
2. You can rescue standard ruby errors, i.e. OpenSSL::SSL::SSLError or the exception raised by socket code when no internet is on.

[forthrin]

1. Where to find an exhaustive list of network exceptions that may occur, apart from those defined in this library?
2. On my Mac, IPAddr::AddressFamilyError / OpenSSL::SSL::SSLError occur instead of NoInternetError / OpenSSLErrorForCertificateFailure. Where are the latter defined? And why don't they occur like you suggest?
3. Is there a way to handle all unexpected events without raise_for_status? Or a more generally terse way like:

def x
  response = HTTPX.get(url)
  case response.status
  when 400-599 # or possibly "when false"
    puts "HTTP/#{response.version} #{response.status}"
    pp response.headers
  when 100-399 # or possibly "when true"
    puts response.body
  else # or possibly "when nil"
    puts response.status.class # eg. 'IPAddr::AddressFamilyError'
    puts response.details # eg. 'address family must be specified'
  end
end

[HoneyryderChuck]

Sorry if I led you to believe those errors exist, they don't. OpenSSL::SSL::SSLError is the base error for all openssl related errors. For everything else, they're all defined in mainline ruby; httpx only wraps Errno::TIMEDOUT as regular HTTPX::TimeoutError, but beyond that, anything from the Errno family can be raised, as well as other socket/resolv related errors.