no_proxy Parameter in Manual Proxy Configuration

[anusreePrakash]

Basic Info

• Faraday Version: 2.1.0
• Ruby Version: 2.7.2

Issue description

I am attempting to create a manual proxy using Faraday and would like to understand if there is support for providing a no_proxy parameter to bypass specific hosts.
I have checked the Faraday code and noticed that it supports the no_proxy functionality when set as an environment variable. However, I cannot rely on environment variables in my use case and would like to manually provide the no_proxy parameter in the manual proxy configuration.

Consider the following example use case where a manual proxy is created with various configurations:

host = 'http://example.com'
ssl_config = { verify: true }
open_timeout = 15
timeout = 30
proxy_url = 'http://proxy.example.com:8080'
no_proxy_hosts = ['internal.example.com']

conn = Faraday.new(
  host,
  ssl: ssl_config,
  request: { open_timeout: open_timeout, timeout: timeout },
  proxy: proxy_url,
 no_proxy: no_proxy_hosts
)

[iMacTia]

Hi @anusreePrakash and thank you for opening this.
This may be due to a simplified example, but I'm not really sure what the use case would be.
When you initialise the faraday connection, you pass the host as a parameter, so if the application knows the list of no_proxy_hosts, you can simply omit the proxy param from connections using one of those.

The proxy/no_proxy ENV variable support has been added for cases where the application code does not know which domains should be called through the proxy or not, as that moves the responsibility over to the infrastructure/environment provider. But in your case, it seems like you're dealing with the proxy entirely at the application code, so the no_proxy option seems redundant.

Could you please elaborate a bit more on how you'd be using it and why you can't simply do something like the following:

host = 'http://example.com'
ssl_config = { verify: true }
open_timeout = 15
timeout = 30
proxy_url = 'http://proxy.example.com:8080'
no_proxy_hosts = ['internal.example.com']

conn_params = {
  ssl: ssl_config,
  request: { open_timeout: open_timeout, timeout: timeout }
}

conn_params.merge!(proxy: proxy_url) unless no_proxy_hosts.include?(host)

conn = Faraday.new(host, conn_options)

[anusreePrakash]

Thank you for your response. I appreciate your suggestion and understand that in many cases, omitting the proxy parameter based on the host can be a viable solution.

However, in our use case, the need for the no_proxy parameter arises when the initial request host might redirect to another host dynamically. Consider the scenario where the initial host responds with a redirection to a different host, and that redirected host is on the no_proxy list. In such cases, we need to ensure that the redirection respects the proxy settings, even if the original host is on the no_proxy list.

Here's an example to illustrate the use case:

original_host = 'http://example.com'
ssl_config = { verify: true }
open_timeout = 15
timeout = 30
proxy_url = 'http://proxy.example.com:8080'
no_proxy_hosts = ['internal.example.com', 'http://redirected.example.com']

# Initial request to the original host
conn = Faraday.new(
  original_host,
  ssl: ssl_config,
  request: { open_timeout: open_timeout, timeout: timeout },
  proxy: proxy_url,
  no_proxy: no_proxy_hosts
)

# Assume the original host responds with a redirection to a different host
# In this case, we want to ensure that the redirection respects the proxy settings,
# even if the original host is on the `no_proxy` list.
conn = Faraday.new(
  host,
  ssl: ssl_config,
  request: { open_timeout: open_timeout, timeout: timeout },
  proxy: proxy_url,
 no_proxy: no_proxy_hosts
)

[iMacTia]

Right, redirection was definitely something I didn't consider, thank you for bringing it up!

Even in this case though, I don't see why you can't use conditions if you're manually setting up both connections:

original_host = 'http://example.com'
ssl_config = { verify: true }
open_timeout = 15
timeout = 30
proxy_url = 'http://proxy.example.com:8080'
no_proxy_hosts = ['example.com']

conn_params = {
  ssl: ssl_config,
  request: { open_timeout: open_timeout, timeout: timeout }
}

# Initial request to the original host.
# We're not passing the proxy here because `original_host` is in the no_proxy list
conn = Faraday.new(original_host, conn_params)

res = conn.get(...)

# assuming a redirect happens
if [301, 302].include?(res.status)
  new_url = safe_escape(response['location'] || '')
  new_host = URI.parse(new_url).host
  redirect_params = no_proxy_hosts.include?(new_host) ?
    conn_params :
    conn_params.merge(proxy: proxy_url)
  
  conn = Faraday.new(new_host, redirect_params)
end

This is just explanatory, I haven't actually run the code, but it's to give you an idea of how you can check the response and still manage the connection parameters to use the proxy or not based on that.

Now, if you're instead using the follow-redirect middleware, then this becomes a little more complicated, but it should still be doable thanks to the middleware callback option. If that's your use-case and are not sure how to make that work, then feel free to let me know and I can help further.

[iMacTia]

I'd also like to add, as much as I'd like to have this feature built into Faraday itself, it's not really straightforward to do because the proxy is currently managed at the connection level, not at the individual request.
Further investigation would be necessary to find a way to make this work, but I first need to understand exactly what the expected usage would be.

In your example, you're manually setting up two separate connections, so to address that I'd need to check if the host provided to the connection is on the list of no_proxy_hosts, but since both parameters are coming from the user, I don't really see the value in doing that