resolve option (DNS cache) not honored with persistent handles

[afflerbach]

A client with persistent handle sends requests to unexpected hosts when using the pre-populated DNS cache via the resolve option:

<?php

$request = new http\Client\Request('GET', 'http://example.com/');

$client = new http\Client('curl', 'persistentID');
$client->enqueue($request);
$client->send(); // opens persistent connection to example.com

$request2 = new http\Client\Request('GET', 'http://example.com/');
$options = [ 'resolve' => ['example.com:80:127.0.0.1'] ];
$request2->setOptions($options); // remap example.com to 127.0.0.1
$client->enqueue($request2);
$client->send(); // Unexpectedly re-uses handle, issues 2nd request to example.com
print(substr($client->getResponse($request2)->getBody(), 0, 150));

I expected it to open a fresh connection when the resolve option is actually effective.

[m6w6]

Thanks four your report!

We cannot add the IP to the persistent handle ID, because we do not know it beforehand.

We'd rather have to look how to handle issues with the DNS cache.

I'll first try to determine the intended behavior of related options within libcurl.

[filex]

We cannot add the IP to the persistent handle ID, because we do not know it beforehand.

Is the resolve option (or better: its mapped IP address) already known when handle ID is generated?

If so, we could seed the handle with the resolve-option's IP for a particular domain/port pair. Normally, without a resolve map (or one that doesn't match), that seed would be null. Would this work to separate the persistent handle IDs?

[m6w6]

resolve actually works like a pre-population of the DNS cache, so if the connection already exists for host example.com, the DNS (cache) won't even be consulted.

At a first glance, everything is working as expected: https://curl.haxx.se/libcurl/c/CURLOPT_RESOLVE.html

Implementing CURLOPT_FRESH_CONNECT/CURLOPT_FORBID_REUSE might be an option.

[m6w6]

See

• https://mdref.m6w6.name/http/Client/Curl#Connection.handling and
• https://curl.haxx.se/libcurl/c/CURLOPT_FRESH_CONNECT.html resp.
• https://curl.haxx.se/libcurl/c/CURLOPT_FORBID_REUSE.html

[afflerbach]

Let me try to explain the issue in a different way: I am using the resolve option to map a hostname X to an IP address Y. Persistent connections are enabled.

What I observe seems to be the following:
A HTTP request to host X is issued. Due to the DNS cache, a TCP connection to Y is openend. The HTTP response body contains a resource from Y. The connection is kept persistent for later reuse. So far, so good.

If later another HTTP request to X is issued, and at the same time the persistent TCP connection is about to be closed (netstat -t indicates CLOSE_WAIT) it seems, that that connection is still considered to be ready for reuse.

However, a fresh connection will be opened - this time without taking the resolve option into account. Therefore, the next HTTP request will go to X and not, as expected, to Y.

Unfortunately, in this way, the resolve option is not reliably usable together with persistent handles.

[m6w6]

I'm still not sure I understand correctly.

However, a fresh connection will be opened - this time without taking the resolve option into account. Therefore, the next HTTP request will go to X and not, as expected, to Y.

This sounds like we should check for a bug in libcurl.

Do you set ["resolve" => ["X:80:Y"]] for every request?

[filex]

Do you set ["resolve" => ["X:80:Y"]] for every request?

Yes, we do! However, it is complicated to reproduce :)

I use this script in PHP-FPM:

$url = 'http://example.com/';
$request = new http\Client\Request('GET', $url);
$options = [ 'resolve' => ['example.com:80:127.0.0.1'] ];
$request->setOptions($options);

$client = new http\Client('curl', 'foo');
$client->enqueue($request);
$client->send(); // opens persistent connection to localhost

print('<pre>');
print($client->getResponse($request)->getBody());
print("PID: " . getmypid() ."\n");
print("Port: " . $client->getTransferInfo($request)->local_port . "\n");

restart php-fpm to clear any connections and then call the script a couple of times. all requests go to localhost:80 where we have an apache running.

From the PID and Port output, I see that I switch between two FPM children when I reload, each of them has a persistent connection (local port not changing).

Then comes the boring part: netstat -nt |grep 127.0.0.1:80 until one of the connections changes from ESTABLISHED to CLOSE_WAIT:

$  netstat -nt |grep 127.0.0.1:80
tcp        0      0 127.0.0.1:41939         127.0.0.1:80            VERBUNDEN  
tcp        1      0 127.0.0.1:41936         127.0.0.1:80            CLOSE_WAIT 
tcp6       0      0 127.0.0.1:80            127.0.0.1:41936         FIN_WAIT2  
tcp6       0      0 127.0.0.1:80            127.0.0.1:41939         VERBUNDEN  

When I now hit reload in the browser, my request goes to the real example.com and not to localhost.

This sounds like we should check for a bug in libcurl.

Maybe :) We use the CentOS 7 system package libcurl-7.29.0-25.el7.centos.x86_64.

[m6w6]

Oh, somehow I thought you were using libcurl >= 7.41 -- no idea where I got his from :-/

Does the CentOS 7 system package use an asynchronous resolver (c-ares or threaded; check with curl-config --features | grep AsynchDNS)? If so, IIRC I fixed cache lookups for async resolvers in libcurl-7.38.

Would you mind to check whether your issue disappears with a libcurl >= 7.38?

Thank you!

[m6w6]

I'm still unable to reproduce this with libcurl built from tag curl-7_29_0. Neither with system, threaded nor c-ares resolver.

Restarting the target webserver, shuts down the socket on the server side and puts it in CLOSE_WAIT state. For me, libcurl detects the dead connection and correctly creates a new one with the information found in the DNS cache.

[filex]

Does the CentOS 7 system package use an asynchronous resolver (c-ares or threaded; check with curl-config --features | grep AsynchDNS)?

yes, it does.

If so, IIRC I fixed cache lookups for async resolvers in libcurl-7.38.

Would you mind to check whether your issue disappears with a libcurl >= 7.38?

I can reproduce the error with 7.29.0 (the base for the centos 7.2 libcurl) with this configuration

./configure --enable-threaded-resolver --prefix=/tmp/curl-test-7.29.0

and LD_PRELOAD'ing the .so from /tmp/curl-test-7.29.0/lib/ before restarting my php-fpm.

I cannot reproduce it with curl >= 7.35.

[filex]

Restarting the target webserver, shuts down the socket on the server side and puts it in CLOSE_WAIT state. For me, libcurl detects the dead connection and correctly creates a new one with the information found in the DNS cache.

At first, I have experimented with restarting the webserver, too. But that doesn't seem to fail reliably.

After establishing the connection to localhost I have to wait for about a minute before the "server" part of the connection disappears from netstat. Then, the re-connect goes to example.com (reliably).

I have not yet figured out which timeout is responsible. Lowering tcp_fin_timeout didn't make my day less boring :)

[m6w6]

Okay, I've been able to reproduce it with 7.29 and setting dns_cache_timeout to 1 second.

So, it looks like an old libcurl bug to me where those entries were expiring, while they shouldn't.

I couldn't find an explicit change log entry about such an issue, but, ugh, there've been more than 20 releases since.

[filex]

Okay, I've been able to reproduce it with 7.29 and setting dns_cache_timeout to 1 second.

you're debugging 60 times faster than me :)

So, it looks like an old libcurl bug to me where those entries were expiring, while they shouldn't.

does that mean, that any resolve-option will be useless after the first dns_cache_timeout has passed, until the fpm-child restarts?

I couldn't find an explicit change log entry about such an issue, but, ugh, there've been more than 20 releases since.

hm. do you know any workaround that would avoid this issue with 7.29/centos7?

[m6w6]

That's a hard one.

Maybe a dns_cache_timeout higher than the maximum time of a php-fpm process cycle?

A better solution might be to run and control your own DNS forwarder/cache to be used by libcurl https://mdref.m6w6.name/http/Client/Curl#$dns_servers.

[m6w6]

Gosh, that's only available with the c-ares resolver (unless you want the whole host to use the custom DNS cache/forwarder).

[afflerbach]

Maybe a dns_cache_timeout higher than the maximum time of a php-fpm process cycle?

We'll try that one.

somehow I thought you were using libcurl >= 7.41 -- no idea where I got his from :-/

Might be because of @rcanavans comment in #37. We support multiple distributions and therefore link against different versions of libcurl. 7.29 currently is the oldest.

[filex]

Does the $dns_cache_timeout option only affect the TTL of the DNS Cache entries that are created by the particular request with this options? Therefore, can I use a high TTL for all requests with resolve option, and no $dns_cache_timeout for all other requests to have the default TTL there?

[m6w6]

Quite a good question, actually.

My wild guess is "no, you can't". All requests (curl easy handles) share the DNS cache of the client (curl multi handle), so I think each request with different options affects the shared DNS cache.

[filex]

As you suggested, our workaround is now setting the TTL to a day on requests that use pre-resolved DNS. Thus, the entries stay long enough in curl's DNS cache to be still available when a stale persistent connection is reconnected. (Problem solved).

In our setup a domain is either always or never pre-resolved. So we shouldn't have problems with conflicting DNS entries. As our FPM processes have a life span that is usually much short than a day, I don't expect side effects of the long TTL. (Hopefully: No new problem opened).

I'd say, let's close this as "documented" :)

Thanks a lot for your help!

[m6w6]

Alright, re-open at will! Thanks.