[THREESCALE-9301] Fix dns cache miss

[tkan145]

What

Fix https://issues.redhat.com/browse/THREESCALE-9301

Notes

Before

Description	qname	queries	answer	cached key
shortname	backend-listener	backend-listener. backend-listener.default.svc.cluster.loca backend-listener.svc.cluster.local backend-listener.cluster.local	name: backend-listener.default.svc.cluster.local address: 10.7.0.1 type: A	backend-listener.default.svc.cluster.local
FQDN	backend-listener.default.svc.cluster.local	backend-listener.default.svc.cluster.local. backend-listener.default.svc.cluster.local.default.svc.cluster.local backend-listener.default.svc.cluster.local.svc.cluster.local backend-listener.default.svc.cluster.local.cluster.local	name: backend-listener.default.svc.cluster.local. address:10.7.0.1 type: A	backend-listener.default.svc.cluster.local.
FQDN	foo.example.com	foo.example.com. foo.example.com.default.svc.cluster.local foo.example.svc.cluster.local foo.example.cluster.local	name: foo.example.com. address: 10.7.0.1	foo.example.com.
Absolute FQDN	foo.example.com.	foo.example.com.	name: foo.example.com. address: 10.7.0.1	foo.example.com.

Now

Description	qname	queries	answer	cached key
shortname	backend-listener	backend-listener. backend-listener.default.svc.cluster.loca backend-listener.svc.cluster.local backend-listener.cluster.local	name: backend-listener.default.svc.cluster.local address: 10.7.0.1 type: A	backend-listener:1
FQDN	backend-listener.default.svc.cluster.local	backend-listener.default.svc.cluster.local. backend-listener.default.svc.cluster.local.default.svc.cluster.local backend-listener.default.svc.cluster.local.svc.cluster.local backend-listener.default.svc.cluster.local.cluster.local	name: backend-listener.default.svc.cluster.local. address:10.7.0.1 type: A	backend-listener.default.svc.cluster.local:1
FQDN	foo.example.com	foo.example.com. foo.example.com.default.svc.cluster.local foo.example.svc.cluster.local foo.example.cluster.local	name: foo.example.com. address: 10.7.0.1	foo.example.com:1
Absolute FQDN	foo.example.com.	foo.example.com.	name: foo.example.com. address: 10.7.0.1	foo.example.com.:1

Verification steps

• Checkout master branch
• Start development environment

make development

• Create a apicast-config.json file with the following content

cat <<EOF >apicast-config.json
{
    "services": [
        {
            "id": "1",
            "backend_version": "1",
            "proxy": {
                "hosts": [
                    "one"
                ],
                "api_backend": "https://echo-api.3scale.net:443",
                "backend": {
                    "endpoint": "http://127.0.0.1:8081",
                    "host": "backend"
                },
                "policy_chain": [
                    {
                        "name": "apicast.policy.apicast"
                    }
                ],
                "proxy_rules": [
                    {
                        "http_method": "GET",
                        "pattern": "/",
                        "metric_system_name": "hits",
                        "delta": 1,
                        "parameters": [],
                        "querystring_parameters": {}
                    }
                ]
            }
        }
    ]
} 
EOF

• Start APIcast

THREESCALE_DEPLOYMENT_ENV=staging APICAST_LOG_LEVEL=debug APICAST_WORKER=1 APICAST_CONFIGURATION_LOADER=lazy APICAST_CONFIGURATION_CACHE=0 THREESCALE_CONFIG_FILE=apicast-config.json ./bin/apicast

• Capture APIcast IP

APICAST_IP=$(docker inspect apicast_build_0-development-1 | yq e -P '.[0].NetworkSettings.Networks.apicast_build_0_default.IPAddress' -)

• Send request

curl -i -k -H "Host: one" -H "Accept: application/json" "http://${APICAST_IP}:8080/?user_key="

• Send another request

curl -i -k -H "Host: one" -H "Accept: application/json" "http://${APICAST_IP}:8080/?user_key="

• Check the log, you should see something like the follow

22 resolver.lua:350: lookup(): resolver query: echo-api.3scale.net, requestID=41877adab1e263e2d68f62c699a4aea2                                          
22 cache.lua:115: fetch_answers(): resolver cache read echo-api.3scale.net 1 entries, requestID=41877adab1e263e2d68f62c699a4aea2                        
22 cache.lua:115: fetch_answers(): resolver cache read echo-api.pro-saas.3sca.net 3 entries, requestID=41877adab1e263e2d68f62c699a4aea2                 
22 cache.lua:190: get(): resolver cache hit: echo-api.3scale.net 3.213.166.152, 54.157.201.86, 52.72.133.221, requestID=41877adab1e263e2d68f62c699a4aea2
22 resolver.lua:370: lookup(): resolver query: echo-api.3scale.net finished with 4 answers, requestID=41877adab1e263e2d68f62c699a4aea2                  

• Stop APIcast

CTRL-C

• Checkout this branch
• Start APIcast again

THREESCALE_DEPLOYMENT_ENV=staging APICAST_LOG_LEVEL=debug APICAST_WORKER=1 APICAST_CONFIGURATION_LOADER=lazy APICAST_CONFIGURATION_CACHE=0 THREESCALE_CONFIG_FILE=apicast-config.json ./bin/apicast

• Send 2 requests
• Check the log, it should use key with <qname:qtype> format

cache.lua:130: fetch_answers(): resolver cache read echo-api.3scale.net:1 3 entries, requestID=a48f515c452e71c0fafb79c1f75d0205                        
cache.lua:205: get(): resolver cache hit: echo-api.3scale.net:1 3.213.166.152, 54.157.201.86, 52.72.133.221, requestID=a48f515c452e71c0fafb79c1f75d0205
resolver.lua:405: get_servers(): resolver query: echo-api.3scale.net finished with 3 answers, requestID=a48f515c452e71c0fafb79c1f75d0205