Serve stale content when DNS lookup fails #8484

serrislew · 2021-11-01T21:07:24Z

Proxies should serve stale content even if DNS lookup to OS is unsuccessful.

For hierarchical caching, if a child proxy matches successfully to a parent proxy and child has requested stale object in its cache, it should not need to make a DNS request for the origin.
- It can validate with the parent and return the object as expected.
Without hierarchical caching, if stale content is being requested from a proxy and the DNS lookup to the OS is unsuccessful, it should also serve the stale object
- this behavior is similar to when the origin server is down and the proxy serves stale content with a warning header

These changes still respect the max_stale_age configurations and do not serve stale content pass the expiration date.

Autest tests the following using PUSH command to store stale content in proxy's cache:

With child mapping to an origin server without a DNS entry and child matching successfully to parent, a request from child after content becomes stale returns 200
- After max_stale_age has passed, request from child returns 500 as stale content is no longer returnable
Without hierarchical caching and proxy mapping to an origin server without a DNS entry, a request from proxy after content becomes stale returns 200
- After max_stale_age has passed, request from proxy returns 500 as stale content is no longer returnable

ywkaras · 2021-11-19T19:54:08Z

tests/gold_tests/proxy_protocol/proxy_serve_stale_dns_fail.test.py

+
+# Object to push to proxies
+stale_5 = "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public, max-age=5\n\nCACHED"
+stale_30 = "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public, max-age=30\n\nCACHED"


Could this be changed to max-age=10 (or other value less than 30)? @zwoop may only give you a fruit cake for X-mas if you make Au regression test runs take 30 seconds longer.

ywkaras · 2021-12-10T22:08:00Z

tests/gold_tests/proxy_protocol/proxy_serve_stale_dns_fail.test.py

+    f'map http://localhost:{ts_parent.Variables.port} {server_name}'
+)
+ts_parent.Disk.remap_config.AddLine(
+    f'map {server_name} {server_name}'


Why do you need this?

I am testing for non-hierarchical scenarios as well so the ts_parent server is mapped to the OS and gets called directly

Alternatively you can also do:

trafficserver/tests/gold_tests/pluginTest/test_hooks/test_hooks.test.py

Line 50 in 5774921

'proxy.config.url_remap.remap_required': 0,

But the way you have it is fine.

ywkaras · 2021-12-10T22:09:51Z

I forget now, did you say Apple is already running this on production proxies?

ywkaras · 2021-12-11T00:27:46Z

tests/gold_tests/proxy_protocol/proxy_serve_stale_dns_fail.test.py

+    'proxy.config.http.parent_proxy.self_detect': 0,
+})
+ts_child.Disk.parent_config.AddLine(
+    f'dest_domain=. parent=localhost:{ts_parent.Variables.port} round_robin=consistent_hash go_direct=false'


So even with this configured, child TS will still do a DNS lookup on unkown.domain.com ? Did you see the DNS lookup in the http debug trace output?

The http debug trace output shows the child doing PPDNS lookup to the parent and the parent doing a DNS lookup to unknown.domain.com (OS). After the DNS lookup fails for the parent but the parent has the stale returnable object in its cache, the child will revalidate from the parent (without doing a DNS lookup) and return the object to the client. Let me know if you are seeing something different in the debug trace output!

Lately we've been including DNS servers in our Au tests:

trafficserver/tests/gold_tests/next_hop/strategies_ch/strategies_ch.test.py

Line 71 in 5774921

'proxy.config.dns.nameservers': f"127.0.0.1:{dns.Variables.Port}", # Only nameservers if resolv_conf NULL.

I don't understand why ATS would do a DNS lookup on the request host when it's not using the request host as the next hop. But I suppose that's beyond the scope of this PR.

ywkaras · 2021-12-14T23:49:49Z

tests/gold_tests/proxy_protocol/proxy_serve_stale_dns_fail.test.py

+ts_parent = Test.MakeATSProcess("ts_parent")
+server_name = "http://unknown.domain.com/"
+
+Test.testName = "STALE"


Most of our Au tests don't assigned testName, I'm not sure what it's for:

wkaras ~/REPOS/TS3/tests/gold_tests M$ find . -name \*.test.py | wc -l 239 wkaras ~/REPOS/TS3/tests/gold_tests M$ fgrep Test.testName $( find . -name \*.test.py ) | wc -l 44 wkaras ~/REPOS/TS3/tests/gold_tests M$

ywkaras · 2022-01-04T01:38:33Z

We discussed this in today's meeting, it turns out microdns is trickier to use than I thought, and not well-documented, so approving this as is.

* serve stale when dns lookups fails * updated max-age in autest to run shorter Co-authored-by: Serris Lew <lserris@apple.com> (cherry picked from commit 85be282)

zwoop · 2022-04-11T22:19:21Z

Cherry-picked to v9.2.x

* asf/9.2.x: Updated ChangeLog Serve stale content when DNS lookup fails (apache#8484) Reuse TSMutex for ts_lua_http_intercept_handler (apache#8687) include <cstring> for access to C-string operations (apache#8786) Propagate proxy.config.net.sock_option_flag_in to newly accepted connections (apache#8784)

Delete debian/patches/0014-serve-stale-when-dns-lookup-fails.patch since it was cherry-picked to 9.2.x apache/trafficserver#8484 (comment)

* serve stale when dns lookups fails * updated max-age in autest to run shorter Co-authored-by: Serris Lew <lserris@apple.com> (cherry picked from commit 85be282)

serve stale when dns lookups fails

e6e492d

serrislew requested review from bryancall and zwoop as code owners November 1, 2021 21:07

bryancall assigned serrislew Nov 1, 2021

bryancall requested a review from SolidWallOfCode November 1, 2021 23:07

bryancall added Cache New Feature labels Nov 1, 2021

bryancall added this to the 10.0.0 milestone Nov 1, 2021

masaori335 mentioned this pull request Nov 18, 2021

Compile warnings by LLVM 13 #8432

Closed

ywkaras reviewed Nov 19, 2021

View reviewed changes

updated max-age in autest to run shorter

407c5cb

ywkaras reviewed Dec 10, 2021

View reviewed changes

ywkaras reviewed Dec 11, 2021

View reviewed changes

ywkaras reviewed Dec 14, 2021

View reviewed changes

ywkaras approved these changes Jan 4, 2022

View reviewed changes

vmamidi merged commit 85be282 into apache:master Jan 11, 2022

vmamidi mentioned this pull request Jan 11, 2022

Child makes DNS request for stale objects #2376

Closed

bneradt mentioned this pull request Jan 19, 2022

cache-request-method AuTest fails on master #8616

Closed

zwoop pushed a commit that referenced this pull request Apr 11, 2022

Serve stale content when DNS lookup fails (#8484)

b1eed52

* serve stale when dns lookups fails * updated max-age in autest to run shorter Co-authored-by: Serris Lew <lserris@apple.com> (cherry picked from commit 85be282)

zwoop modified the milestones: 10.0.0, 9.2.0 Apr 11, 2022

Serve stale content when DNS lookup fails #8484

Serve stale content when DNS lookup fails #8484

Uh oh!

Conversation

serrislew commented Nov 1, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ywkaras commented Dec 10, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serrislew Dec 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ywkaras commented Jan 4, 2022

Uh oh!

zwoop commented Apr 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

serrislew Dec 14, 2021 •

edited

Loading