strict round robin parent select doesn't work as expected

[jvgutierrez]

We've identified two issues regarding strict parent proxy select policy.
Our parent.config looks like this:

dest_domain=. parent="10.132.0.112:3120,10.132.0.112:3121,10.132.0.112:3122,10.132.0.112:3123,10.132.0.112:3124,10.132.0.112:3125,10.132.0.112:3126,10.132.0.112:3127" parent_is_proxy=false round_robin=strict

on long-lived ATS instances ss shows that all the traffic is being handled by the first parent. This seems to be related to the fact that rr_next (declared as uint32_t in https://github.com/apache/trafficserver/blob/master/proxy/ParentSelection.h#L143) gets casted as int32_t in https://github.com/apache/trafficserver/blob/master/proxy/ParentRoundRobin.cc#L102 and as soon as the value overflows, cur_index always begins on a negative value, returning always the parent with index 0:

Jan 14 17:40:41 cp4027 traffic_manager[20323]: [Jan 14 17:40:41.847] {0x2ade5d766700} DEBUG: <ParentRoundRobin.cc:61 (selectParent)> (parent_select) In ParentRoundRobin::selectParent(): Using a round robin parent selection strategy.
Jan 14 17:40:41 cp4027 traffic_manager[20323]: [Jan 14 17:40:41.847] {0x2ade5d766700} DEBUG: <ParentRoundRobin.cc:140 (selectParent)> (parent_select) cur_index: -4, result->start_parent: -4
Jan 14 17:40:41 cp4027 traffic_manager[20323]: [Jan 14 17:40:41.847] {0x2ade5d766700} DEBUG: <ParentRoundRobin.cc:140 (selectParent)> (parent_select) cur_index: -3, result->start_parent: -4
Jan 14 17:40:41 cp4027 traffic_manager[20323]: [Jan 14 17:40:41.847] {0x2ade5d766700} DEBUG: <ParentRoundRobin.cc:140 (selectParent)> (parent_select) cur_index: -2, result->start_parent: -4
Jan 14 17:40:41 cp4027 traffic_manager[20323]: [Jan 14 17:40:41.847] {0x2ade5d766700} DEBUG: <ParentRoundRobin.cc:140 (selectParent)> (parent_select) cur_index: -1, result->start_parent: -4
Jan 14 17:40:41 cp4027 traffic_manager[20323]: [Jan 14 17:40:41.847] {0x2ade5d766700} DEBUG: <ParentRoundRobin.cc:140 (selectParent)> (parent_select) cur_index: 0, result->start_parent: -4

On short-lived ATS instance, the debug output looks as expected:

Jan 14 18:10:41 cp5012 traffic_manager[48369]: [Jan 14 18:10:41.335] {0x2b4690202700} DEBUG: <ParentRoundRobin.cc:172 (selectParent)> (parent_select) Chosen parent = 10.132.0.112.3120
Jan 14 18:10:41 cp5012 traffic_manager[48369]: [Jan 14 18:10:41.335] {0x2b4690202700} DEBUG: <ParentRoundRobin.cc:172 (selectParent)> (parent_select) Chosen parent = 10.132.0.112.3121
Jan 14 18:10:43 cp5012 traffic_manager[48369]: [Jan 14 18:10:43.890] {0x2b46905e9700} DEBUG: <ParentRoundRobin.cc:172 (selectParent)> (parent_select) Chosen parent = 10.132.0.112.3122
Jan 14 18:10:43 cp5012 traffic_manager[48369]: [Jan 14 18:10:43.890] {0x2b46905e9700} DEBUG: <ParentRoundRobin.cc:172 (selectParent)> (parent_select) Chosen parent = 10.132.0.112.3123
Jan 14 18:10:44 cp5012 traffic_manager[48369]: [Jan 14 18:10:44.970] {0x2b46908e9700} DEBUG: <ParentRoundRobin.cc:172 (selectParent)> (parent_select) Chosen parent = 10.132.0.112.3124
Jan 14 18:10:44 cp5012 traffic_manager[48369]: [Jan 14 18:10:44.970] {0x2b46908e9700} DEBUG: <ParentRoundRobin.cc:172 (selectParent)> (parent_select) Chosen parent = 10.132.0.112.3125
Jan 14 18:10:45 cp5012 traffic_manager[48369]: [Jan 14 18:10:45.405] {0x2b4690a8c700} DEBUG: <ParentRoundRobin.cc:172 (selectParent)> (parent_select) Chosen parent = 10.132.0.112.3126
Jan 14 18:10:45 cp5012 traffic_manager[48369]: [Jan 14 18:10:45.405] {0x2b4690a8c700} DEBUG: <ParentRoundRobin.cc:172 (selectParent)> (parent_select) Chosen parent = 10.132.0.112.3127

but ss only shows traffic hitting the odd parents:

vgutierrez@cp5012:~$ for port in {3120..3127}; do ss  "( dport = $port or sport = $port )" |wc -l; done
1
28
1
40
2
40
1
35

[jvgutierrez]

From slack conversations with @jrushford, the issue described affecting short-lived ATS instances (traffic only hitting odd parents) seems to be related to proxy.config.http.no_dns_just_forward_to_parent being enabled, disabling it fixes the issue on ATS 8.0.5

[jrushford]

@jvgutierrez apparently it's caused by this section of code HttpTransact.cc:1443. If you change your parent.config to use hostnames rather than the ip's, the problem goes away. Trying to understand what's the flow, here in HttpTransact. It looks like it copies the previously used address info maybe rather than the selected addess.

[jrushford]

With proxy.config.http.no_dns_just_forward_to_parent enabled and parents in parent.config are ip's, RoundRobin::selectParent() is getting called twice per transaction. See this debug:

[Jan 15 19:40:51.577] [ET_NET 35] DEBUG: <ParentRoundRobin.cc:180 (selectParent)> (parent_select) Chosen parent = ipcdn-cache-59.cdnlab.foo.com.3126
[Jan 15 19:40:51.577] [ET_NET 35] DEBUG: <ParentRoundRobin.cc:180 (selectParent)> (parent_select) Chosen parent = ipcdn-cache-59.cdnlab.foo.com.3127
[Jan 15 19:40:51.577] [ET_NET 35] DEBUG: <HttpTransact.cc:1643 (PPDNSLookup)> (http_trans) [23] [PPDNSLookup] DNS lookup for sm_id[23] successful IP: 69.252.122.50:3127
[Jan 15 19:40:53.842] [ET_NET 37] DEBUG: <ParentRoundRobin.cc:180 (selectParent)> (parent_select) Chosen parent = ipcdn-cache-59.cdnlab.foo.com.3120
[Jan 15 19:40:53.843] [ET_NET 37] DEBUG: <ParentRoundRobin.cc:180 (selectParent)> (parent_select) Chosen parent = ipcdn-cache-59.cdnlab.foo.com.3121
[Jan 15 19:40:53.843] [ET_NET 37] DEBUG: <HttpTransact.cc:1643 (PPDNSLookup)> (http_trans) [24] [PPDNSLookup] DNS lookup for sm_id[24] successful IP: 69.252.122.50:3121
[Jan 15 19:40:56.106] [ET_NET 38] DEBUG: <ParentRoundRobin.cc:180 (selectParent)> (parent_select) Chosen parent = ipcdn-cache-59.cdnlab.foo.com.3122
[Jan 15 19:40:56.107] [ET_NET 38] DEBUG: <ParentRoundRobin.cc:180 (selectParent)> (parent_select) Chosen parent = ipcdn-cache-59.cdnlab.foo.com.3123
[Jan 15 19:40:56.107] [ET_NET 38] DEBUG: <HttpTransact.cc:1643 (PPDNSLookup)> (http_trans) [25] [PPDNSLookup] DNS lookup for sm_id[25] successful IP: 69.252.122.50:3123``

[vmamidi]

That implies that when proxy.config.http.no_dns_just_forward_to_parent is enabled, SM is rewound from a wrong state.

[vmamidi]

@jvgutierrez apparently it's caused by this section of code HttpTransact.cc:1443. If you change your parent.config to use hostnames rather than the ip's, the problem goes away. Trying to understand what's the flow, here in HttpTransact. It looks like it copies the previously used address info maybe rather than the selected addess.

State Machine runs differently based on the existence of IP-based rules in hosting/parent configurations.

[jrushford]

@jvgutierrez Okay, I found the problem. When proxy.config.http.no_dns_just_forward_to_parent is enabled, HttpTransact::HandleResponse() will call the ParentConfigParams::parentExists to verify if there is a parent proxy to forward the request too. parentExists() calls findParent todo the verification which increments the index iterator on the ParentRoundRobin instance then parentExists() just verifies that the result is PARENT_SPECIFIED, nothing further is done with this result instead when necessary to go upstream, findParent is called again which again increments the index iterator in ParentRoundRobin which results in only every other parent in the round robin list being used.

Working on a fix for this.