Kong 3.7: Upstream Hash Fallback Issue - All Requests Route to First Target When Hash on header is Missing

[zhou-cl]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.7.1

Current Behavior

Hi Kong Community,
I'm encountering an issue with Kong 3.7's upstream hash configuration. I'm hoping someone can help me understand why all requests are being routed to the first target when the Hash on header header is missing.
Environment:
Kong Version: 3.7
Upstream Configuration:
{
"hash_fallback": "none",
"name": "admin-scm_Upstream_testaws",
"hash_on_header": "pp-depot-hashid",
"hash_on_query_arg": null,
"id": "1633806f-163a-46dd-ad34-0611cdd1a0ee",
"hash_on_uri_capture": null,
"hash_on_cookie": null,
"hash_on_cookie_path": "/",
"hash_fallback_query_arg": null,
"hash_fallback_uri_capture": null,
"host_header": null,
"client_certificate": null,
"use_srv_name": false,
"tags": ["k8s-1739002503300704098"],
"algorithm": "consistent-hashing",
"slots": 10000,
"hash_on": "header",
"hash_fallback_header": "pp-userid",
"created_at": 1589536188,
"updated_at": 1739002503
}
I have three targets configured for this upstream.
When the pp-depot-hashid header is missing, all requests are being routed to the first target.
This behavior does not occur in Kong 1.5.1, where requests are distributed among all targets when pp-depot-hashid is missing.
Expected Behavior:
When pp-depot-hashid is missing, requests should be distributed among all targets using the default load balancing strategy (e.g., round-robin).
Actual Behavior:
All requests are routed to the first target when pp-depot-hashid is missing.
Steps to Reproduce:
Configure an upstream with hash_on_header: pp-depot-hashid and hash_fallback: none.
Add three targets to the upstream.
Send requests without the pp-depot-hashid header.
Observe that all requests are routed to the first target.
Questions:
Is this a known issue in Kong 3.7?
Are there any configuration changes or workarounds to ensure requests are distributed among all targets when pp-depot-hashid is missing?
Why does this behavior differ from Kong 1.5.1?
Thank you in advance for any insights or assistance!

[xianghai2]

Hi

There have been significant changes for the "consistent-hashing" algorithm since Kong 1.5.1, below are details:

• Kong 1.5.1 was based on "resty.dns.balancer.ring", same as "round-robin" algorithm. ( https://github.com/Kong/kong/blob/1.5.1/kong/runloop/balancer.lua#L211 )
• Kong 2.2.0-beta.1 was changed to "resty.dns.balancer.consistent_hashing". ( https://github.com/Kong/kong/blob/2.2.0-beta.1/kong/runloop/balancer.lua#L224 ）
• Kong 2.6.0 was changed to "kong.runloop.balancer.consistent_hashing". （ https://github.com/Kong/kong/blob/2.6.0/kong/runloop/balancer/balancers.lua#L109 ）

With recent versions of Kong, the "hash_on_header" field is required when "hash_on" is "header", which means it should not be missed.

And you may set "algorithm" to "round-robin" or set "hash_on" to "none" if a round-robin behavior is desired?

References:
https://docs.konghq.com/gateway/latest/how-kong-works/load-balancing/#advanced-load-balancing
https://docs.konghq.com/gateway/api/admin-ee/3.7.0.x/#/Upstreams/create-upstream
https://github.com/Kong/kong/blob/3.7.0/kong/db/schema/entities/upstreams.lua

[zhou-cl]

Hi

There have been significant changes for the "consistent-hashing" algorithm since Kong 1.5.1, below are details:

• Kong 1.5.1 was based on "resty.dns.balancer.ring", same as "round-robin" algorithm. ( https://github.com/Kong/kong/blob/1.5.1/kong/runloop/balancer.lua#L211 )
• Kong 2.2.0-beta.1 was changed to "resty.dns.balancer.consistent_hashing". ( https://github.com/Kong/kong/blob/2.2.0-beta.1/kong/runloop/balancer.lua#L224 ）
• Kong 2.6.0 was changed to "kong.runloop.balancer.consistent_hashing". （ https://github.com/Kong/kong/blob/2.6.0/kong/runloop/balancer/balancers.lua#L109 ）

With recent versions of Kong, the "hash_on_header" field is required when "hash_on" is "header", which means it should not be missed.

And you may set "algorithm" to "round-robin" or set "hash_on" to "none" if a round-robin behavior is desired?

References: https://docs.konghq.com/gateway/latest/how-kong-works/load-balancing/#advanced-load-balancing https://docs.konghq.com/gateway/api/admin-ee/3.7.0.x/#/Upstreams/create-upstream https://github.com/Kong/kong/blob/3.7.0/kong/db/schema/entities/upstreams.lua

Hello,
Based on the information provided, I would like to ask if the value of the hash_on_header field cannot be empty. If the value of the hash_on_header field, such as userid, is set to none, what load balancing strategy will be executed? Will it not execute the default round-robin strategy?
Thank you.

[zhou-cl]

hello
For example, hash_on="header",
hash_on_header = "X-Request-ID",
hash_fallback = "header",
hash_fallback_header = "User-Agent", What I mean is that in actual requests, if the value of X-Request-ID is empty, what load balancing strategy will be executed, and if the value of User Agent is also empty, what load balancing strategy will be executed
thanks、

[xianghai2]

hello For example, hash_on="header", hash_on_header = "X-Request-ID", hash_fallback = "header", hash_fallback_header = "User-Agent", What I mean is that in actual requests, if the value of X-Request-ID is empty, what load balancing strategy will be executed, and if the value of User Agent is also empty, what load balancing strategy will be executed thanks、

it depends on a few factors.
with this example, assume "consistent-hashing" algorithm, it will seek for "User-Agent" header for hashing if the value of X-Request-ID is missing. And it will report error if both header values are missing.

References:
https://github.com/Kong/kong/blob/3.7.0/kong/runloop/balancer/init.lua#L107
https://github.com/Kong/kong/blob/3.7.0/kong/runloop/balancer/init.lua#L347

https://github.com/Kong/kong/blob/3.7.0/kong/runloop/balancer/consistent_hashing.lua#L152

[zhou-cl]

hello For example, hash_on="header", hash_on_header = "X-Request-ID", hash_fallback = "header", hash_fallback_header = "User-Agent", What I mean is that in actual requests, if the value of X-Request-ID is empty, what load balancing strategy will be executed, and if the value of User Agent is also empty, what load balancing strategy will be executed thanks、

it depends on a few factors. with this example, assume "consistent-hashing" algorithm, it will seek for "User-Agent" header for hashing if the value of X-Request-ID is empty. And it will report error if both header values are empty.

References: https://github.com/Kong/kong/blob/3.7.0/kong/runloop/balancer/init.lua#L107 https://github.com/Kong/kong/blob/3.7.0/kong/runloop/balancer/init.lua#L347

https://github.com/Kong/kong/blob/3.7.0/kong/runloop/balancer/consistent_hashing.lua#L152

What I mean is that in actual requests, if the value of X-Request-ID is empty, what load balancing strategy will be executed, and if the value of User Agent is also empty, what load balancing strategy will be executed ,Is it an error request failure or will the request node be executed according to what strategy

thanks、

[zhou-cl]

The situation I am currently facing is that only Hash on and Hash on header are set. When the field set in Hash on header has no value, it seems to default to null. The hash calculation only reaches one node and there is no error exit request. Does it mean that when the value of Hash on header is empty, if the Hash fallback header is also empty, an error will be reported, instead of calculating to one node with a value of null

[zhou-cl]

I see hash-value=get-value_to-hash (upstream, ctx) or ""
If get-value_to-hash returns nil, hash-value will be set to the empty string ''.
This means that even if the hash field is empty, no error will be thrown and execution will continue.

[xianghai2]

I see hash-value=get-value_to-hash (upstream, ctx) or "" If get-value_to-hash returns nil, hash-value will be set to the empty string ''. This means that even if the hash field is empty, no error will be thrown and execution will continue.

right.

The empty value of a header is a different thing with a missing header. The "" value will be used for hash calculation, that may lead to the same hash value, and then the same target node.

An error will be reported if Hash on header is missed. ( will report error if both Hash on header and Hash fallback header are missed when hash fallback is configure as 'header' )

[zhou-cl]

I see hash-value=get-value_to-hash (upstream, ctx) or "" If get-value_to-hash returns nil, hash-value will be set to the empty string ''. This means that even if the hash field is empty, no error will be thrown and execution will continue.

right.

The empty value of a header is a different thing with a missing header. The "" value will be used for hash calculation, that may lead to the same hash value, and then the same target node.

An error will be reported if Hash on header is missed. ( will report error if both Hash on header and Hash fallback header are missed when hash fallback is configure as 'header' )

But if it is empty, won't it return an empty string? In what scenario will it return nil?

[xianghai2]

Hi

Regarding the nil case, get_value_to_hash() will return nil if hash_on is 'none‘ or headers are missed.

Prior to version 2.6.1, it will use the returned nil as it is, and this will trigger an error. ( https://github.com/Kong/kong/blob/2.6.0/kong/runloop/balancer/init.lua#L215 )

And with a fix since version 2.6.1, the hash_value will be set to empty string if nil is returned. ( 8ea508b )

[zhou-cl]

hello
When was the default round robin canceled when the header was empty? Is there any way to downgrade to round robin if a Consistent hashing miss occurs

[xianghai2]

Hi,
No way to downgrade to round robin as the your upstream was configured the "algorithm" as "consistent-hashing".

[zhou-cl]

thanks

你好， 无法降级到循环，因为你的上游将“算法”配置为“一致性哈希”。