Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Queue Groups on leaf clusters not balancing correctly when messages are routed in from hub cluster [v2.10.22] - Variant 2 #6040

Closed
roeschter opened this issue Oct 24, 2024 · 3 comments · Fixed by #6043
Assignees
Labels
defect Suspected defect such as a bug or regression

Comments

@roeschter
Copy link

Observed behavior

Queue Groups on leaf clusters not balancing (only one subscriber receiving messages) in certain assymetrically connected leaf cluster

Expected behavior

Queue Groups always balance when all subs connection in the same leaf cluster

See also: #5972

Server and client version

Server 2.10.22
nat-cli from main

Host environment

Windows (Michael) - Linux (customer)

Steps to reproduce

Set up a hub cluster with nodes HUB1, HUB2, HUB3
Set up a leaf cluster with nodes LEAF1, LEAF2, LEAF3
Connect the leaf nodes as follows: (LEAF1 and LEAF2 connect to the SAME Hub - this is NOT a typo)
LEAF1 → HUB1
LEAF2 → HUB1
LEAF3 → HUB3
Start queue group listeners on LEAF1 and LEAF2
nats --context LEAF1 sub --queue=q1 foo
nats --context LEAF2 sub --queue=q1 foo

Publish to HUB3
nats --context HUB3 pub foo Hello

@roeschter
Copy link
Author

Config
20241022_queue_group_issue_2.zip

@kozlovic
Copy link
Member

@roeschter I believe I fixed the issue in #6043. If you want to give it a shot.

derekcollison added a commit that referenced this issue Oct 25, 2024
…des (#6043)

When receiving a message from a route, if the queue interest is only
routes and leaf connections, the server will always favor the leaf kind.
However, if there were more than one leaf connections for the same queue
group, the server will always pick the same (the last one). It would
change if new leaf were to connect or disconnect/reconnect, but with a
stable set, it would always be the same.

This PR makes this selection somewhat random.

Resolves #6040

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
@roeschter
Copy link
Author

Question from end user. Whats the approximate release of this in 2.10? (2.11.23?)

neilalexander pushed a commit that referenced this issue Oct 29, 2024
When receiving a message from a route, if the queue interest is
only routes and leaf connections, the server will always favor
the leaf kind. However, if there were more than one leaf connections
for the same queue group, the server will always pick the same
(the last one). It would change if new leaf were to connect or
disconnect/reconnect, but with a stable set, it would always
be the same.

This PR makes this selection somewhat random.

Resolves #6040

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
neilalexander pushed a commit that referenced this issue Oct 29, 2024
When receiving a message from a route, if the queue interest is
only routes and leaf connections, the server will always favor
the leaf kind. However, if there were more than one leaf connections
for the same queue group, the server will always pick the same
(the last one). It would change if new leaf were to connect or
disconnect/reconnect, but with a stable set, it would always
be the same.

This PR makes this selection somewhat random.

Resolves #6040

Signed-off-by: Ivan Kozlovic <ivan@synadia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
defect Suspected defect such as a bug or regression
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants