Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metric name consistent hashing distribution #31

Open
anatolijd opened this issue Jul 28, 2021 · 3 comments
Open

metric name consistent hashing distribution #31

anatolijd opened this issue Jul 28, 2021 · 3 comments

Comments

@anatolijd
Copy link

anatolijd commented Jul 28, 2021

hi,

we are having issue with statsrelay - how it distributes metrics between statsite hosts. We run 4 statsrelay servers, and 5 statsite servers at the baclkend. All are configured identically.

It appears, that one statsite node is always getting higher amount of metrics, look at the is-1941b :

statsrelay

We tried to add INSTANCE values (in 'HOST:PORT:INSTANCE') , but it did not make any visible effect.

Our current idea is that consistent hashing is not distributing metrics in efficient way.

We sometimes use very long metric names, for example:
env.analyze.document_analyzer.analyzeDocument.entities.InternetDomainName.source.DataProviderNameHERE.DocumentNameHashHere.occurrences

In our case, it looks like all metric names prefixed env.analyze.document_analyzer.analyzeDocument.entities are forwarded to the same single statsite backend. Which is not good, we would expect those metric names to be distributed between different statsite backends.

Any ideas how to solve ?

@jjneely
Copy link
Owner

jjneely commented Jul 28, 2021 via email

@anatolijd
Copy link
Author

here it is:

/opt/statsrelay/statsrelay --bind 0.0.0.0 --port 8125 --packetlen 8000 --prefix statsrelay.it-s-0f5c57688efa79273 --pprof --pprof-bind :8123 --sendproto UDP \
is-1935b:8125 is-1939b:8125 is-1940b:8125 is-1941b:8125 is-1942b:8125

.
.

Yesterday, we changed it a bit by populating hash ring with multiple aliases:

/opt/statsrelay/statsrelay --bind 0.0.0.0 --port 8125 --packetlen 8000 --prefix statsrelay.it-s-0f5c57688efa79273 --pprof --pprof-bind :8123 --sendproto UDP \
is-1935b:8125:1   is-1939b:8125:2   is-1940b:8125:3   is-1941b:8125:4   is-1942b:8125:5 \
is-1935b:8125:10  is-1939b:8125:20  is-1940b:8125:30  is-1941b:8125:40  is-1942b:8125:50 \
is-1935b:8125:100 is-1939b:8125:200 is-1940b:8125:300 is-1941b:8125:400 is-1942b:8125:500

, and we've got a better distribution:
statsreray-2

@jjneely
Copy link
Owner

jjneely commented Jul 30, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants