Receive: Allow setting hashing algorithm per tenant in hashrings config #5653

haanhvu · 2022-08-28T21:28:31Z

Signed-off-by: haanhvu haanh6594@gmail.com

fixes #5567

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

Receive: Allow setting hashing algorithm per tenant in hashrings config

Verification

pkg/receive/hashring.go

haanhvu · 2022-08-29T12:21:01Z

@fpetkovski could you pls review if this is what you mean in the issue?

fpetkovski

Thanks, this looks good to me 👍

pkg/receive/hashring_test.go

yeya24 · 2022-09-03T21:21:14Z

pkg/receive/hashring.go

+	case AlgorithmKetama:
+		return newKetamaHashring(endpoints, SectionsPerNode, replicationFactor)
+	default:
+		return simpleHashring(endpoints)


If an invalid algorithm is specified, do we just fallback to the hashmod algorithm?
I feel a better way is to either:

Return non-exist error and let users know this is wrong.

Fall back to the default one. But have a log to tell that the specified algorithm is wrong.

Those 2 options are better than ignoring the error.

@yeya24 I think Option 1 is better. Thought of that myself. But that would change the logic of the current newMultiHashring() a little bit. Would that cause a backward compatibility problem?

I see that we have an errors package at https://github.com/thanos-io/thanos/tree/main/pkg/errors but that doesn't seem to be used much? If we go with option 1, should I use that one or the github.com/pkg/errors one?

@fpetkovski @matej-g

What would be the behavior though? Right now we do not return error as hashring creation (so to speak) "cannot" fail. Should we return error and ignore a hashring with invalid algorithm? Or how would we like to handle this?

This is why option 2 looked better, since we can just warn and fall back.

When I look at this issue now, I feel either way is okay to me.
If we go with option 1, even if it is wrong, it is just one-off. If we go with option 2, it is also fine to me as this is better than current behavior.

matej-g

Would be good to address @yeya24's comment, otherwise looks good!

matej-g · 2022-09-12T12:51:01Z

CHANGELOG.md

@@ -15,6 +15,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
 - [#5655](https://github.com/thanos-io/thanos/pull/5655) Receive: Fix recreating already pruned tenants.

 ### Added
+- [#5653](https://github.com/thanos-io/thanos/pull/5653) Receive: Allow setting hashing algorithm per tenant in hashrings config


Nit: Can we specify that the hashring config from file has precedence over the flag?

Yeah I'm thinking we can have a log telling where we take the algorithm from.

Also, we need to edit the receive.md doc and the description of receive.hashrings-algorithm flag (anything else?)

Where can I edit the description of the receive.hashrings-algorithm flag?

In here: https://github.com/thanos-io/thanos/blob/main/cmd/thanos/receive.go#L820, I believe the .md is automatically generated based on this description

haanhvu · 2022-09-13T02:02:54Z

Just thought of this case, currently if users specify the algorithm as Ketama, not ketama, we would consider it invalid and fall back to hashmod?

thanos/pkg/receive/hashring.go

Lines 26 to 27 in fb11fc7

    
           AlgorithmHashmod HashringAlgorithm = "hashmod" 
        
           AlgorithmKetama  HashringAlgorithm = "ketama"

@matej-g @fpetkovski @yeya24

matej-g · 2022-09-13T10:10:32Z

Just thought of this case, currently if users specify the algorithm as Ketama, not ketama, we would consider it invalid and fall back to hashmod?

Preferably we should convert the string to lowercase (if this is not already done by the library we use for flags?) and accept it, more user friendly.

fpetkovski · 2022-09-13T11:20:54Z

I am not sure I like where this rabbit hole is taking us :)

What about adding validation here so that we can take the guesswork out: https://github.com/thanos-io/thanos/blob/main/pkg/receive/hashring.go#L289? That should resolve @yeya24's comment if I'm not mistaken.

haanhvu · 2022-09-25T18:21:16Z

I made all the required changes. There're still some failed checks because I'm having a problem with Makefile (go version problem). Will look into it tomorrow.

sonatype-lift · 2022-09-25T18:23:00Z

pkg/receive/hashring.go

@@ -296,6 +292,21 @@ func HashringFromConfig(algorithm HashringAlgorithm, replicationFactor uint64, c
 	if len(config) == 0 {
 		return nil, errors.Wrapf(err, "failed to load configuration")
 	}
+


gofmt: File is not gofmt-ed with -s

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

douglascamata · 2022-09-30T09:57:44Z

pkg/receive/hashring.go

@@ -296,6 +292,21 @@ func HashringFromConfig(algorithm HashringAlgorithm, replicationFactor uint64, c
 	if len(config) == 0 {
 		return nil, errors.Wrapf(err, "failed to load configuration")
 	}
+
+	if (algorithm != "hashmod") && (algorithm != "ketama") {
+		log.Println("The specified algorithm is incorrect. Fall back to hashmod algorithm")


Can we log more information about the hashring to help debugging & problem solving? For example, the hashring name and tenants could be useful.

Thanks for your contribution, @haanhvu 🙇

@douglascamata Do you think log.Println is good enough for all cases?

Unfortunately, I don't think so, @haanhvu. It's better to use our chosen logging library ("github.com/go-kit/log"), which might require that you send down the logger as a parameter to this function and then do something like:

# import "github.com/go-kit/log/level" level.Warning(logger).Log(...)

I think Warning is the best level as this doesn't stop Receive from starting up.

matej-g · 2022-10-27T15:09:44Z

@haanhvu any updates on this one?

haanhvu · 2022-10-27T15:12:51Z

@matej-g There're some small fixes left to do (mostly the log as @douglascamata suggested). But I need to finish #5777 first before getting back to this. Can't be in two rabbit holes at the same time...

matej-g · 2022-10-27T15:15:13Z

No worries @haanhvu, I was just sifting through PRs and wanted to know latest status, carry on and thank you 😊

haanhvu · 2023-01-03T16:21:06Z

If an invalid algorithm is specified, do we just fallback to the hashmod algorithm?
I feel a better way is to either:
Return non-exist error and let users know this is wrong.
Fall back to the default one. But have a log to tell that the specified algorithm is wrong.
Those 2 options are better than ignoring the error.

Can we log more information about the hashring to help debugging & problem solving? For example, the hashring name and tenants could be useful.

@matej-g @douglascamata @yeya24 Hey about the log, I'm not sure where to put it for now. In #5777 we edited and deleted a lot of functions in hashring.go. If these changes are merged (which will be merged soon, I think), maybe I'll need to put the hashring algorithm checking log to receive cmd or config.go, instead of hashring.go like currently.

So I think we can add the log later, after #5777 merged. Is that ok?

douglascamata · 2023-01-09T14:10:45Z

Sounds good to me. Not sure what the maintainers think.

Signed-off-by: haanhvu <haanh6594@gmail.com>

haanhvu · 2023-02-05T19:41:25Z

I made all the required changes. Please help review!

douglascamata

Neat and clean. LGTM! 🚀

yeya24

Thanks!

haanhvu · 2023-02-09T05:38:43Z

I completely forgot. Do we need to add test? Like a simple test that shows the algorithm in hashring config will overwrite the global algorithm?

douglascamata · 2023-02-09T10:06:40Z

@haanhvu tests are always welcome! :D

pull-request-size bot added the size/M label Aug 28, 2022

haanhvu changed the title ~~Issue5567~~ Receive: Allow setting hashing algorithm per tenant in hashrings config Aug 28, 2022

haanhvu marked this pull request as draft August 28, 2022 21:31

sonatype-lift bot reviewed Aug 28, 2022

View reviewed changes

pkg/receive/hashring.go Outdated Show resolved Hide resolved

sonatype-lift bot reviewed Aug 28, 2022

View reviewed changes

pkg/receive/hashring.go Outdated Show resolved Hide resolved

sonatype-lift bot reviewed Aug 28, 2022

View reviewed changes

pkg/receive/hashring.go Outdated Show resolved Hide resolved

haanhvu marked this pull request as ready for review August 28, 2022 22:05

yeya24 reviewed Aug 28, 2022

View reviewed changes

pkg/receive/hashring.go Show resolved Hide resolved

haanhvu mentioned this pull request Aug 29, 2022

Remove hardcoded labels in store shard matcher and inject unshardable label in query analyzer #5641

Merged

2 tasks

fpetkovski previously approved these changes Aug 29, 2022

View reviewed changes

haanhvu dismissed fpetkovski’s stale review via 8d82f16 August 30, 2022 14:13

haanhvu force-pushed the issue5567 branch from 8d82f16 to 5b0550d Compare August 30, 2022 14:20

haanhvu marked this pull request as draft August 30, 2022 15:05

sonatype-lift bot reviewed Aug 30, 2022

View reviewed changes

pkg/receive/hashring_test.go Outdated Show resolved Hide resolved

haanhvu force-pushed the issue5567 branch from 2cf4450 to e5d544b Compare August 30, 2022 15:24

haanhvu marked this pull request as ready for review August 30, 2022 15:26

yeya24 reviewed Sep 3, 2022

View reviewed changes

matej-g previously approved these changes Sep 9, 2022

View reviewed changes

matej-g reviewed Sep 12, 2022

View reviewed changes

haanhvu dismissed matej-g’s stale review via 8067669 September 25, 2022 17:24

haanhvu force-pushed the issue5567 branch from e5d544b to 8067669 Compare September 25, 2022 17:24

sonatype-lift bot reviewed Sep 25, 2022

View reviewed changes

douglascamata suggested changes Sep 30, 2022

View reviewed changes

haanhvu added 2 commits February 6, 2023 01:33

Fix conflicts

b758905

Signed-off-by: haanhvu <haanh6594@gmail.com>

Specify overwriting behavior in flag and add validation

0e87249

Signed-off-by: haanhvu <haanh6594@gmail.com>

haanhvu force-pushed the issue5567 branch from 38fccca to 0e87249 Compare February 5, 2023 18:39

haanhvu marked this pull request as draft February 5, 2023 18:41

Add log and doc

7930c75

Signed-off-by: haanhvu <haanh6594@gmail.com>

haanhvu marked this pull request as ready for review February 5, 2023 19:41

haanhvu requested review from douglascamata and yeya24 and removed request for douglascamata and yeya24 February 5, 2023 19:42

douglascamata approved these changes Feb 6, 2023

View reviewed changes

yeya24 approved these changes Feb 7, 2023

View reviewed changes

yeya24 merged commit 4bb560e into thanos-io:main Feb 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Receive: Allow setting hashing algorithm per tenant in hashrings config #5653

Receive: Allow setting hashing algorithm per tenant in hashrings config #5653

haanhvu commented Aug 28, 2022 •

edited

Loading

haanhvu commented Aug 29, 2022

fpetkovski left a comment

yeya24 Sep 3, 2022

haanhvu Sep 11, 2022 •

edited

Loading

matej-g Sep 12, 2022

yeya24 Jan 27, 2023

matej-g left a comment

matej-g Sep 12, 2022 •

edited

Loading

haanhvu Sep 13, 2022 •

edited

Loading

matej-g Sep 13, 2022 •

edited

Loading

haanhvu commented Sep 13, 2022 •

edited

Loading

matej-g commented Sep 13, 2022

fpetkovski commented Sep 13, 2022 •

edited

Loading

haanhvu commented Sep 25, 2022

sonatype-lift bot Sep 25, 2022

douglascamata Sep 30, 2022

haanhvu Sep 30, 2022

douglascamata Sep 30, 2022

haanhvu Oct 4, 2022

douglascamata Oct 4, 2022

matej-g commented Oct 27, 2022

haanhvu commented Oct 27, 2022

matej-g commented Oct 27, 2022

haanhvu commented Jan 3, 2023 •

edited

Loading

douglascamata commented Jan 9, 2023

haanhvu commented Feb 5, 2023

douglascamata left a comment

yeya24 left a comment

haanhvu commented Feb 9, 2023

douglascamata commented Feb 9, 2023

Command	Usage
`@sonatype-lift ignore`	Leave out the above finding from this PR
`@sonatype-lift ignoreall`	Leave out all the existing findings from this PR
`@sonatype-lift exclude <file\|issue\|path\|tool>`	Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file

Receive: Allow setting hashing algorithm per tenant in hashrings config #5653

Receive: Allow setting hashing algorithm per tenant in hashrings config #5653

Conversation

haanhvu commented Aug 28, 2022 • edited Loading

Changes

Verification

haanhvu commented Aug 29, 2022

fpetkovski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haanhvu Sep 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matej-g left a comment

Choose a reason for hiding this comment

matej-g Sep 12, 2022 • edited Loading

Choose a reason for hiding this comment

haanhvu Sep 13, 2022 • edited Loading

Choose a reason for hiding this comment

matej-g Sep 13, 2022 • edited Loading

Choose a reason for hiding this comment

haanhvu commented Sep 13, 2022 • edited Loading

matej-g commented Sep 13, 2022

fpetkovski commented Sep 13, 2022 • edited Loading

haanhvu commented Sep 25, 2022

sonatype-lift bot Sep 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matej-g commented Oct 27, 2022

haanhvu commented Oct 27, 2022

matej-g commented Oct 27, 2022

haanhvu commented Jan 3, 2023 • edited Loading

douglascamata commented Jan 9, 2023

haanhvu commented Feb 5, 2023

douglascamata left a comment

Choose a reason for hiding this comment

yeya24 left a comment

Choose a reason for hiding this comment

haanhvu commented Feb 9, 2023

douglascamata commented Feb 9, 2023

haanhvu commented Aug 28, 2022 •

edited

Loading

haanhvu Sep 11, 2022 •

edited

Loading

matej-g Sep 12, 2022 •

edited

Loading

haanhvu Sep 13, 2022 •

edited

Loading

matej-g Sep 13, 2022 •

edited

Loading

haanhvu commented Sep 13, 2022 •

edited

Loading

fpetkovski commented Sep 13, 2022 •

edited

Loading

haanhvu commented Jan 3, 2023 •

edited

Loading