Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reduce duplicate rdy update requests #282

Merged

Conversation

andyxning
Copy link
Member

This PR will reduct duplicate rdy update requests.

@andyxning
Copy link
Member Author

/cc @ploxiln @mreiferson

@andyxning andyxning force-pushed the reduce_duplicate_rdy_updation_requests branch 4 times, most recently from fa27f35 to 0f433a4 Compare January 15, 2020 09:49
@andyxning
Copy link
Member Author

/cc @ploxiln @mreiferson

mock_test.go Outdated
@@ -343,15 +343,19 @@ func TestConsumerRequeueNoBackoff(t *testing.T) {
"RDY 0",
fmt.Sprintf("REQ %s 0", msgIDRequeueNoBackoff),
"RDY 1",
"RDY 1",
"RDY 1", // Duplicate rdy update will be ignored.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like, if you just comment out this item completely, none of the additional changes below are needed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ploxiln No. I just added a comment about the result of duplicate rdy update.

Copy link
Member Author

@andyxning andyxning Feb 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ploxiln Sorry for not understanding your opinion.

if you just comment out this item completely, none of the additional changes below are needed

Yes. But i think we should test the deduplicate rdy update logic. If you think this is unnecessary, i will remove it.

Copy link
Member

@ploxiln ploxiln Feb 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is not for the input, this line is for the expected output. Commenting out this line does not change what the "rdy update logic" goes through in this test. Commenting out this line is equivalent to, but simpler, than the code added just below to skip over this line, in the only place it could affect anything.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ploxiln Thanks for your tips. The code has been updated. PTAL.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ploxiln PTAL.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @jehiah

@andyxning andyxning force-pushed the reduce_duplicate_rdy_updation_requests branch from 0f433a4 to 5254c93 Compare February 21, 2020 02:52
@ploxiln
Copy link
Member

ploxiln commented Feb 27, 2020

Yeah, seems safe, I might just merge it, since you at least will use it plenty. But I would like to get some input from the other maintainers.

By the way, did you consider this alternative:

--- a/consumer.go
+++ b/consumer.go
@@ -975,7 +975,7 @@ func (r *Consumer) updateRDY(c *Conn, count int64) error {
 }
 
 func (r *Consumer) sendRDY(c *Conn, count int64) error {
-       if count == 0 && c.LastRDY() == 0 {
+       if count == c.LastRDY() {
                // no need to send. It's already that RDY count
                return nil
        }

@andyxning
Copy link
Member Author

@ploxiln Thanks for the suggestion. The key point of implementing using the code in the PR is that if the count is not zero, then SetRDY will update lastRdyTimestamp. Since rdy manage is tricky, i think we should not neglect it.

@ploxiln
Copy link
Member

ploxiln commented Mar 1, 2020

Yeah, fair.

Some observations:

  • Conn.RDY() and Conn.LastRDY() are the exact same
  • Conn.lastRdyTimestamp / Conn.LastRdyTime() are only used in Consumer.redistributeRDY() to check Config.LowRdyTimeout

So this should be fine. Maybe later someone will come up with a good refactor of the whole RDY distribution algorithm (e.g. to address #277).

@ploxiln ploxiln merged commit 93aecae into nsqio:master Mar 1, 2020
@andyxning andyxning deleted the reduce_duplicate_rdy_updation_requests branch March 2, 2020 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants