Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configurable deadlock detector interval for ingester. (resubmit) #1134

Merged
merged 6 commits into from
Nov 12, 2018

Conversation

marqc
Copy link
Contributor

@marqc marqc commented Oct 22, 2018

Resubmit because of github failures today, and previously submitter PR seems broken

Which problem is this PR solving?

Short description of the changes

  • Additional configurable flag for ingester deadlock detector interval
  • Value of 0 disables deadlock_detector

Copy link
Member

@yurishkuro yurishkuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks great, thanks for your contribution

if atomic.LoadUint64(detector.msgConsumed) == 0 {
s.panicFunc(-1)
return // For tests
if s.interval == 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to head of start() & exit immediately instead?

Copy link
Contributor Author

@marqc marqc Oct 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yurishkuro

Cannot really do that, because without initialization of "allPartitionsDeadlockDetector" API gets broken - and NullPointerExceptions will start to appear.

I little change this code not to check for interval inside goroutine.

cmd/ingester/app/flags.go Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Oct 23, 2018

Codecov Report

Merging #1134 into master will not change coverage.
The diff coverage is 100%.

Impacted file tree graph

@@          Coverage Diff           @@
##           master   #1134   +/-   ##
======================================
  Coverage     100%    100%           
======================================
  Files         144     144           
  Lines        6822    6839   +17     
======================================
+ Hits         6822    6839   +17
Impacted Files Coverage Δ
cmd/ingester/app/consumer/deadlock_detector.go 100% <100%> (ø) ⬆️
cmd/ingester/app/consumer/consumer.go 100% <100%> (ø) ⬆️
cmd/ingester/app/flags.go 100% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update be62816...f099a4e. Read the comment docs.

@marqc marqc force-pushed the issue-1125 branch 2 times, most recently from 88191d8 to c4c4a8d Compare October 23, 2018 14:00
Value of 0 disables deadlock_detector. #issue1225

Signed-off-by: Chodor Marek <marek.chodor@grupawp.pl>
Copy link
Member

@yurishkuro yurishkuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for staying on it, @marqc. I wonder if there is a simpler solution here. Essentially, when the check interval is zero we want a noop implementation. We can do it explicitly with a separate type, but it might be a lot of churn as we'd need interfaces as well.

Another option is adding a "disabled" flag that will short-circuit all externally called functions to noop.

An even trickier variant of disabled monitor is to return mail from constructor and short-circuit functions based on nil receiver. This might not work if we expose deadlock monitor in fx module.

In all cases we don't want to create any channels. close() functions can exit immediately.

Signed-off-by: Chodor Marek <marek.chodor@grupawp.pl>
@marqc
Copy link
Contributor Author

marqc commented Oct 26, 2018

@yurishkuro I got rid of closed flag, and introduced disabled flag with short circuits as You suggested. Also changed logic to not create unnecessary channels.

Signed-off-by: Chodor Marek <marek.chodor@grupawp.pl>
cmd/ingester/app/consumer/deadlock_detector.go Outdated Show resolved Hide resolved
cmd/ingester/app/consumer/deadlock_detector.go Outdated Show resolved Hide resolved
cmd/ingester/app/consumer/deadlock_detector.go Outdated Show resolved Hide resolved
cmd/ingester/app/consumer/deadlock_detector.go Outdated Show resolved Hide resolved
cmd/ingester/app/consumer/deadlock_detector.go Outdated Show resolved Hide resolved
cmd/ingester/app/consumer/deadlock_detector.go Outdated Show resolved Hide resolved
cmd/ingester/app/consumer/deadlock_detector.go Outdated Show resolved Hide resolved
cmd/ingester/app/flags_test.go Outdated Show resolved Hide resolved
Chodor Marek added 2 commits November 5, 2018 14:10
Signed-off-by: Chodor Marek <marek.chodor@grupawp.pl>
Signed-off-by: Chodor Marek <marek.chodor@grupawp.pl>
@marqc
Copy link
Contributor Author

marqc commented Nov 5, 2018

@yurishkuro I applied your suggestions

@ghost ghost assigned yurishkuro Nov 12, 2018
@ghost ghost added the review label Nov 12, 2018
@yurishkuro yurishkuro mentioned this pull request Nov 12, 2018
w.incrementMsgCount()
w.incrementAllPartitionMsgCount()
assert.Zero(t, len(w.closePartitionChannel()))
w.close()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not clear what this is testing. Why calling increments should have any effect on the closePartitionChannel?

@pavolloffay pavolloffay merged commit b2bcba5 into jaegertracing:master Nov 12, 2018
@ghost ghost removed the review label Nov 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants