Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid double decrement on current query counter #24922

Merged

Conversation

jasontedor
Copy link
Member

@jasontedor jasontedor commented May 27, 2017

This commit fixes a double decrement bug on the current query counter. The double decrement arises in a situation when the fetch phase is inlined for a query that is only touching one shard. After the query phase succeeds we decrement the current query counter. If the fetch phase ultimately fails, an exception is thrown and we decrement the current query counter again in the catch block. We also add assertions that all current stats counters remain non-negative at all times.

Relates #22996, closes #24872

This commit fixes a double decrement bug on the current query
counter. The double decrement arises in a situation when the fetch phase
is inlined for a query that is only touching one shard. After the query
phase succeeds we decrement the current query counter. If the fetch
phase ultimately fails, an exception is thrown and we decrement the
current query counter again in the catch block. We also add assertions
that all current stats counters remain non-negative at all
times.
@rjernst
Copy link
Member

rjernst commented May 27, 2017

LGTM

@jasontedor
Copy link
Member Author

This bug exists since v5.3.0. Writing a test for this specific situation seems challenging yet the existing test suite HighlighterSearchIT already fails on the seed 85A57E5C8816392C with the assertions added but not the fix; with the fix the suite succeeds again. I am going to consider this sufficient unless someone has a clever idea for an easy way to write a test here. If it's agreed that writing a test for this situation is indeed not easy yet we insist on having a test, I would lean towards it being done in a follow-up as we've started to see a few reports of this issue and we should not hold a fix up.

@jasontedor jasontedor merged commit 3448028 into elastic:master May 27, 2017
jasontedor added a commit that referenced this pull request May 27, 2017
This commit fixes a double decrement bug on the current query
counter. The double decrement arises in a situation when the fetch phase
is inlined for a query that is only touching one shard. After the query
phase succeeds we decrement the current query counter. If the fetch
phase ultimately fails, an exception is thrown and we decrement the
current query counter again in the catch block. We also add assertions
that all current stats counters remain non-negative at all
times.

Relates #24922
jasontedor added a commit that referenced this pull request May 27, 2017
This commit fixes a double decrement bug on the current query
counter. The double decrement arises in a situation when the fetch phase
is inlined for a query that is only touching one shard. After the query
phase succeeds we decrement the current query counter. If the fetch
phase ultimately fails, an exception is thrown and we decrement the
current query counter again in the catch block. We also add assertions
that all current stats counters remain non-negative at all
times.

Relates #24922
@jasontedor jasontedor deleted the double-decrement-query-current branch May 27, 2017 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

INTERNAL_SERVER_ERROR when calling _stats
3 participants