Upgrade joni from 2.1.6 to 2.1.29 #47374
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change `Grok` class to use searchInterruptible(...) instead of search(...) otherwise we can't interrupt long running matching via the thread watch dog. Joni now also provided another way to interrupt long running matches. By invoking the interrupt() method on the matcher. We need then to refactor the watch thread dog to keep track of Matchers instead of Threads, but it is a better of doing this, since interrupting would be more direct (not every 30k iterations) and efficient (checking a volatile field). This work needs to be done in a follow up.
martijnvg
added
>non-issue
:Data Management/Ingest Node
|
Pinging @elastic/es-core-features |
martijnvg
changed the title
|
@elasticmachine run elasticsearch-ci/oss-distro-docs |
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this pull request
Oct 4, 2019
Changed the Grok class to use searchInterruptible(...) instead of search(...) otherwise we can't interrupt long running matching via the thread watch dog. Joni now also provides another way to interrupt long running matches. By invoking the interrupt() method on the Matcher. We need then to refactor the watch thread dog to keep track of Matchers instead of Threads, but it is a better way of doing this, since interrupting would be more direct (not every 30k iterations) and efficient (checking a volatile field). This work needs to be done in a follow up.
martijnvg
added a commit
that referenced
this pull request
Oct 4, 2019
Backport of #47374 Changed the Grok class to use searchInterruptible(...) instead of search(...) otherwise we can't interrupt long running matching via the thread watch dog. Joni now also provides another way to interrupt long running matches. By invoking the interrupt() method on the Matcher. We need then to refactor the watch thread dog to keep track of Matchers instead of Threads, but it is a better way of doing this, since interrupting would be more direct (not every 30k iterations) and efficient (checking a volatile field). This work needs to be done in a follow up.
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this pull request
Oct 10, 2019
This prevents the following warning from being printed to console:
`regular expression has redundant nested repeat operator + /%\{(?<name>(?<pattern>[A-z0-9]+)(?::(?<subname>[[:alnum:]@\[\]_:.-]+))?)(?:=(?<definition>(?:(?:[^{}]+|\.+)+)+))?\}/`
The current grok expression is not failing, but just this warning is being printed.
The warning started being printed after upgrading joni (elastic#47374).
Closes elastic#47861
martijnvg
added a commit
that referenced
this pull request
Oct 14, 2019
This prevents the following warning from being printed to console:
`regular expression has redundant nested repeat operator + /%\{(?<name>(?<pattern>[A-z0-9]+)(?::(?<subname>[[:alnum:]@\[\]_:.-]+))?)(?:=(?<definition>(?:(?:[^{}]+|\.+)+)+))?\}/`
The current grok expression is not failing, but just this warning is being printed.
The warning started being printed after upgrading joni (#47374).
Closes #47861
martijnvg
added a commit
that referenced
this pull request
Oct 14, 2019
This prevents the following warning from being printed to console:
`regular expression has redundant nested repeat operator + /%\{(?<name>(?<pattern>[A-z0-9]+)(?::(?<subname>[[:alnum:]@\[\]_:.-]+))?)(?:=(?<definition>(?:(?:[^{}]+|\.+)+)+))?\}/`
The current grok expression is not failing, but just this warning is being printed.
The warning started being printed after upgrading joni (#47374).
Closes #47861
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this pull request
Oct 22, 2019
There is a watchdog in order to avoid long running (and expensive) grok expressions. Currently the watchdog is thread based, threads that run grok expressions are registered and after completion unregister. If these threads stay registered for too long then the watch dog interrupts these threads. Joni (the library that powers grok expressions) has a mechanism that checks whether the current thread is interrupted and if so abort the pattern matching. Newer versions have an additional method to abort long running pattern matching inside joni. Instead of checking the thread's interrupted flag, joni now also checks a volatile field that can be set via a `Matcher` instance. This is more efficient method for aborting long running matches. (joni checks each 30k iterations whether interrupted flag is set vs. just checking a volatile field) Recently we upgraded to a recent joni version (elastic#47374), and this PR is a followup of that PR. This change should also fix elastic#43673, since it appears when unit tests are ran the a test runner thread's interrupted flag may already have been set, due to some thread reuse.
martijnvg
added a commit
that referenced
this pull request
Oct 24, 2019
…48346) There is a watchdog in order to avoid long running (and expensive) grok expressions. Currently the watchdog is thread based, threads that run grok expressions are registered and after completion unregister. If these threads stay registered for too long then the watch dog interrupts these threads. Joni (the library that powers grok expressions) has a mechanism that checks whether the current thread is interrupted and if so abort the pattern matching. Newer versions have an additional method to abort long running pattern matching inside joni. Instead of checking the thread's interrupted flag, joni now also checks a volatile field that can be set via a `Matcher` instance. This is more efficient method for aborting long running matches. (joni checks each 30k iterations whether interrupted flag is set vs. just checking a volatile field) Recently we upgraded to a recent joni version (#47374), and this PR is a followup of that PR. This change should also fix #43673, since it appears when unit tests are ran the a test runner thread's interrupted flag may already have been set, due to some thread reuse.
martijnvg
added a commit
that referenced
this pull request
Oct 24, 2019
…48346) There is a watchdog in order to avoid long running (and expensive) grok expressions. Currently the watchdog is thread based, threads that run grok expressions are registered and after completion unregister. If these threads stay registered for too long then the watch dog interrupts these threads. Joni (the library that powers grok expressions) has a mechanism that checks whether the current thread is interrupted and if so abort the pattern matching. Newer versions have an additional method to abort long running pattern matching inside joni. Instead of checking the thread's interrupted flag, joni now also checks a volatile field that can be set via a `Matcher` instance. This is more efficient method for aborting long running matches. (joni checks each 30k iterations whether interrupted flag is set vs. just checking a volatile field) Recently we upgraded to a recent joni version (#47374), and this PR is a followup of that PR. This change should also fix #43673, since it appears when unit tests are ran the a test runner thread's interrupted flag may already have been set, due to some thread reuse.
This was referenced Nov 19, 2019
This was referenced Feb 3, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changed the
Grokclass to usesearchInterruptible(...)instead ofsearch(...)otherwise we can't interrupt long running matching via the thread watch
dog.
Joni now also provides another way to interrupt long running matches.
By invoking the
interrupt()method on theMatcher. We need then to refactorthe watch thread dog to keep track of
Matchers instead ofThreads, butit is a better way of doing this, since interrupting would be more direct
(not every 30k iterations) and efficient (checking a volatile field).
This work needs to be done in a follow up.