Watcher history may not have the correct mappings after an upgrade to 7.7.0 #56732
Labels
Comments
|
Pinging @elastic/es-core-features (:Core/Features/Watcher) |
elasticmachine
added
the
Team:Data Management
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
May 14, 2020
[WatcherIndexTemplateRegistry](https://github.com/elastic/elasticsearch/blob/7.7/x-pack/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/support/WatcherIndexTemplateRegistry.java#L74) as of elastic#52962 requires all nodes to be on 7.7.0 before it allows the version 11 index template to be installed. While in a mixed cluster, nothing prevents Watcher from running on the new host before the all of the nodes are on 7.7.0. This will result in the .watcher-history-11* index without the proper mappings. Without the proper mapping a single document (for a large watch) can exceed the default 1000 field limit and cause error to show in the logs. This commit ensures the same logic for writing to the index is applied as for installing the template. In a mixed cluster, the `10` index template will continue to be written. Only once all of nodes are on 7.7.0+ will the `11` index template be installed and used. Note - this PR targets 7.x and will be back ported to 7.7.next. This conditional installation of the template is not present in master. closes elastic#56732
jakelandis
changed the title
jakelandis
added a commit
that referenced
this issue
May 15, 2020
#56734) WatcherIndexTemplateRegistry as of #52962 requires all nodes to be on 7.7.0 before it allows the version 11 index template to be installed. While in a mixed cluster, nothing prevents Watcher from running on the new host before the all of the nodes are on 7.7.0. This will result in the .watcher-history-11* index without the proper mappings. Without the proper mapping a single document (for a large watch) can exceed the default 1000 field limit and cause error to show in the logs. This commit ensures the same logic for writing to the index is applied as for installing the template. In a mixed cluster, the `10` index template will continue to be written. Only once all of nodes are on 7.7.0+ will the `11` index template be installed and used. closes #56732
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
May 15, 2020
elastic#56734) WatcherIndexTemplateRegistry as of elastic#52962 requires all nodes to be on 7.7.0 before it allows the version 11 index template to be installed. While in a mixed cluster, nothing prevents Watcher from running on the new host before the all of the nodes are on 7.7.0. This will result in the .watcher-history-11* index without the proper mappings. Without the proper mapping a single document (for a large watch) can exceed the default 1000 field limit and cause error to show in the logs. This commit ensures the same logic for writing to the index is applied as for installing the template. In a mixed cluster, the `10` index template will continue to be written. Only once all of nodes are on 7.7.0+ will the `11` index template be installed and used. closes elastic#56732
jakelandis
added a commit
that referenced
this issue
May 15, 2020
… to use (#56734) (#56848) WatcherIndexTemplateRegistry as of #52962 requires all nodes to be on 7.7.0 before it allows the version 11 index template to be installed. While in a mixed cluster, nothing prevents Watcher from running on the new host before the all of the nodes are on 7.7.0. This will result in the .watcher-history-11* index without the proper mappings. Without the proper mapping a single document (for a large watch) can exceed the default 1000 field limit and cause error to show in the logs. This commit ensures the same logic for writing to the index is applied as for installing the template. In a mixed cluster, the `10` index template will continue to be written. Only once all of nodes are on 7.7.0+ will the `11` index template be installed and used. closes #56732
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
May 15, 2020
elastic#56734) WatcherIndexTemplateRegistry as of elastic#52962 requires all nodes to be on 7.7.0 before it allows the version 11 index template to be installed. While in a mixed cluster, nothing prevents Watcher from running on the new host before the all of the nodes are on 7.7.0. This will result in the .watcher-history-11* index without the proper mappings. Without the proper mapping a single document (for a large watch) can exceed the default 1000 field limit and cause error to show in the logs. This commit ensures the same logic for writing to the index is applied as for installing the template. In a mixed cluster, the `10` index template will continue to be written. Only once all of nodes are on 7.7.0+ will the `11` index template be installed and used. closes elastic#56732
jakelandis
added a commit
that referenced
this issue
May 26, 2020
… to use (#56734) (#56855) WatcherIndexTemplateRegistry as of #52962 requires all nodes to be on 7.7.0 before it allows the version 11 index template to be installed. While in a mixed cluster, nothing prevents Watcher from running on the new host before the all of the nodes are on 7.7.0. This will result in the .watcher-history-11* index without the proper mappings. Without the proper mapping a single document (for a large watch) can exceed the default 1000 field limit and cause error to show in the logs. This commit ensures the same logic for writing to the index is applied as for installing the template. In a mixed cluster, the `10` index template will continue to be written. Only once all of nodes are on 7.7.0+ will the `11` index template be installed and used. closes #56732
|
I believe this has been resolved due to changes in 8.x with the way we install Watcher templates. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7.7.0 introduces a new version (
11) of the Watcher history index template. This index template may not get installed prior to Watcher writing to the .watcher-history* index. This may happen due to some new behavior that requires the all the nodes to be on 7.7.0 before installing the new version (11) of index template. [1] This likely isn't a concern to most users since the .watcher-history index is esentially a just a log of the watches that have run and is generally only used for troubleshooting and this will likely go unoticed. Further, since Watcher uses a daily index and upgrades usually don't span multiple days the next day's index should have the correct mappings.This will most likely only be noticed for large watches where the single record of it's execution can exceed the total field of 1000.
failure [java.lang.IllegalArgumentException: Limit of total fields [1000] in index [.watcher-history-11-2020.05.13] has been exceeded]Once an index is created you can not retroactivley apply an index template, so the suggested workaround to this to copy the exiting version (
10) and install it as version11, then delete that day's .watcher-history index. Once the upgrade is complete delete the installed11template and Watcher will auto install the real11template. Or, you can just ignore the error in the logs for that day, it is pretty harmless since it is effectily a failure to log what happened. However, the error logs can get quite noisy if lots of Watches are running past the 1000 field limit.[1]
WatcherIndexTemplateRegistry as of #52962.
The text was updated successfully, but these errors were encountered: