[CI] ClusterRerouteIT.testDelayWithALargeAmountOfShards timed out waiting for green state #52000
Assignees
Labels
:Distributed Coordination/Cluster Coordination
Cluster formation and cluster state publication, including cluster membership and fault detection.
>test-failure
Triaged test failures from CI
Comments
tlrx
added
>test-failure
|
Pinging @elastic/es-distributed (:Distributed/Cluster Coordination) |
|
Failed again on 7.6 - https://gradle-enterprise.elastic.co/s/wmy4bjcpxqnte |
|
Failed again on master: https://gradle-enterprise.elastic.co/s/getynn6hnzfm2 Looked a bit into it, looks like retention lease sync causes the shards to become stale one by one, not sure if this is part of why it is timing out, but it is at least a significant difference since this test was written. Did not reproduce locally. |
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Feb 18, 2020
It looks like elastic#52000 is caused by a slowdown in cluster state application (maybe due to elastic#50907) but I would like to understand the details to ensure that there's nothing else going on here too before simply increasing the timeout. This commit enables some relevant `DEBUG` loggers and also captures stack traces from all threads rather than just the three hottest ones.
DaveCTurner
added a commit
that referenced
this issue
Feb 18, 2020
It looks like #52000 is caused by a slowdown in cluster state application (maybe due to #50907) but I would like to understand the details to ensure that there's nothing else going on here too before simply increasing the timeout. This commit enables some relevant `DEBUG` loggers and also captures stack traces from all threads rather than just the three hottest ones.
DaveCTurner
added a commit
that referenced
this issue
Feb 18, 2020
It looks like #52000 is caused by a slowdown in cluster state application (maybe due to #50907) but I would like to understand the details to ensure that there's nothing else going on here too before simply increasing the timeout. This commit enables some relevant `DEBUG` loggers and also captures stack traces from all threads rather than just the three hottest ones.
|
Hmm. All 7 of the failures of this test over the last 30 days were using |
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Feb 18, 2020
Issue elastic#52000 looks like a case of cluster state updates being slower than expected, but it seems that these slowdowns are relatively rare: most invocations of `testDelayWithALargeAmountOfShards` take well under a minute in CI, but there are occasional failures that take 6+ minutes instead. When it fails like this, cluster state persistence seems generally slow: most are slower than expected, with some small updates even taking over 2 seconds to complete. The failures all have in common that they use `WindowsFS` to emulate Windows' behaviour of refusing to delete files that are still open, by tracking all files (really, inodes) and validating that deleted files are really closed first. There is a suggestion that this is a little slow in the Lucene test framework [1]. To see if we can attribute the slowdown to that common factor, this commit suppresses the use of `WindowsFS` for this test suite. [1] https://github.com/apache/lucene-solr/blob/4a513fa99f638cb65e0cae59bfdf7af410c0327a/lucene/test-framework/src/java/org/apache/lucene/util/TestRuleTemporaryFilesCleanup.java#L166
DaveCTurner
added a commit
that referenced
this issue
Feb 19, 2020
Issue #52000 looks like a case of cluster state updates being slower than expected, but it seems that these slowdowns are relatively rare: most invocations of `testDelayWithALargeAmountOfShards` take well under a minute in CI, but there are occasional failures that take 6+ minutes instead. When it fails like this, cluster state persistence seems generally slow: most are slower than expected, with some small updates even taking over 2 seconds to complete. The failures all have in common that they use `WindowsFS` to emulate Windows' behaviour of refusing to delete files that are still open, by tracking all files (really, inodes) and validating that deleted files are really closed first. There is a suggestion that this is a little slow in the Lucene test framework [1]. To see if we can attribute the slowdown to that common factor, this commit suppresses the use of `WindowsFS` for this test suite. [1] https://github.com/apache/lucene-solr/blob/4a513fa99f638cb65e0cae59bfdf7af410c0327a/lucene/test-framework/src/java/org/apache/lucene/util/TestRuleTemporaryFilesCleanup.java#L166
DaveCTurner
added a commit
that referenced
this issue
Feb 19, 2020
Issue #52000 looks like a case of cluster state updates being slower than expected, but it seems that these slowdowns are relatively rare: most invocations of `testDelayWithALargeAmountOfShards` take well under a minute in CI, but there are occasional failures that take 6+ minutes instead. When it fails like this, cluster state persistence seems generally slow: most are slower than expected, with some small updates even taking over 2 seconds to complete. The failures all have in common that they use `WindowsFS` to emulate Windows' behaviour of refusing to delete files that are still open, by tracking all files (really, inodes) and validating that deleted files are really closed first. There is a suggestion that this is a little slow in the Lucene test framework [1]. To see if we can attribute the slowdown to that common factor, this commit suppresses the use of `WindowsFS` for this test suite. [1] https://github.com/apache/lucene-solr/blob/4a513fa99f638cb65e0cae59bfdf7af410c0327a/lucene/test-framework/src/java/org/apache/lucene/util/TestRuleTemporaryFilesCleanup.java#L166
|
Echoing Henning's comment here: if after a couple of weeks we decide that
|
|
We discussed this today and decided that we'll take no further action unless we see another failure, and therefore we can close this; please reopen it if you come across another failure. |
sbourke
pushed a commit
to sbourke/elasticsearch
that referenced
this issue
Feb 19, 2020
It looks like elastic#52000 is caused by a slowdown in cluster state application (maybe due to elastic#50907) but I would like to understand the details to ensure that there's nothing else going on here too before simply increasing the timeout. This commit enables some relevant `DEBUG` loggers and also captures stack traces from all threads rather than just the three hottest ones.
sbourke
pushed a commit
to sbourke/elasticsearch
that referenced
this issue
Feb 19, 2020
Issue elastic#52000 looks like a case of cluster state updates being slower than expected, but it seems that these slowdowns are relatively rare: most invocations of `testDelayWithALargeAmountOfShards` take well under a minute in CI, but there are occasional failures that take 6+ minutes instead. When it fails like this, cluster state persistence seems generally slow: most are slower than expected, with some small updates even taking over 2 seconds to complete. The failures all have in common that they use `WindowsFS` to emulate Windows' behaviour of refusing to delete files that are still open, by tracking all files (really, inodes) and validating that deleted files are really closed first. There is a suggestion that this is a little slow in the Lucene test framework [1]. To see if we can attribute the slowdown to that common factor, this commit suppresses the use of `WindowsFS` for this test suite. [1] https://github.com/apache/lucene-solr/blob/4a513fa99f638cb65e0cae59bfdf7af410c0327a/lucene/test-framework/src/java/org/apache/lucene/util/TestRuleTemporaryFilesCleanup.java#L166
|
Another failure from today: https://gradle-enterprise.elastic.co/s/gws7m6tms6coy |
DaveCTurner
added a commit
that referenced
this issue
Mar 5, 2020
Issue #52000 looks like a case of cluster state updates being slower than expected, but it seems that these slowdowns are relatively rare: most invocations of `testDelayWithALargeAmountOfShards` take well under a minute in CI, but there are occasional failures that take 6+ minutes instead. When it fails like this, cluster state persistence seems generally slow: most are slower than expected, with some small updates even taking over 2 seconds to complete. The failures all have in common that they use `WindowsFS` to emulate Windows' behaviour of refusing to delete files that are still open, by tracking all files (really, inodes) and validating that deleted files are really closed first. There is a suggestion that this is a little slow in the Lucene test framework [1]. To see if we can attribute the slowdown to that common factor, this commit suppresses the use of `WindowsFS` for this test suite. [1] https://github.com/apache/lucene-solr/blob/4a513fa99f638cb65e0cae59bfdf7af410c0327a/lucene/test-framework/src/java/org/apache/lucene/util/TestRuleTemporaryFilesCleanup.java#L166
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The test
ClusterRerouteIT » testDelayWithALargeAmountOfShardsfailed today on CI for the7.xbranch with the error:Build scan: https://gradle-enterprise.elastic.co/s/3l2rtho4qznu6
Looking at the build stats it also fails on 7.6 (https://gradle-enterprise.elastic.co/s/bxlllkb4jubxs) and seems to be related to the new way of persisting metadata state (see #50907), in which case tests might need to be adapted?
The text was updated successfully, but these errors were encountered: