[CI] ApiKeyIntegTests#testExpiredApiKeysBehaviorWhenKeysExpired1WeekBeforeAnd1DayBefore fails

[cbuescher]

On 7.x: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=centos-6/143/console

Cannot reproduce locally:

./gradlew :x-pack:plugin:security:test --tests "org.elasticsearch.xpack.security.authc.ApiKeyIntegTests.testExpiredApiKeysBehaviorWhenKeysExpired1WeekBeforeAnd1DayBefore" \
  -Dtests.seed=84E00A79A9BD633 \
  -Dtests.security.manager=true \
  -Dtests.locale=ar-SD \
  -Dtests.timezone=Asia/Makassar \
  -Dcompiler.java=12 \
  -Druntime.java=8

java.lang.AssertionError: 
Expected: is <3>
     but: was <2>
	at __randomizedtesting.SeedInfo.seed([84E00A79A9BD633:D7D332CC16CF3C5A]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.xpack.security.authc.ApiKeyIntegTests.testExpiredApiKeysBehaviorWhenKeysExpired1WeekBeforeAnd1DayBefore(ApiKeyIntegTests.java:366)

[elasticmachine]

Pinging @elastic/es-security

[cbuescher]

I found another similar looking one from April 24th:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=centos-6/126/console

[davidkyle]

Another: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+multijob-unix-compatibility/os=oraclelinux-6/157/console

Same error as above

java.lang.AssertionError: 
Expected: is <3>
     but: was <2>
	at __randomizedtesting.SeedInfo.seed([2FD576BE197C4C79:F04844D59528A610]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.xpack.security.authc.ApiKeyIntegTests.testExpiredApiKeysBehaviorWhenKeysExpired1WeekBeforeAnd1DayBefore(ApiKeyIntegTests.java:362)

Does not reproduce:

./gradlew :x-pack:plugin:security:unitTest \
  -Dtests.seed=2FD576BE197C4C79 \
  -Dtests.class=org.elasticsearch.xpack.security.authc.ApiKeyIntegTests \
  -Dtests.method="testExpiredApiKeysBehaviorWhenKeysExpired1WeekBeforeAnd1DayBefore" \
  -Dtests.security.manager=true \
  -Dtests.locale=pl-PL \
  -Dtests.timezone=America/North_Dakota/Center \
  -Dcompiler.java=12 \
  -Druntime.java=8

[davidkyle]

Muted:

• master 6fe07e6
• 7.x c0d6791
• 6.7 874bf43