[CI] ReloadingDatabasesWhilePerformingGeoLookupsIT#test() failure

[martijnvg]

Build scan: https://gradle-enterprise.elastic.co/s/3wd2py7tycbjm

Repro line: ./gradlew ':modules:ingest-geoip:internalClusterTest' --tests "org.elasticsearch.ingest.geoip.ReloadingDatabasesWhilePerformingGeoLookupsIT.test" -Dtests.seed=1327A9992993281D -Dtests.security.manager=true -Dtests.locale=es-GT -Dtests.timezone=Indian/Chagos -Druntime.java=11

Reproduces locally?: Yes (when running with -Dtests.iters=1024, the test seed is irrelevant, it is a timing issue)

Applicable branches: master

Failure history: https://gradle-enterprise.elastic.co/scans/tests?search.relativeStartTime=P7D&search.timeZoneId=Europe/Amsterdam&tests.container=org.elasticsearch.ingest.geoip.ReloadingDatabasesWhilePerformingGeoLookupsIT&tests.sortField=FAILED&tests.test=test&tests.unstableOnly=true

Failure excerpt:

java.lang.AssertionError: |  
-- | --
  | Expected: null |  
  | but: was <java.lang.AssertionError> |  

  |   | at __randomizedtesting.SeedInfo.seed([1327A9992993281D:9B739643876F45E5]:0) |  
-- | -- | -- | --
  |   | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  |   | at org.junit.Assert.assertThat(Assert.java:956) |  
  |   | at org.junit.Assert.assertThat(Assert.java:923) |  
  |   | at org.elasticsearch.ingest.geoip.ReloadingDatabasesWhilePerformingGeoLookupsIT.test(ReloadingDatabasesWhilePerformingGeoLookupsIT.java:142)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[martijnvg]

This is the actual underlying issue:

java.lang.AssertionError: null
	at org.elasticsearch.ingest.geoip.DatabaseRegistry.removeStaleEntries(DatabaseRegistry.java:258) ~[main/:?]
	at org.elasticsearch.ingest.geoip.ReloadingDatabasesWhilePerformingGeoLookupsIT.lambda$test$1(ReloadingDatabasesWhilePerformingGeoLookupsIT.java:102) [internalClusterTest/:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]

When deleting a db file during the test and we remove the entry for from the DatabaseRegistry, the entry for that is missing. Looks like a testing issue.