FrozenIndexInputTests testRandomReads test failure OOME

[benwtrent]

Build scan:
https://gradle-enterprise.elastic.co/s/rj6bcysmi47pc
Repro line:

Reproduces locally?:
no
Applicable branches:
master
Failure history:
This seems like the first failure
Failure excerpt:

java.lang.OutOfMemoryError: Java heap space |  
-- | --
  | Dumping heap to /mnt/secret/elastic+elasticsearch+master+EAR/x-pack/plugin/searchable-snapshots/build/heapdump/java_pid324394.hprof ... |  
  | Heap dump file created [359374873 bytes in 0.621 secs] |  
  |   |  
  |   |  
  | org.elasticsearch.index.store.cache.FrozenIndexInputTests > testRandomReads FAILED |  
  | com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=35, name=TEST-FrozenIndexInputTests#randomReadAndSlice[T#1], state=RUNNABLE, group=TGRP-FrozenIndexInputTests] |  
  | at __randomizedtesting.SeedInfo.seed([32975B678D00B803:795F99467AC1C29]:0) |  
  |   |  
  | Caused by: |  
  | java.lang.OutOfMemoryError: Java heap space |  
  | at __randomizedtesting.SeedInfo.seed([32975B678D00B803]:0) |  
  | at java.base/jdk.internal.misc.Unsafe.allocateUninitializedArray0(Unsafe.java:1276) |  
  | at java.base/jdk.internal.misc.Unsafe.allocateUninitializedArray(Unsafe.java:1269) |  
  | at java.base/java.lang.invoke.StringConcatFactory$MethodHandleInlineCopyStrategy.newArray(StringConcatFactory.java:1633) |  
  | at java.base/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder) |  
  | at java.base/java.lang.invoke.LambdaForm$MH/0x00000001004dec40.invoke(LambdaForm$MH) |  
  | at java.base/java.lang.invoke.LambdaForm$MH/0x00000001004df040.invoke(LambdaForm$MH) |  
  | at java.base/java.lang.invoke.Invokers$Holder.linkToTargetMethod(Invokers$Holder) |  
  | at org.elasticsearch.index.store.BaseSearchableSnapshotIndexInput.getFullSliceDescription(BaseSearchableSnapshotIndexInput.java:232) |  
  | at org.elasticsearch.index.store.cache.FrozenIndexInput.slice(FrozenIndexInput.java:623) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:75) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:92) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:76) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:92) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:76) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:76) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:76) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.lambda$randomReadAndSlice$0(ESIndexInputTestCase.java:164) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase$$Lambda$591/0x00000001004eb440.get(Unknown Source) |  
  | at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:408) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:164) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:76) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:76) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:76) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:76) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:76) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.lambda$randomReadAndSlice$0(ESIndexInputTestCase.java:164) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase$$Lambda$591/0x00000001004eb440.get(Unknown Source) |  
  | at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:408) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:164) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:76) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.lambda$randomReadAndSlice$0(ESIndexInputTestCase.java:164) |  
  | at org.elasticsearch.common.lucene.store.ESIndexInputTestCase$$Lambda$591/0x00000001004eb440.get(Unknown Source)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[ywelsch]

The issue here can be reliably reproduced by using the test's seed as well as lowering the amount of heap space available to the test (-Dtests.seed=32975B678D00B803 -Xmx256m). With the given seed, this test uses > 256MB of heap space and throws an OOME:

java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([32975B678D00B803]:0)
	at java.base/jdk.internal.misc.Unsafe.allocateUninitializedArray(Unsafe.java:1402)
	at java.base/java.lang.StringConcatHelper.newArray(StringConcatHelper.java:470)
	at java.base/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
	at java.base/java.lang.invoke.LambdaForm$MH/0x0000000800c8fc40.invoke(LambdaForm$MH)
	at java.base/java.lang.invoke.Invokers$Holder.linkToTargetMethod(Invokers$Holder)
	at org.elasticsearch.index.store.BaseSearchableSnapshotIndexInput.getFullSliceDescription(BaseSearchableSnapshotIndexInput.java:232)
	at org.elasticsearch.index.store.cache.FrozenIndexInput.slice(FrozenIndexInput.java:623)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:76)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:94)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:77)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:77)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:77)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.lambda$randomReadAndSlice$0(ESIndexInputTestCase.java:167)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase$$Lambda$597/0x0000000800e668a8.get(Unknown Source)
	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:408)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:167)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:77)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:77)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:77)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:77)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:77)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.lambda$randomReadAndSlice$0(ESIndexInputTestCase.java:167)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase$$Lambda$597/0x0000000800e668a8.get(Unknown Source)
	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:408)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:167)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:77)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.lambda$randomReadAndSlice$0(ESIndexInputTestCase.java:167)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase$$Lambda$597/0x0000000800e668a8.get(Unknown Source)
	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:408)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:167)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:77)
	at org.elasticsearch.common.lucene.store.ESIndexInputTestCase.randomReadAndSlice(ESIndexInputTestCase.java:77)