Use of IOContext.DEFAULT leading to exhaustion of maps

[gbbafna]

Description

Hi,

OpenSearch had to change IOContext from READONCE to DEFAULT , for its remote store feature. This is because in remote store, the thread which closes the IndexInput is different from the one which opens it .

Reference : opensearch-project/OpenSearch#17502

Post switching to DEFAULT, we are seeing maps exhaustion in our production workloads. So far we have not been able to reproduce the issue.

One related issue/repro is opensearch-project/k-NN#2665 (comment) which points to one open index input preventing the whole arena from getting freed up , and ultimately exhaustion of maps.

[2025-05-24T07:00:02,721][WARN ][o.o.i.s.RemoteStoreRefreshListener] [24b9ade6de8c7c3be1382651135d04a1] [vpc-flowlogs-2025.05.24-000673][0] Exception while uploading new segments to the remote segment store
java.io.IOException: Map failed: MemorySegmentIndexInput(path="/hdd1/mnt/env/root/ES-PATH/var/es/data/nodes/0/indices/index-uuid/0/index/_6yh.cfs") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 2714694334 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
    at java.base/sun.nio.ch.FileChannelImpl.mapInternal(FileChannelImpl.java:1319)
    at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1218)
    at org.apache.lucene.store.MemorySegmentIndexInputProvider.map(MemorySegmentIndexInputProvider.java:134)
    at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:76)
    at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:33)
    at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:394)
    at org.opensearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:181)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
    at org.opensearch.index.store.Store$MetadataSnapshot.checksumFromLuceneFile(Store.java:1213)
    at org.opensearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:1183)
    at org.opensearch.index.store.Store.getSegmentMetadataMap(Store.java:392)
    at org.opensearch.index.shard.IndexShard.computeReplicationCheckpoint(IndexShard.java:1887)
    at org.opensearch.index.shard.RemoteStoreRefreshListener.syncSegments(RemoteStoreRefreshListener.java:251)
    at org.opensearch.index.shard.RemoteStoreRefreshListener.performAfterRefreshWithPermit(RemoteStoreRefreshListener.java:159)
    at org.opensearch.index.shard.ReleasableRetryableRefreshListener.runAfterRefreshWithPermit(ReleasableRetryableRefreshListener.java:167)
    at org.opensearch.index.shard.ReleasableRetryableRefreshListener.lambda$scheduleRetry$2(ReleasableRetryableRefreshListener.java:127)
    at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:964)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583)

With IOContext.Default , we are seeing exhaustion of maps . System limits are as below :

❯ ulimit -m
unlimited

❯ulimit -v 
unlimited

❯ sysctl vm.max_map_count
vm.max_map_count = 1048576

To solve this we tried with -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1 . But surprisingly, we were seeing the same issue after some time as well.

We had to disable Memory Segment Index Input to get over this org.apache.lucene.store.MMapDirectory.enableMemorySegments=false . But this fallback option is longer available in latest lucene version .

Does anyone has any pointers to how to go about this issue ? One option we are exploring is to change the IOContext to READONCE, by closing it in the same thread . But even the read is performed in different thread. So we are not sure if READONCE will work fine for us.

Is there any option we can make MemorySegmentIndexInput work for us ?

Version and environment details

OpenSearch Version - 2.19
Lucene Version - 9.12

[onestardao]

Thanks for the clear report — tracking this as #15054.

From your description/logs, the exhaustion appears only after switching the open calls to IOContext.DEFAULT for remote store. That strongly suggests a lifecycle / caching leak tied to thread-local or map-based input caches:

In several Lucene paths, DEFAULT enables buffered/random access and may hit helpers that cache IndexInput/buffers per thread (e.g., via CloseableThreadLocal or a Map keyed by thread/file).

Because your open and close happen on different threads in remote store, those per-thread entries may never get removed, leading to an unbounded map growth and the “exhaustion of maps” you’re seeing.

Quick triage you can try:

Verify same-thread close: ensure every IndexInput opened under DEFAULT is closed on the same thread (wrap in try-with-resources and log Thread.currentThread() on open/close).

Heap histo to find the owner: capture a short heap dump when growth starts; look for big ConcurrentHashMap / CloseableThreadLocal instances under *IndexInputProvider* / store layer to confirm what’s retaining entries.

Use READONCE where sequential is sufficient: for bulk/streamed reads (e.g., copy/restore paths) prefer READONCE, reserving DEFAULT only for random access code paths.

Temporary guard: if you do need DEFAULT, add an explicit cleanup hook in the remote store layer to remove per-thread cache entries on close (same thread), or avoid thread-keyed caches on that path.

Repro test: a small test that opens via thread A and closes via thread B under load should reproduce the leak if that’s the cause.

If you can share a minimal repro (or heap histo dominant types / the map owner class), it’ll help pinpoint which cache needs a fix in Lucene vs. in the remote store integration.

[gbbafna]

Looks like @onestardao is a bot .

[onestardao]

No, it's human :) anything I can help?

[bharath-techie]

Tagging @uschindler

[uschindler]

Hi,
I don't exactly know what your remote directory is doing. I assume it is used for some synchronization of segment files over the wire so it uses mainly copyFrom.

But nevertheless: When you open files with READONCE, which is correct for synchronizing index segments accross the network, I'd strong request to really use READONCE. If you read/copy the stuff multi-threaded access is totally wrong (unless you have clones of indexinputs that do random access or similar in parallel). I'd strongly recommend to refactor your own code to use READONCE in a way like it is designed for: "like an InputStream and only from one thread". So I'd recommend to have the code possibly multi-threaded but open, read and copy and close the file in one thread. In addition, for just copying files over the wire, I'd recommend to just use NIOFSDirectory for the remote directory. Why do you want to read files one time and add extra pging costs for data that's only copied from somewhere else? Have you though of using Lucene's Replicator module?

The problems with DEFAULT vs READONCE with number of mappings come from the fact that you have for sure some leak on managing open files, as @onestardao said.

The fact that -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1 does not help is also a signal for this. It looks like with a confined arena you would also see the same issue, but due to the problem you an't use it (fails with IllegalStateException). To me this really looks like a leak in closing IndexInputs.

[msfroh]

If you read/copy the stuff multi-threaded access is totally wrong (unless you have clones of indexinputs that do random access or similar in parallel).

For the record, that's usually what you want to do when uploading a large file to an object store with parallel multipart upload. You split the file into chunks and each thread uploads a chunk at a time. (The object store accepts chunks in whatever order and stitches them together server-side.)

I could be mistaken, but I think the best way to do that is for each part of the multipart upload to open an IndexInput with READONCE, seek to the part's start offset, upload to the part's end offset, then close the IndexInput.

Of course, it may be easier to ignore the Lucene APIs altogether and just upload files, since the major object store clients all seem to make it pretty easy to upload a file with parallel multipart uploads.

[navneet1v]

If you read/copy the stuff multi-threaded access is totally wrong (unless you have clones of indexinputs that do random access or similar in parallel).

+1 on this.
On JDK docs it says that shared arena has no owner thread if I am reading it correctly.

One possible solution I can see here is:

• Lucene groups the segment files in a shared arena. What if for the remote store uploads, @gbbafna you can try using a different grouping criteria(it shouldn't be hard to build and test, since there is a setGroupingFunction api for MMapDirectory). This will ensure that arena created for uploads is separate from the readers. Now once all the IndexInputs are closed for uploads the arena would get free. This should also be easy to test.

Reason for suggesting above solution is: As per my reading of code I see that Lucene don't close the arena for a segment till all the references of IndexInputs are closed. So what this mean is during the upload you are creating IndexInputs(which I would say is a bad practice, we should always do clone of IndexInput) and then these IndexInputs do get closed, but same segments are getting searched are still open, hence arena will never be closed which means the VM space will still be occupied.

The fact that -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1 does not help

@uschindler I tested this code using a small toy program here but I don't think it helps. The only way I was able to reduce the mmaps if I set grouping to NO grouping. All the opening and closing of files was happening on the main thread only.

[gbbafna]

Thanks @uschindler , @navneet1v , @onestardao , @msfroh

@uschindler , what is the reason behind using READONCE ? Is there some issue with DEFAULT ? We have take heap dumps and verified that IndexInput does get closed . If there is a leak , there would be a problem even with READONCE as well , right ?

If you read/copy the stuff multi-threaded access is totally wrong (unless you have clones of indexinputs that do random access or similar in parallel). I'd strongly recommend to refactor your own code to use READONCE in a way like it is designed for: "like an InputStream and only from one thread". So I'd recommend to have the code possibly multi-threaded but open, read and copy and close the file in one thread

@msfroh is right about it . For multipart upload, we need all threads to read the file concurrently . Can you please share the reasoning of why multi-threaded access is totally wrong ? Is there a limitation which is undocumented ?

To me this really looks like a leak in closing IndexInputs.

We have take heap dumps and verified that IndexInput does get closed .

[dweiss]

@onestardao please do not hijack the discussion thread with irrelevant links.

[uschindler]

Thanks @uschindler , @navneet1v , @onestardao , @msfroh

@uschindler , what is the reason behind using READONCE ? Is there some issue with DEFAULT ? We have take heap dumps and verified that IndexInput does get closed . If there is a leak , there would be a problem even with READONCE as well , right ?

If you read/copy the stuff multi-threaded access is totally wrong (unless you have clones of indexinputs that do random access or similar in parallel). I'd strongly recommend to refactor your own code to use READONCE in a way like it is designed for: "like an InputStream and only from one thread". So I'd recommend to have the code possibly multi-threaded but open, read and copy and close the file in one thread

@msfroh is right about it . For multipart upload, we need all threads to read the file concurrently . Can you please share the reasoning of why multi-threaded access is totally wrong ? Is there a limitation which is undocumented ?

It is wrong if you do this on the same IndexInput without really sequential access. If you want to slice your IndexInput and use multiple threads then it is no longer (strictly speaking) "read once". Because the kernel can't do read-ahead correctly. IndexInput is also not the correct classes to do this. If you want to do that really effective use asynchronous file channels and so on. This is something that Lucene's Directory abstractions are not made for.

For copying files over the network, MMapDirectory is a bad choice, sorry. And read-once only works correct if you have a single stream and copy it.

To me this really looks like a leak in closing IndexInputs.

We have take heap dumps and verified that IndexInput does get closed .

There is a lot of information missing to us. We can't figure out what you are doing without an reproducer. Do you have index files open on the same directory at the same time?

If you have heap dumps: Are the shared arenas still alive, is the refcounter of the grouping arena 0 or not 0? We need more input. The current code works well for Lucene and passs all tests, so theres no refcounting bug on how the code is used in Lucene itsself.

Basically I have more the optinoon, that your code fails because you really have to many shared arenas because too many indexes and at same time replication. If you have hundreds of indexes on a single machine you get close to the map limit. If you then enable replication it gets more crazy. With the READONCE its a bit different, as the shared arenas are not used at all and the additional overhead of closing shared arenas goes away (this slows down multithreaded usage). This can have an impact on how "fast" you run into the issue. So I am sure you will run in the same problems with READONCE, but you can't test this as it does not work.

[uschindler]

@uschindler , what is the reason behind using READONCE ? Is there some issue with DEFAULT ? We have take heap dumps and verified that IndexInput does get closed . If there is a leak , there would be a problem even with READONCE as well , right ?

The reason behind read once is that the Linux kernel optimzes for read-ahead. So this is ideal for use cases where you read the file once (but with the limitation from a single thread). Lucene uses this for files that are only read exactly once (mostly small files or those loaded onto heap). This speeds up reading and at the same time minimizes expensive shared arenas.

You can use DEFAULT IOContext, but you have to live with limitations of MMapDirectory. By default it uses grouping of arena for files from same segment.

If you are only using the directory for replication only (no open indexes on it) consider one of the following:

• disable grouping in MMapDirecory (used for remote copy) using https://lucene.apache.org/core/9_12_1/core/org/apache/lucene/store/MMapDirectory.html#setGroupingFunction(java.util.function.Function): mmapDir.setGroupingFunction(MMapDirectory.NO_GROUPING) - keep in mind that this will cause a JVM safepoint with deoptimization of all top stack frames for all threads on each IndexInput#close(). This will cause serious havoc for heavy multithreaded workloads!
• Rewrite your code to use READONCE and a single thread (which I think is still better to do, sorry). If you need multipe threads your network is the limitation and you should improve that.
• use NIOFSDirectory for plain copy operations
• reimplement your replication to use asynchronous file channels

[uschindler]

If you really want to figure out what the issue with shared arenas is, we need a reproducer and a proof that the sahred arenas are not closed correctly.