[BUG] Frequent stats calls causing memory mapped segments to bloat up

[Bukhtawar]

Describe the bug

Noticed in one of the clusters that the VmSize from /proc/<pid>/status is going upto 200 TB possibly exhausting the virtual address space and that the maps show repeated entries(555) for the same compound files(approx 200 MB)

 cat /proc/$(cat <pid>/maps| grep "indices"| awk '{print $6}'| sort -nr | uniq -c | sort -n| head -3

    555 /hdd1/mnt/env/root/data/nodes/0/indices/dFQt4piDT-SHm3_0mzuGxA/4/index/_2u1d.cfs
    555 /hdd1/mnt/env/root/data/nodes/0/indices/sLwKTsOJRpO7ukMmIkmlxA/6/index/_2n86.cfs
    555 /hdd1/mnt/env/root/data/nodes/0/indices/sLwKTsOJRpO7ukMmIkmlxA/6/index/_2n88.cfs

On taking a look looks like the stats calls are periodically trying to mmap the cfs files, using an IOContext.DEFAULT specified in the Lucene's CompoundReader implementation

2025-09-30T08:34:08,286][WARN ][o.o.i.e.Engine           ][9716ea11db09a681400a3a17bfa977be]  [logs-2025.09.25][0]Error when opening compound reader for Directory [store(ByteSizeCachingDirectory(OpensearchDirectory(HybridDirectory@/hdd1/mnt/env/root/data/nodes/0/indices/sJzkGGh_RzCloFt7cjK3Xw/0/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@15a9506)))] and SegmentCommitInfo [_2qs3(10.2.1):c65358:[diagnostics={os=Linux, timestamp=1758844782380, mergeMaxNumSegments=-1, lucene.version=10.2.1, source=merge, os.arch=aarch64, java.runtime.version=21.0.8+9-LTS, mergeFactor=10}]:[attributes={Lucene90StoredFieldsFormat.mode=BEST_SPEED}] :id=ciiiy8pv84l0jeda3ormywnaq]
java.io.IOException: Map failed: MemorySegmentIndexInput(path="/hdd1/mnt/env/root/data/nodes/0/indices/sJzkGGh_RzCloFt7cjK3Xw/0/index/_2qs3.cfs") [this may be caused by lack of enough unfragmented virtual address space or too restrictive virtual memory limits enforced by the operating system, preventing us to map a chunk of 53462188 bytes. Please review 'ulimit -v', 'ulimit -m' (both should return 'unlimited'), and 'sysctl vm.max_map_count'. More information: https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html]
        at java.base/sun.nio.ch.FileChannelImpl.mapInternal(FileChannelImpl.java:1319)
        at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1218)
        at org.apache.lucene.store.MemorySegmentIndexInputProvider.map(MemorySegmentIndexInputProvider.java:123)
        at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:68)
        at org.apache.lucene.store.MemorySegmentIndexInputProvider.openInput(MemorySegmentIndexInputProvider.java:32)
        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:268)
        at org.opensearch.index.store.FsDirectoryFactory$HybridDirectory.openInput(FsDirectoryFactory.java:173)
        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:101)
        at org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.<init>(Lucene90CompoundReader.java:78)
        at org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.getCompoundReader(Lucene90CompoundFormat.java:86)
        at org.opensearch.index.engine.Engine.getSegmentFileSizes(Engine.java:991)
        at org.opensearch.index.engine.Engine.fillSegmentStats(Engine.java:977)
        at org.opensearch.index.engine.Engine.segmentsStats(Engine.java:933)
        at org.opensearch.index.shard.IndexShard.segmentStats(IndexShard.java:1651)
        at org.opensearch.action.admin.indices.stats.CommonStats.<init>(CommonStats.java:230)
        at org.opensearch.indices.IndicesService.indexShardStats(IndicesService.java:886)
        at org.opensearch.indices.IndicesService.statsByShard(IndicesService.java:837)
        at org.opensearch.indices.IndicesService.stats(IndicesService.java:825)
        at org.opensearch.node.NodeService.stats(NodeService.java:253)
        at org.opensearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:105)
        at org.opensearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:200)
        at org.opensearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:332)
        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:118)
        at org.opensearch.performanceanalyzer.transport.RTFPerformanceAnalyzerTransportRequestHandler.messageReceived(RTFPerformanceAnalyzerTransportRequestHandler.java:92)
        at org.opensearch.wlm.WorkloadManagementTransportInterceptor$RequestHandler.messageReceived(WorkloadManagementTransportInterceptor.java:63)
        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:108)
        at org.opensearch.transport.TransportService$7.doRun(TransportService.java:1048)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

It's possible we are leaking an IndexInput but since this code path hasn't been touched in a while, it makes me feel there is some issue with the index input for compound files

Tagging @ashking94 @uschindler @mch2 @shourya035 for thoughts on the same.

Related component

No response

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Memory to stay flat

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[jainankitk]

Could this be related to #19072? Although, that is specific to multi part upload for remote store.

[uschindler]

Which lucene version is this?

Today somebody opened a Lucene issue about many not closed CFS files: apache/lucene#15262

Maybe this is related. Solr and Elasticsearch mostly use no CFS files, so this could be an issue specifically about CFS files.

Can you have a look also on this issue? I think they don't use mmap and instead niofsdir, but effect is same if the inputs are not closed.

[Bukhtawar]

Thanks @uschindler for taking a look, the issue you linked is Lucene 9.x which is likely due to deleted files being referenced. However one major difference is that we are on Lucene 10 and files exist on the disk(active working set), it's just that we see same cfs files mmaped a very high number of times, which could possibly be due to a leak as well, which is where we are trying to isolate. The rest of the code uses IOContext ReadOnce so wasn't sure if IOContext Default has any role to play or if setting the max permit for the shared arena to one would be of any help

[uschindler]

Hi the other issue is about deleted files yes, but still it looks like the file handles are still open, although the index is closed.

Because of this I see a relationship:

• Lucene 9 and Lucene 10 have same CompoundFileDir impl (mostly). The user in Lucene's issue see open file handles on CFS files (because they use NIOFSDir) after closing the index. This comes from a possible leak in CFS dir's IndexInput not closed.
• In your case you see mapped memory not released (on each stat call it re-maps the CFS file). This looks like the CFS file IndexInput is not closed and memory unmapped.

So we should look in Lucene and figure out if the CompoundFileDirectory is correctly closed after usage. A leak there would cause NIOFSDir to open file handles and not close them, while mmap maps memory segments over an over without releasing them.

[bharath-techie]

Setting -Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits=1 seems to help..

I'm able to reproduce by calling segment stats call which opens the compound reader and closes it. So every 10 seconds i make one call to indices stats.

This is on a static index with no new documents/ refreshes/ merges or searches.

Before :

while true; do
    echo "$(date) - $(cat /proc/2968414/maps | grep '\.cfs$' | wc -l) segment files"
    sleep 5
done

Wed Oct  1 09:00:39 UTC 2025 - 710 segment files
Wed Oct  1 09:00:44 UTC 2025 - 710 segment files
Wed Oct  1 09:00:49 UTC 2025 - 724 segment files
Wed Oct  1 09:00:54 UTC 2025 - 724 segment files
Wed Oct  1 09:00:59 UTC 2025 - 738 segment files
Wed Oct  1 09:01:04 UTC 2025 - 738 segment files
Wed Oct  1 09:01:09 UTC 2025 - 752 segment files
Wed Oct  1 09:01:14 UTC 2025 - 752 segment files

We can see every 10 seconds number of maps to same CFS files increase.

After setting max permits to 1:

while true; do
     echo "$(date) - $(cat /proc/2972172/maps | grep '\.cfs$' | wc -l) segment files";
     sleep 5; 
done

Wed Oct  1 09:05:33 UTC 2025 - 80 segment files
Wed Oct  1 09:05:38 UTC 2025 - 80 segment files
Wed Oct  1 09:05:43 UTC 2025 - 80 segment files
Wed Oct  1 09:05:48 UTC 2025 - 80 segment files
Wed Oct  1 09:05:53 UTC 2025 - 80 segment files
Wed Oct  1 09:05:58 UTC 2025 - 80 segment files
Wed Oct  1 09:06:03 UTC 2025 - 80 segment files
Wed Oct  1 09:06:08 UTC 2025 - 80 segment files

There is no similar increase.

[bharath-techie]

https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/engine/Engine.java#L991 - this is the method which calls compoundReader.open during stats call.

https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/engine/Engine.java#L1059 -- close gets called in same thread as long as there are no exceptions.

@uschindler @Bukhtawar since max permits = 1 is helping, any guesses on what could be the issue ?

[Bukhtawar]

@uschindler @Bukhtawar since max permits = 1 is helping, any guesses on what could be the issue ?

This certainly doesn't seem like a leak to me. I generally do think that IOContext default doesn't work without setting the shared arena max permits to 1. The only other option is to switch to IOContext ReadOnce

[uschindler]

The first question is: Why do you open the same file over and over again. The Shared Arenas don't implement any caching there, so every file which is opened again leads to a new shared arena. As files in Lucene never change (they are all write-once, except the segment info master file), if the filename does not change the contents have not changed. The files which are not unmodifiable (only one type) is definitely read with read-once, so it doe snot use a shared arena at all.

So basically: Please refactor your code to not open the same file over an over. If its filename does not change its still the same, so theres no need to read it every 10 seconds.

As another note: Please keep in mind that newest Lucene version reduced the shared permits to 64 (also in Lucene 9.x), as the default of 1024 was way too high. Reducing the shared permits to 1 will make high refresh rates of Lucene indexes with many indexes open in parallel and accessed by many threads very, very slow! So the caching is there for a reason. But misusing the index apis to re-read the same file should be avoided.

So before trying to fix anyting here think about how to implement the whole thing correctly. If you just need some statistics, you can retrieve them once. No need to do that in a loop.

[navneet1v]

@Bukhtawar do we know in the code from where we are opening the same .cfs file more than once?