ConcurrentModificationException in BookmarkManager

[harrel56]

Hello team,

I have an issue that ConcurrentModificationException gets thrown when executing multiple concurrent queries (reads).

Neo4j Version: 5.25.1 Community
Neo4j Mode: Single instance
Driver version: Java driver 5.26.3
Operating System: Windows 11 and Debian 12

I'm running these queries in a pooled virtual thread executor limited to 128 threads, heavy load.
TBH I don't even use bookmarks explicitly - this is the code which runs the query:

    public boolean exists(Gav gav) {
        Map<String, Object> gavData = objectMapper.convertValue(gav, new TypeReference<>() {});
        gavData.computeIfAbsent("classifier", k -> "");
        return !driver.executableQuery("""
                        MATCH (root:Artifact)
                        WHERE
                            root.groupId = $props.groupId
                            AND root.artifactId = $props.artifactId
                            AND root.version = $props.version
                            AND root.classifier = $props.classifier
                        RETURN root""")
                .withParameters(Map.of("props", gavData))
                .execute()
                .records()
                .isEmpty();
    }

Stacktrace:

java.util.ConcurrentModificationException
	at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1606)
	at java.base/java.util.HashMap$KeyIterator.next(HashMap.java:1629)
	at java.base/java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1079)
	at java.base/java.util.AbstractCollection.addAll(AbstractCollection.java:337)
	at java.base/java.util.HashSet.<init>(HashSet.java:122)
	at org.neo4j.driver.internal.async.NetworkSession.determineBookmarks(NetworkSession.java:593)
	at org.neo4j.driver.internal.async.NetworkSession.lambda$beginTransactionAsync$21(NetworkSession.java:291)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1194)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2223)
	at org.neo4j.driver.internal.bolt.pooledimpl.PooledBoltConnectionProvider.release(PooledBoltConnectionProvider.java:609)
	at org.neo4j.driver.internal.bolt.pooledimpl.PooledBoltConnectionProvider.lambda$release$39(PooledBoltConnectionProvider.java:613)
	at org.neo4j.driver.internal.bolt.pooledimpl.PooledBoltConnection.lambda$close$17(PooledBoltConnection.java:304)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:931)
	at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2401)
	at java.base/java.util.concurrent.CompletableFuture$MinimalStage.whenComplete(CompletableFuture.java:2992)
	at org.neo4j.driver.internal.bolt.pooledimpl.PooledBoltConnection.lambda$close$18(PooledBoltConnection.java:304)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:978)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:955)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2223)
	at org.neo4j.driver.internal.bolt.api.BasicResponseHandler.onComplete(BasicResponseHandler.java:149)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$ResponseHandleImpl.runIgnoringError(BoltConnectionImpl.java:665)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$ResponseHandleImpl.onComplete(BoltConnectionImpl.java:653)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$ResponseHandleImpl.lambda$new$0(BoltConnectionImpl.java:555)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:885)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2223)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$ResponseHandleImpl.handleSummary(BoltConnectionImpl.java:659)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$ResponseHandleImpl.onResetSummary(BoltConnectionImpl.java:618)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$9.onSummary(BoltConnectionImpl.java:326)
	at org.neo4j.driver.internal.bolt.basicimpl.BoltConnectionImpl$9.onSummary(BoltConnectionImpl.java:316)
	at org.neo4j.driver.internal.bolt.basicimpl.messaging.v3.BoltProtocolV3.lambda$reset$8(BoltProtocolV3.java:310)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:885)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2223)
	at org.neo4j.driver.internal.bolt.basicimpl.handlers.ResetResponseHandler.resetCompleted(ResetResponseHandler.java:61)
	at org.neo4j.driver.internal.bolt.basicimpl.handlers.ResetResponseHandler.onSuccess(ResetResponseHandler.java:40)
	at org.neo4j.driver.internal.bolt.basicimpl.async.inbound.InboundMessageDispatcher.handleSuccessMessage(InboundMessageDispatcher.java:82)
	at org.neo4j.driver.internal.bolt.basicimpl.messaging.common.CommonMessageReader.unpackSuccessMessage(CommonMessageReader.java:59)
	at org.neo4j.driver.internal.bolt.basicimpl.messaging.common.CommonMessageReader.read(CommonMessageReader.java:49)
	at org.neo4j.driver.internal.bolt.basicimpl.async.inbound.InboundMessageHandler.channelRead0(InboundMessageHandler.java:78)
	at org.neo4j.driver.internal.bolt.basicimpl.async.inbound.InboundMessageHandler.channelRead0(InboundMessageHandler.java:33)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at org.neo4j.driver.internal.bolt.basicimpl.async.inbound.MessageDecoder.channelRead(MessageDecoder.java:40)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1503)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1366)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1415)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1575)

Side note: Unfortunately, the original cause gets stripped down (I couldn't see the exact location where the exception is being thrown) - the stacktrace I pasted here was obtained via debugging.

While this exception frequency differs between my local runs on Windows 11 vs remote Debian 12 it's probably the matter of computation power.
While analyzing the stacktraces the Neo4jBookmarkManager code indeed looks faulty (please correct if it's not the case): calling Collections.unmodifiableSet(this.bookmarks) doesn't make the returned collection thread-safe - the underlying collection can still be modified which will be reflected on returned unmodifiable view. I believe that any such changes while iterating over unmodifiable collection (copy constructor) will likely result in ConcurrentModificationException.

I would really appreciate any kind of workaround (like opting-out of these bookmarks?).

Thanks in advance!

PS probably not necessary but here is the link to the whole codebase: https://github.com/harrel56/jar-hell/blob/97ed256128b9cca7cfaa8db8565ea59bf5d304c9/server/src/main/java/dev/harrel/jarhell/repo/ArtifactRepository.java#L77

[harrel56]

Well, I did disable the bookmark manager explicitly and the issue is gone - so in terms of a workaround I'm all good. Adjusted code:

    public boolean exists(Gav gav) {
        Map<String, Object> gavData = objectMapper.convertValue(gav, new TypeReference<>() {});
        gavData.computeIfAbsent("classifier", k -> "");
        try (var session = session()) {
            return session.executeRead(tx -> {
                Result res = tx.run(new Query("""
                        MATCH (root:Artifact)
                        WHERE
                            root.groupId = $props.groupId
                            AND root.artifactId = $props.artifactId
                            AND root.version = $props.version
                            AND root.classifier = $props.classifier
                        RETURN root""",
                        parameters("props", gavData))
                );
                return res.hasNext();
            });
        }
    }

    private Session session() {
        return driver.session(SessionConfig.builder().withBookmarkManager(null).build());
    }

But still, I think the original concurrency issue should be addressed

[injectives]

@harrel56 , thanks for reporting this issue. #1604 should fix it.