ConcurrentModificationException through async getTotalCount

[AntoineDuComptoirDesPharmacies]

In our code, we load the query count in background to enhance server performance. However, in some case we receive a Sentry from production containing ConcurrentModificationException.

We tried to reproduce this error locally without success.

Note : This happen when the specific code is called on a burst. (Thousands of call in few seconds)
Note 2 : We can see the query associated to method getList in database logs but not the query associated to getTotalCount.

Current version used :
Ebean : 13.22.0-jakarta

Expected behavior

There should not be any concurreny problem when calling getTotalCount on PagedList .

Actual behavior

We are not sure if the bug come from the way we use Ebean or of it is intenal to the library.

Steps to reproduce

PagedList<T> pagedList = query
                .setFirstRow(page * limit)
                .setMaxRows(limit)
                .findPagedList();

        pagedList.loadCount();

        Supplier<Integer> totalVisible = () -> pagedList.getTotalCount();

        return new SearchResult(
                pagedList.getList(),
                totalVisible.get(),
        );

java.util.ConcurrentModificationException: null
    at java.util.LinkedHashMap$LinkedHashIterator.nextNode
    at java.util.LinkedHashMap$LinkedValueIterator.next
    at io.ebeaninternal.server.persist.BatchedPstmtHolder.closeStatements(BatchedPstmtHolder.java:153)
    at io.ebeaninternal.server.persist.BatchedPstmtHolder.clear(BatchedPstmtHolder.java:147)
    at io.ebeaninternal.server.persist.BatchControl.flushOnCommit(BatchControl.java:249)
    at io.ebeaninternal.server.transaction.JdbcTransaction.batchFlush(JdbcTransaction.java:550)
    at io.ebeaninternal.server.transaction.JdbcTransaction.internalBatchFlush(JdbcTransaction.java:665)
    at io.ebeaninternal.server.transaction.JdbcTransaction.flush(JdbcTransaction.java:654)
    at io.ebeaninternal.api.SpiTransactionProxy.flush(SpiTransactionProxy.java:313)
    at io.ebeaninternal.server.query.DefaultOrmQueryEngine.flushJdbcBatchOnQuery(DefaultOrmQueryEngine.java:59)
    at io.ebeaninternal.server.query.DefaultOrmQueryEngine.findCount(DefaultOrmQueryEngine.java:83)
    at io.ebeaninternal.server.core.OrmQueryRequest.findCount(OrmQueryRequest.java:358)
    at io.ebeaninternal.server.core.DefaultServer.findCountWithCopy(DefaultServer.java:1182)
    at io.ebeaninternal.server.query.CallableQueryCount.call(CallableQueryCount.java:30)
    at io.ebeaninternal.server.query.CallableQueryCount.call(CallableQueryCount.java:11)
    at java.util.concurrent.FutureTask.run
    at io.ebeaninternal.server.executor.DefaultBackgroundExecutor.lambda$logExceptions$0(DefaultBackgroundExecutor.java:76)
    at java.util.concurrent.Executors$RunnableAdapter.call
    at java.util.concurrent.FutureTask.run
    at java.util.concurrent.ThreadPoolExecutor.runWorker
    at java.util.concurrent.ThreadPoolExecutor$Worker.run
    at java.lang.Thread.run
java.util.concurrent.ExecutionException: java.util.ConcurrentModificationException
    at java.util.concurrent.FutureTask.report
    at java.util.concurrent.FutureTask.get
    at io.ebeaninternal.server.query.BaseFuture.get(BaseFuture.java:30)
    at io.ebeaninternal.server.query.LimitOffsetPagedList.getTotalCount(LimitOffsetPagedList.java:93)
    at fr.lcdp.ebean.utils.searchResult.executors.BaseQueryExecutor.lambda$doExecute$2(BaseQueryExecutor.java:94)
    at fr.lcdp.ebean.utils.searchResult.executors.BaseQueryExecutor.doExecute(BaseQueryExecutor.java:102)
    at fr.lcdp.ebean.utils.searchResult.executors.DomainQueryExecutor.execute(DomainQueryExecutor.java:23)
    at domain.ads.AdDatabaseBook.getAdsFromDatabase(AdDatabaseBook.java:307)
    at domain.ads.AdDatabaseBook.getAdsFromDatabaseDomain(AdDatabaseBook.java:325)
    at domain.ads.action.validators.BasicAdValidator.checkUniqueAdOnProduct(BasicAdValidator.java:284)
    at domain.ads.action.validators.BasicAdValidator.laboratoryValidation(BasicAdValidator.java:264)
    at domain.ads.action.validators.BasicAdValidator.validate(BasicAdValidator.java:82)
    at domain.ads.action.validators.BasicAdCreateValidator.validate(BasicAdCreateValidator.java:24)
    at domain.ads.action.BasicAdFactory.newCreate(BasicAdFactory.java:136)
    at domain.ads.AdDatabaseRepository.createVersion(AdDatabaseRepository.java:262)
    at service.AdService.createVersion(AdService.java:318)
    at provide.saleOffer.controllers.ManageSaleOfferApiControllerImp.createSaleOfferVersion(ManageSaleOfferApiControllerImp.java:118)
    at provide.saleOffer.controllers.ManageSaleOfferApiController.createSaleOfferVersion(ManageSaleOfferApiController.java:173)
    at saleOffer.Routes$$anonfun$routes$1.$anonfun$applyOrElse$24(Routes.scala:372)
    at play.core.routing.HandlerInvokerFactory$$anon$8.resultCall(HandlerInvoker.scala:160)
    at play.core.routing.HandlerInvokerFactory$JavaActionInvokerFactory$$anon$3$$anon$4$$anon$5.invocation(HandlerInvoker.scala:125)
    at play.core.j.JavaAction$$anon$1.$anonfun$call$1(JavaAction.scala:127)
    at play.api.mvc.BodyParser$.$anonfun$parseBody$4(Action.scala:241)
    at scala.Option.getOrElse(Option.scala:201)
    at play.api.mvc.BodyParser$.parseBody(Action.scala:239)
    at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:128)
    at play.http.DefaultActionCreator$1.call(DefaultActionCreator.java:31)
    at actions.SentryOn5XXAction.call(SentryOn5XXAction.java:36)
    at be.objectify.deadbolt.java.actions.AbstractDeadboltAction.authorizeAndExecute(AbstractDeadboltAction.java:280)
    at be.objectify.deadbolt.java.ConstraintLogic.pass(ConstraintLogic.java:475)
    at be.objectify.deadbolt.java.ConstraintLogic.lambda$restrict$5(ConstraintLogic.java:141)
    at java.util.concurrent.CompletableFuture.uniComposeStage
    at java.util.concurrent.CompletableFuture.thenCompose
    at java.util.concurrent.CompletableFuture.thenCompose
    at be.objectify.deadbolt.java.ConstraintLogic.restrict(ConstraintLogic.java:129)
    at be.objectify.deadbolt.java.actions.RestrictAction.applyRestriction(RestrictAction.java:73)
    at be.objectify.deadbolt.java.actions.AbstractRestrictiveAction.lambda$null$1(AbstractRestrictiveAction.java:59)
    at java.util.Optional.orElseGet
    at be.objectify.deadbolt.java.actions.AbstractRestrictiveAction.lambda$execute$2(AbstractRestrictiveAction.java:59)
    at java.util.concurrent.CompletableFuture.uniComposeStage
    at java.util.concurrent.CompletableFuture.thenCompose
    at java.util.concurrent.CompletableFuture.thenCompose
    at be.objectify.deadbolt.java.actions.AbstractRestrictiveAction.execute(AbstractRestrictiveAction.java:58)
    at be.objectify.deadbolt.java.actions.AbstractDeadboltAction.call(AbstractDeadboltAction.java:121)
    at play.core.j.JavaAction.$anonfun$apply$8(JavaAction.scala:184)
    at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687)
    at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
    at play.core.j.ClassLoaderExecutionContext.$anonfun$execute$1(ClassLoaderExecutionContext.scala:64)
    at datadog.trace.bootstrap.instrumentation.java.concurrent.RunnableWrapper.run(RunnableWrapper.java:24)
    at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:65)
    at play.core.j.ClassLoaderExecutionContext.execute(ClassLoaderExecutionContext.scala:59)
    at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:429)
    at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:338)
    at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:312)
    at scala.concurrent.impl.Promise$DefaultPromise.map(Promise.scala:182)
    at scala.concurrent.Future$.apply(Future.scala:687)
    at play.core.j.JavaAction.apply(JavaAction.scala:185)
    at play.api.mvc.Action.$anonfun$apply$6(Action.scala:83)
    at play.api.mvc.BodyParser$.$anonfun$runParserThenInvokeAction$1(Action.scala:260)
    at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:470)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker
    at java.util.concurrent.ThreadPoolExecutor$Worker.run
    at java.lang.Thread.run
jakarta.persistence.PersistenceException: java.util.concurrent.ExecutionException: java.util.ConcurrentModificationException
    at io.ebeaninternal.server.query.LimitOffsetPagedList.getTotalCount(LimitOffsetPagedList.java:95)
    at fr.lcdp.ebean.utils.searchResult.executors.BaseQueryExecutor.lambda$doExecute$2(BaseQueryExecutor.java:94)
    at fr.lcdp.ebean.utils.searchResult.executors.BaseQueryExecutor.doExecute(BaseQueryExecutor.java:102)
    at fr.lcdp.ebean.utils.searchResult.executors.DomainQueryExecutor.execute(DomainQueryExecutor.java:23)
    at domain.ads.AdDatabaseBook.getAdsFromDatabase(AdDatabaseBook.java:307)
    at domain.ads.AdDatabaseBook.getAdsFromDatabaseDomain(AdDatabaseBook.java:325)
    at domain.ads.action.validators.BasicAdValidator.checkUniqueAdOnProduct(BasicAdValidator.java:284)
    at domain.ads.action.validators.BasicAdValidator.laboratoryValidation(BasicAdValidator.java:264)
    at domain.ads.action.validators.BasicAdValidator.validate(BasicAdValidator.java:82)
    at domain.ads.action.validators.BasicAdCreateValidator.validate(BasicAdCreateValidator.java:24)
    at domain.ads.action.BasicAdFactory.newCreate(BasicAdFactory.java:136)
    at domain.ads.AdDatabaseRepository.createVersion(AdDatabaseRepository.java:262)
    at service.AdService.createVersion(AdService.java:318)
    at provide.saleOffer.controllers.ManageSaleOfferApiControllerImp.createSaleOfferVersion(ManageSaleOfferApiControllerImp.java:118)
    at provide.saleOffer.controllers.ManageSaleOfferApiController.createSaleOfferVersion(ManageSaleOfferApiController.java:173)
    at saleOffer.Routes$$anonfun$routes$1.$anonfun$applyOrElse$24(Routes.scala:372)
    at play.core.routing.HandlerInvokerFactory$$anon$8.resultCall(HandlerInvoker.scala:160)
    at play.core.routing.HandlerInvokerFactory$JavaActionInvokerFactory$$anon$3$$anon$4$$anon$5.invocation(HandlerInvoker.scala:125)
    at play.core.j.JavaAction$$anon$1.$anonfun$call$1(JavaAction.scala:127)
    at play.api.mvc.BodyParser$.$anonfun$parseBody$4(Action.scala:241)
    at scala.Option.getOrElse(Option.scala:201)
    at play.api.mvc.BodyParser$.parseBody(Action.scala:239)
    at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:128)
    at play.http.DefaultActionCreator$1.call(DefaultActionCreator.java:31)
    at actions.SentryOn5XXAction.call(SentryOn5XXAction.java:36)
    at be.objectify.deadbolt.java.actions.AbstractDeadboltAction.authorizeAndExecute(AbstractDeadboltAction.java:280)
    at be.objectify.deadbolt.java.ConstraintLogic.pass(ConstraintLogic.java:475)
    at be.objectify.deadbolt.java.ConstraintLogic.lambda$restrict$5(ConstraintLogic.java:141)
    at java.util.concurrent.CompletableFuture.uniComposeStage
    at java.util.concurrent.CompletableFuture.thenCompose
    at java.util.concurrent.CompletableFuture.thenCompose
    at be.objectify.deadbolt.java.ConstraintLogic.restrict(ConstraintLogic.java:129)
    at be.objectify.deadbolt.java.actions.RestrictAction.applyRestriction(RestrictAction.java:73)
    at be.objectify.deadbolt.java.actions.AbstractRestrictiveAction.lambda$null$1(AbstractRestrictiveAction.java:59)
    at java.util.Optional.orElseGet
    at be.objectify.deadbolt.java.actions.AbstractRestrictiveAction.lambda$execute$2(AbstractRestrictiveAction.java:59)
    at java.util.concurrent.CompletableFuture.uniComposeStage
    at java.util.concurrent.CompletableFuture.thenCompose
    at java.util.concurrent.CompletableFuture.thenCompose
    at be.objectify.deadbolt.java.actions.AbstractRestrictiveAction.execute(AbstractRestrictiveAction.java:58)
    at be.objectify.deadbolt.java.actions.AbstractDeadboltAction.call(AbstractDeadboltAction.java:121)
    at play.core.j.JavaAction.$anonfun$apply$8(JavaAction.scala:184)
    at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687)
    at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
    at play.core.j.ClassLoaderExecutionContext.$anonfun$execute$1(ClassLoaderExecutionContext.scala:64)
    at datadog.trace.bootstrap.instrumentation.java.concurrent.RunnableWrapper.run(RunnableWrapper.java:24)
    at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:65)
    at play.core.j.ClassLoaderExecutionContext.execute(ClassLoaderExecutionContext.scala:59)
    at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:429)
    at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:338)
    at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:312)
    at scala.concurrent.impl.Promise$DefaultPromise.map(Promise.scala:182)
    at scala.concurrent.Future$.apply(Future.scala:687)
    at play.core.j.JavaAction.apply(JavaAction.scala:185)
    at play.api.mvc.Action.$anonfun$apply$6(Action.scala:83)
    at play.api.mvc.BodyParser$.$anonfun$runParserThenInvokeAction$1(Action.scala:260)
    at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:470)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:94)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker
    at java.util.concurrent.ThreadPoolExecutor$Worker.run
    at java.lang.Thread.run

Thanks in advance for your valuable insights.
Yours faithfully,
LCDP

[rbygrave]

Is there any explicit transaction that spans the loadCount() and getList() ?

[rob-bygrave]

Looking at this part:

    at io.ebeaninternal.api.SpiTransactionProxy.flush(SpiTransactionProxy.java:313)
    at io.ebeaninternal.server.query.DefaultOrmQueryEngine.flushJdbcBatchOnQuery(DefaultOrmQueryEngine.java:59)
    at io.ebeaninternal.server.query.DefaultOrmQueryEngine.findCount(DefaultOrmQueryEngine.java:83)
    at io.ebeaninternal.server.core.OrmQueryRequest.findCount(OrmQueryRequest.java:358)
    at io.ebeaninternal.server.core.DefaultServer.findCountWithCopy(DefaultServer.java:1182)

The thought is that ... we don't really want this findCountWithCopy query to trigger a flushJdbcBatchOnQuery.

Said differently ... the "async findCount running in background" query is a special case and should never trigger flushJdbcBatchOnQuery - it should be exempt and not trigger that flush somehow.

[AntoineDuComptoirDesPharmacies]

You seems right !
Rephrasing te cause, our main thread executing the DB queries (for example getList) may modify the array of transaction's statements (through add) while the findCountWithCopy is looping over them to flush.

Is my understanding correct ?
Thanks for your help

[rbygrave]

Is my understanding correct ?

Yes. That is what I think is happening here. The fix I'm going to push up is that future queries (findFutureCount, findFutureList, findFutureIds) do NOT trigger a flush of the batched prepared statement holder.