Capacity exceeds max buffer capacity: 134217728

[pareshD-1991]

Hi Team,

While fetching or deleteing the table we are getting below exception.
Exception we are getting from janusgraph, also the value is hardcoded -> Is there any way we can overide this value.
Any other workaround

2024-06-19 10:21:05,812 ERROR org.janusgraph.graphdb.database.StandardJanusGraph: [NotificationHookConsumer thread-0]: Could not commit transaction [93619] due to exception java.lang.IllegalArgumentException: Capacity exceeds max buffer capacity: 134217728 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:191) ~[guava-32.0.1-jre.jar:?] at org.janusgraph.diskstorage.util.WriteByteBuffer.require(WriteByteBuffer.java:56) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.diskstorage.util.WriteByteBuffer.putBytes(WriteByteBuffer.java:106) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.graphdb.database.serialize.StandardSerializer$StandardDataOutput.putBytes(StandardSerializer.java:427) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.diskstorage.util.BufferUtil.writeBuffer(BufferUtil.java:111) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.diskstorage.util.BufferUtil.writeEntry(BufferUtil.java:94) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.graphdb.database.log.TransactionLogHeader.logRelations(TransactionLogHeader.java:90) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.graphdb.database.log.TransactionLogHeader.serializeModifications(TransactionLogHeader.java:81) ~[janusgraph-core-0.6.1.jar:?] at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:733) [janusgraph-core-0.6.1.jar:?] at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1525) [janusgraph-core-0.6.1.jar:?] at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.doCommit(JanusGraphBlueprintsGraph.java:322) [janusgraph-core-0.6.1.jar:?] at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:104) [gremlin-core-3.5.5.jar:3.5.5] at org.janusgraph.graphdb.tinkerpop.JanusGraphBlueprintsGraph$GraphTransaction.commit(JanusGraphBlueprintsGraph.java:300) [janusgraph-core-0.6.1.jar:?] at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraph.commit(AtlasJanusGraph.java:248) [atlas-graphdb-janus-2.1.0.7.2.17.400-56.jar:2.1.0.7.2.17.400-56] at org.apache.atlas.GraphTransactionInterceptor.doCommit(GraphTransactionInterceptor.java:190) [atlas-repository-2.1.0.7.2.17.400-56.jar:?] at org.apache.atlas.GraphTransactionInterceptor.doCommitOrRollback(GraphTransactionInterceptor.java:185) [atlas-repository-2.1.0.7.2.17.400-56.jar:?] at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:120) [atlas-repository-2.1.0.7.2.17.400-56.jar:?] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) [spring-aop-5.3.27.jar:5.3.27] at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) [spring-aop-5.3.27.jar:5.3.27] at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) [spring-aop-5.3.27.jar:5.3.27] at org.apache.atlas.repository.store.graph.v2.AtlasEntityStoreV2$$EnhancerBySpringCGLIB$$d856b08e.deleteByUniqueAttributes(<generated>) [atlas-repository-2.1.0.7.2.17.400-56.jar:2.1.0.7.2.17.400-56] at org.apache.atlas.notification.SerialEntityProcessor.handleMessage(SerialEntityProcessor.java:476) [atlas-webapp-2.1.0.7.2.17.400-56.jar:2.1.0.7.2.17.400-56] at org.apache.atlas.notification.SerialEntityProcessor.handleMessage(SerialEntityProcessor.java:274) [atlas-webapp-2.1.0.7.2.17.400-56.jar:2.1.0.7.2.17.400-56] at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.handleMessage(NotificationHookConsumer.java:489) [atlas-webapp-2.1.0.7.2.17.400-56.jar:2.1.0.7.2.17.400-56] at org.apache.atlas.notification.NotificationHookConsumer$HookConsumer.run(NotificationHookConsumer.java:402) [atlas-webapp-2.1.0.7.2.17.400-56.jar:2.1.0.7.2.17.400-56] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_402]

[hadoopmarc]

128M seems a reasonable maximum, related to all kinds of HD hardware buffers. The problem is more likely in one of the callers, e.g.

janusgraph/janusgraph-core/src/main/java/org/janusgraph/diskstorage/locking/consistentkey/ConsistentKeyLockerSerializer.java

Line 35 in 13619a1

WriteBuffer b = new WriteByteBuffer(key.length() + column.length() + 4);

This method could rather read the max buffer size and do sequences of puts to the buffer.

Of course, you should also check where the large data in your database stems from and whether it is a pattern you want to support and think the community should support.

[pareshD-1991]

I think community should support Capacity size. It should be configurable such as through property or any jvm set.

[pareshD-1991]

Hi Team, Any workaround for this issue.

[chaitalicod]

Hello @hadoopmarc , Iam also stuck with similar error ,Do you know if there is any workaround to unblock in such situations ?

[hadoopmarc]

If you really want to override this limit, you have to change the value in the code and do a custom build. Because building might be troublesome to set up (the project is huge), it could be easier to just build this class against the jars from the distribution and put it in a separate jar. Since the JVM only loads classes that have not been loaded yet, you can have prevail you class over the WriteByteBuffer class from the distribution by having it in front of the existing CLASSPATH. Obviously, this solution is not suitable for production, but at least you can test whether this solves your problems or produces new ones.