uv-l-relationaldiffToCkan - OutOfMemoryError: Requested array size exceeds VM limit

[hanecak]

Following pipeline:

https://odn.opendata.sk/unifiedviews/#!PipelineEdit/3

tries to load http://odn.opendata.sk/datanest-mirror/organizations/organizations.csv into CKAN using uv-l-relationaldiffToCkan.

organizations.csv is 1GB big. uv-t-tabularToRelational is able to extract data from it but an attempt to load that data into CKAN DataStore by uv-l-relationaldiffToCkan fails with:

Execution failed because: DPU.innerExecute throws throwable.eu.unifiedviews.dpu.DPUException: DPU.innerExecute throws throwable. at eu.unifiedviews.helpers.dpu.exec.AbstractDpu.execute(AbstractDpu.java:141) at cz.cuni.mff.xrg.odcs.backend.execution.dpu.DPUExecutor.executeInstance(DPUExecutor.java:248) at cz.cuni.mff.xrg.odcs.backend.execution.dpu.DPUExecutor.execute(DPUExecutor.java:386) at cz.cuni.mff.xrg.odcs.backend.execution.dpu.DPUExecutor.run(DPUExecutor.java:468) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.util.Arrays.copyOf(Arrays.java:2367) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535) at java.lang.StringBuffer.append(StringBuffer.java:322) at java.io.StringWriter.write(StringWriter.java:94) at org.glassfish.json.JsonGeneratorImpl.flushBuffer(JsonGeneratorImpl.java:629) at org.glassfish.json.JsonGeneratorImpl.writeString(JsonGeneratorImpl.java:586) at org.glassfish.json.JsonGeneratorImpl.writeEscapedString(JsonGeneratorImpl.java:545) at org.glassfish.json.JsonGeneratorImpl.writeName(JsonGeneratorImpl.java:171) at org.glassfish.json.JsonGeneratorImpl.write(JsonGeneratorImpl.java:182) at org.glassfish.json.JsonGeneratorImpl.write(JsonGeneratorImpl.java:365) at org.glassfish.json.JsonGeneratorImpl.write(JsonGeneratorImpl.java:286) at org.glassfish.json.JsonGeneratorImpl.write(JsonGeneratorImpl.java:351) at org.glassfish.json.JsonWriterImpl.writeObject(JsonWriterImpl.java:124) at org.glassfish.json.JsonWriterImpl.write(JsonWriterImpl.java:143) at org.glassfish.json.JsonObjectBuilderImpl$JsonObjectImpl.toString(JsonObjectBuilderImpl.java:267) at org.opendatanode.plugins.loader.relationaldifftockan.RelationalDiffToCkan.updateAuditedDatastoreFromTable(RelationalDiffToCkan.java:590) at org.opendatanode.plugins.loader.relationaldifftockan.RelationalDiffToCkan.innerExecute(RelationalDiffToCkan.java:205) at eu.unifiedviews.helpers.dpu.exec.AbstractDpu.execute(AbstractDpu.java:133) ... 4 more

ODN 1.2.3 is used.

It may be an issue of simply trying too much at one go. Increasing heap size does not help.

[hanecak]

Issue still present in ODN 1.4.0, with slightly different stack trace:

Execution failed because: DPU.innerExecute throws throwable.eu.unifiedviews.dpu.DPUException: DPU.innerExecute throws throwable. at eu.unifiedviews.helpers.dpu.exec.AbstractDpu.execute(AbstractDpu.java:141) at cz.cuni.mff.xrg.odcs.backend.execution.dpu.DPUExecutor.executeInstance(DPUExecutor.java:248) at cz.cuni.mff.xrg.odcs.backend.execution.dpu.DPUExecutor.execute(DPUExecutor.java:386) at cz.cuni.mff.xrg.odcs.backend.execution.dpu.DPUExecutor.run(DPUExecutor.java:468) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2367) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535) at java.lang.StringBuffer.append(StringBuffer.java:322) at java.io.StringWriter.write(StringWriter.java:94) at org.glassfish.json.JsonGeneratorImpl.flushBuffer(JsonGeneratorImpl.java:629) at org.glassfish.json.JsonGeneratorImpl.writeString(JsonGeneratorImpl.java:586) at org.glassfish.json.JsonGeneratorImpl.writeEscapedString(JsonGeneratorImpl.java:545) at org.glassfish.json.JsonGeneratorImpl.write(JsonGeneratorImpl.java:183) at org.glassfish.json.JsonGeneratorImpl.write(JsonGeneratorImpl.java:365) at org.glassfish.json.JsonGeneratorImpl.write(JsonGeneratorImpl.java:286) at org.glassfish.json.JsonGeneratorImpl.write(JsonGeneratorImpl.java:351) at org.glassfish.json.JsonWriterImpl.writeObject(JsonWriterImpl.java:124) at org.glassfish.json.JsonWriterImpl.write(JsonWriterImpl.java:143) at org.glassfish.json.JsonObjectBuilderImpl$JsonObjectImpl.toString(JsonObjectBuilderImpl.java:267) at org.opendatanode.plugins.loader.relationaldifftockan.RelationalDiffToCkan.updateAuditedDatastoreFromTable(RelationalDiffToCkan.java:590) at org.opendatanode.plugins.loader.relationaldifftockan.RelationalDiffToCkan.innerExecute(RelationalDiffToCkan.java:205) at eu.unifiedviews.helpers.dpu.exec.AbstractDpu.execute(AbstractDpu.java:133) ... 4 more