OutOfMemoryError when using SPARQLResultsJSONWriter

[AlyHdr]

Current Behavior

We are trying to write results back using a SPARQLResultsJSONWriter but we get an OutOfMemoryError on the given output stream. The code snippet below shows how we initiate the writer and how the result is returned back:

public ByteArrayOutputStream executeJson(String sparqlQuery, int timeout) throws Exception {
        RepositoryConnection connection = repository.getConnection();
        logger.info("Running given sparql query: " + sparqlQuery);
        ParsedQuery parsedQuery =
                QueryParserUtil.parseQuery(QueryLanguage.SPARQL, sparqlQuery, null);

        if (parsedQuery instanceof ParsedTupleQuery) {
            TupleQuery query = connection.prepareTupleQuery(sparqlQuery);
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            TupleQueryResultHandler writer = new SPARQLResultsJSONWriter(out);
            query.setMaxExecutionTime(timeout);
            try {
                query.evaluate(writer);
            } catch (QueryEvaluationException q){
                logger.error("This exception was caught ["+q+"]");
            }
            finally {
                connection.close();
            }
            return out;
      }
}

The stack trace is the following:

java.lang.OutOfMemoryError: null
        at java.base/java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:125) ~[na:na]
        at java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:119) ~[na:na]
        at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95) ~[na:na]
        at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156) ~[na:na]
        at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[na:na]
        at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) ~[na:na]
        at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:233) ~[na:na]
        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:303) ~[na:na]
        at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:281) ~[na:na]
        at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) ~[na:na]
        at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:208) ~[na:na]
        at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator._flushBuffer(WriterBasedJsonGenerator.java:1994) ~[jackson-core-2.10.1.jar!/:2.10.1]
        at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.writeRawLong(WriterBasedJsonGenerator.java:616) ~[jackson-core-2.10.1.jar!/:2.10.1]
        at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.writeRaw(WriterBasedJsonGenerator.java:549) ~[jackson-core-2.10.1.jar!/:2.10.1]
        at com.fasterxml.jackson.core.util.DefaultPrettyPrinter.writeObjectFieldValueSeparator(DefaultPrettyPrinter.java:304) ~[jackson-core-2.10.1.jar!/:2.10.1]
        at com.fasterxml.jackson.core.json.JsonGeneratorImpl._verifyPrettyValueWrite(JsonGeneratorImpl.java:224) ~[jackson-core-2.10.1.jar!/:2.10.1]
        at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator._verifyValueWrite(WriterBasedJsonGenerator.java:894) ~[jackson-core-2.10.1.jar!/:2.10.1]
        at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.writeStartObject(WriterBasedJsonGenerator.java:296) ~[jackson-core-2.10.1.jar!/:2.10.1]
        at org.eclipse.rdf4j.query.resultio.sparqljson.AbstractSPARQLJSONWriter.writeValue(AbstractSPARQLJSONWriter.java:278) ~[rdf4j-queryresultio-sparqljson-3.4.6-SNAPSHOT.jar!/:3.4.6-SNAPSHOT+b43cea0]
        at org.eclipse.rdf4j.query.resultio.sparqljson.SPARQLResultsJSONWriter.writeValue(SPARQLResultsJSONWriter.java:69) ~[rdf4j-queryresultio-sparqljson-3.4.6-SNAPSHOT.jar!/:3.4.6-SNAPSHOT+b43cea0]
        at org.eclipse.rdf4j.query.resultio.sparqljson.AbstractSPARQLJSONWriter.handleSolutionImpl(AbstractSPARQLJSONWriter.java:160) ~[rdf4j-queryresultio-sparqljson-3.4.6-SNAPSHOT.jar!/:3.4.6-SNAPSHOT+b43cea0]
        at org.eclipse.rdf4j.query.resultio.AbstractQueryResultWriter.handleSolution(AbstractQueryResultWriter.java:78) ~[rdf4j-queryresultio-api-3.4.6-SNAPSHOT.jar!/:3.4.6-SNAPSHOT+b43cea0]
        at org.eclipse.rdf4j.query.QueryResults.report(QueryResults.java:267) ~[rdf4j-query-3.4.6-SNAPSHOT.jar!/:3.4.6-SNAPSHOT+b43cea0]
        at org.eclipse.rdf4j.repository.sail.SailTupleQuery.evaluate(SailTupleQuery.java:72) ~[rdf4j-repository-sail-3.4.6-SNAPSHOT.jar!/:3.4.6-SNAPSHOT+b43cea0]

P.S: The expected result is huge and we believe that the output stream is getting bigger and bigger up until it throws this error, so is there a way to use some kind of buffering to avoid this problem?

Expected Behavior

Return back the result in json format

Steps To Reproduce

No response

Version

3.4.6

Are you interested in contributing a solution yourself?

No response

Anything else?

No response

[barthanssens]

Well, the ByteArrayOutputStream is kept entirely in memory, so writing to a FileOutputStream or a PipedOutputstream might be an option ? Depends a bit on the application ...

Or eg. Apache Commons-IO DeferredFileOutputStream (you may have to delete the file separately...)

PS you might also move the connection part to a try-with-resources block, which would remove the need for a finally

try (RepositoryConnection connection = repository.getConnection()) {
...
}

[AlyHdr]

Well the application is a REST spring boot application so would like to return this result back over http ... ( converting the ByteArrayOutputStream to a String was as well problematic).

[barthanssens]

Part of the problem is the ByteArrayOutputStream copy / resize behavior.
Spring has a better / more memory-efficient implementation (org.springframework.util.FastByteArrayOutputStream),
and I'd also suggest to initialize it with a reasonably large byte size instead of the default (which is only a few bytes)

Increasing the java heap space of the JVM might also help (-Xmx)

But that still may not make that much of a difference for really big result sets...

Not a Spring expert, but wouldn't it be possible to open an OutputStream outside this method, and pass it in as a parameter, so the results are written directly to the HTTP response, instead of writing / copying things ?

https://www.logicbig.com/how-to/code-snippets/jcode-spring-mvc-streamingresponsebody.html

return new StreamingResponseBody() {
    @Override
    public void writeTo (OutputStream out) throws IOException {
       (obtain query and timeout somewhere)
        executeJson(sparqlQuery, timeout, out);
    }
}

...

executeJson(String sparqlQuery, int timeout, OutputStream out) {
...
   (no need for ByteArrayOutputStream)
    TupleQueryResultHandler writer = new SPARQLResultsJSONWriter(out);
...
}

[hmottestad]

Maybe you want to look into spring StreamingResponseBody or grabbing the output stream from HttpServletResponse.