Configure graphdb for production #692

tobiasschweizer · 2017-12-13T16:58:16Z

Try to find a reasonable setup for graphdb (8.3.1) that prevents the triplestore from crashing but does not break normal queries.

closes #685
closes #549

…time in graphdb

tobiasschweizer · 2017-12-14T10:10:15Z

The last pattern of the WHERE clause in the query below contains ?date <http://www.knora.org/ontology/knora-base#valueHasStartJDN> ?date__valueHasStartJDN . and ?date is not bound. The query is clearly wrong, but it shouldn't make graphdb crash. With KnarQL, people can send all sorts of wrong queries to Knora that are syntactically valid but contain unbound variables. I am not sure if we can check for everyting in a sanity checker (#645).

With the query-timeout set to 2 seconds the following query.

SELECT DISTINCT ?letter (GROUP_CONCAT(?person1; SEPARATOR='') AS ?person1Concat) (GROUP_CONCAT(?date; SEPARATOR='') AS ?dateConcat) (GROUP_CONCAT(?person1__LinkValue; SEPARATOR='') AS ?person1__LinkValueConcat)
 WHERE {
 {
 ?letter <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#Resource> .
 GRAPH <http://www.ontotext.com/explicit> {
     ?letter <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?letter <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/beol#letter> .
 ?letter <http://www.knora.org/ontology/beol#creationDate> ?date .
 GRAPH <http://www.ontotext.com/explicit> {
     ?date <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 } UNION {
 ?letter <http://www.knora.org/ontology/beol#hasAuthor> ?person1 .
 ?letter <http://www.knora.org/ontology/beol#hasAuthorValue> ?person1__LinkValue .
 GRAPH <http://www.ontotext.com/explicit> {
     ?person1__LinkValue <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#LinkValue> .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?person1__LinkValue <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?person1__LinkValue <http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> ?letter .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?person1__LinkValue <http://www.w3.org/1999/02/22-rdf-syntax-ns#object> ?person1 .
 }
 ?person1 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#Resource> .
 GRAPH <http://www.ontotext.com/explicit> {
     ?person1 <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?person1 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/beol#person> .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?date <http://www.knora.org/ontology/knora-base#valueHasStartJDN> ?date__valueHasStartJDN .
 }
 }
 
 GROUP BY ?letter ?date__valueHasStartJDN
 ORDER BY ASC(?date__valueHasStartJDN) ASC(?letter)
 OFFSET 0
 LIMIT 25

is aborted:

[ERROR] 2017-12-14 11:07:32,027 [repositories/knora-test | o.e.r.h.s.r.TupleQueryResultView] Query interrupted
org.eclipse.rdf4j.query.QueryInterruptedException: Query evaluation took too long
	at com.ontotext.trree.SailConnectionImpl$4.throwInterruptedException(SailConnectionImpl.java:1571)
	at org.eclipse.rdf4j.common.iteration.TimeLimitIteration.checkInterrupted(TimeLimitIteration.java:116)
	at org.eclipse.rdf4j.common.iteration.TimeLimitIteration.hasNext(TimeLimitIteration.java:51)
	at org.eclipse.rdf4j.common.iteration.IterationWrapper.hasNext(IterationWrapper.java:63)
	at org.eclipse.rdf4j.common.iteration.LimitIteration.hasNext(LimitIteration.java:72)
	at com.ontotext.trree.query.LoggingClosableIteration.hasNext(LoggingClosableIteration.java:56)
	at org.eclipse.rdf4j.common.iteration.IterationWrapper.hasNext(IterationWrapper.java:63)
	at org.eclipse.rdf4j.query.QueryResults.report(QueryResults.java:256)
	at org.eclipse.rdf4j.http.server.repository.TupleQueryResultView.renderInternal(TupleQueryResultView.java:100)
	at org.eclipse.rdf4j.http.server.repository.QueryResultView.render(QueryResultView.java:64)
	at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1244)
	at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1027)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:971)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at com.ontotext.forest.core.request.RequestFilter.doFilterInternal(RequestFilter.java:41)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at com.github.ziplet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:300)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at com.ontotext.forest.security.AdminDelegatingFilterProxy.doFilter(AdminDelegatingFilterProxy.java:29)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
[INFO ] 2017-12-14 11:07:32,028 [repositories/knora-test | o.e.r.h.s.r.TupleQueryResultView] Request for query 65219118 is finished

tobiasschweizer · 2017-12-14T10:17:54Z

With query-timeout set to 3 seconds, the query is not aborted the first time I run it after having reloaded the data into the triplestore:

[INFO ] 2017-12-14 11:17:15,179 [repositories/knora-test | c.o.f.s.RepositoryController] POST query 65219118
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to /Users/tobi/graphdb/graphdb-se-8.3.1/heapdump.hprof ...
Unable to create /Users/tobi/graphdb/graphdb-se-8.3.1/heapdump.hprof: File exists
#
# java.lang.OutOfMemoryError: GC overhead limit exceeded
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing "kill -9 1407"...
Killed: 9

I restarted the triplestore and then it was aborted correctly. This is hard to reproduce.

When increasing the limit to 5 seconds, it always crashes.

The config looks ok:

#
# RDF4J configuration template for a GraphDB SE repository
#
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>.
@prefix rep: <http://www.openrdf.org/config/repository#>.
@prefix sr: <http://www.openrdf.org/config/repository/sail#>.
@prefix sail: <http://www.openrdf.org/config/sail#>.
@prefix owlim: <http://www.ontotext.com/trree/owlim#>.

[] a rep:Repository ;
    rep:repositoryID "knora-test" ;
    rdfs:label "Knora test repository" ;
    rep:repositoryImpl [
        rep:repositoryType "owlim:MonitorRepository" ;
        sr:sailImpl [
            sail:sailType "owlim:Sail" ;
            
            owlim:owlim-license "" ;
       
            owlim:base-URL "http://data.knora.org/" ;
            owlim:defaultNS "" ;
            owlim:entity-index-size "10000000" ;
            owlim:entity-id-size  "32" ;
            owlim:imports "" ;
        	owlim:repository-type "file-repository" ;
            owlim:ruleset ".../Knora_github/webapi/scripts/KnoraRules.pie" ;
            owlim:storage-folder "storage" ;
 
            owlim:enable-context-index "false" ;

            owlim:enablePredicateList "true" ;

            owlim:in-memory-literal-properties "true" ;
            owlim:enable-literal-index "true" ;

            owlim:check-for-inconsistencies "true" ;
            owlim:disable-sameAs  "true" ;
            owlim:query-timeout  "5" ;
            owlim:query-limit-results  "50000" ;
            owlim:throw-QueryEvaluationException-on-timeout "true" ;
            owlim:read-only "false" ;
        ]
    ].

tobiasschweizer · 2018-01-04T09:29:45Z

The problem is that the concatenated string (GROUP_CONCAT) grows very fast due to the unbound variable in the query.

Ontotext will add an option so that the length of the concatenated string can be limited (as of release 8.5 of graphdb).

tobiasschweizer · 2018-03-01T07:13:02Z

Ontotext integrated the configuration option PARAM_CONCAT_MAX_LENGTH as of version 8.5.0-RC4 (which is a stable release candidate).

I am trying to reproduce the problem now.

tobiasschweizer · 2018-03-01T08:45:35Z

@subotic As of 8.5.0-RC4 query execution stops after reaching the time limit (before the triplestore crashed with large strings produced by GROUP_CONCAT). Productive versions of Knora should thus use graphdb 8.5 as soon as it is available.

…five seconds

tobiasschweizer · 2018-03-01T08:56:30Z

@subotic @benjamingeer I set the max. query execution time to five seconds and the limit of results to 50000. That seems reasonable to me.

subotic

This doesn't break any of our existing tests and older versions of GraphDB don't complain, so LGTM.

As soon as the official release is out, I will update Travis to use the new GraphDB version.

benjamingeer

Very reasonable.

feature (graphdb settings): first attempt to configure max execution …

4652cae

…time in graphdb

Merge branch 'develop' into wip/limit_max_query_time

6702979

tobiasschweizer added 3 commits January 31, 2018 11:42

Merge branch 'develop' into wip/limit_max_query_time

d228296

Merge branch 'develop' into wip/limit_max_query_time

ec63098

Merge branch 'develop' into wip/limit_max_query_time

48163a9

refactor (configuration of triplestore repo): set max. query time to …

8308117

…five seconds

tobiasschweizer requested review from subotic and benjamingeer March 1, 2018 08:56

tobiasschweizer self-assigned this Mar 1, 2018

tobiasschweizer added the enhancement improve existing code or new feature label Mar 1, 2018

tobiasschweizer modified the milestones: v1.3.0, v1.2.0 Mar 1, 2018

subotic approved these changes Mar 1, 2018

View reviewed changes

benjamingeer approved these changes Mar 5, 2018

View reviewed changes

Benjamin Geer and others added 2 commits March 5, 2018 17:21

Merge branch 'develop' into wip/limit_max_query_time

428716b

Merge branch 'develop' into wip/limit_max_query_time

8876513

tobiasschweizer merged commit 1c6b6a1 into develop Mar 6, 2018

tobiasschweizer deleted the wip/limit_max_query_time branch March 6, 2018 09:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configure graphdb for production #692

Configure graphdb for production #692

tobiasschweizer commented Dec 13, 2017 •

edited

Loading

tobiasschweizer commented Dec 14, 2017 •

edited

Loading

tobiasschweizer commented Dec 14, 2017 •

edited

Loading

tobiasschweizer commented Jan 4, 2018 •

edited

Loading

tobiasschweizer commented Mar 1, 2018

tobiasschweizer commented Mar 1, 2018

tobiasschweizer commented Mar 1, 2018

subotic left a comment

benjamingeer left a comment

Configure graphdb for production #692

Configure graphdb for production #692

Conversation

tobiasschweizer commented Dec 13, 2017 • edited Loading

tobiasschweizer commented Dec 14, 2017 • edited Loading

tobiasschweizer commented Dec 14, 2017 • edited Loading

tobiasschweizer commented Jan 4, 2018 • edited Loading

tobiasschweizer commented Mar 1, 2018

tobiasschweizer commented Mar 1, 2018

tobiasschweizer commented Mar 1, 2018

subotic left a comment

Choose a reason for hiding this comment

benjamingeer left a comment

Choose a reason for hiding this comment

tobiasschweizer commented Dec 13, 2017 •

edited

Loading

tobiasschweizer commented Dec 14, 2017 •

edited

Loading

tobiasschweizer commented Dec 14, 2017 •

edited

Loading

tobiasschweizer commented Jan 4, 2018 •

edited

Loading