CPU Problem Executing SPARQL

[naturzukunft]

I am adding models to a repository. to check the versions of the models to be added, I need to query the maximum version for an identifier for each insert.

PREFIX schema: <https://schema.org/> 
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> 
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

SELECT (MAX(?version) as ?maxVersion)
WHERE 
{
	?objectId schema:identifier "%s" .
	?objectId schema:version ?version .
	?objectId rdf:type schema:CreativeWork .
}

At the beginning, determining the versions (via HTTP SPARQL!) takes ~200ms. When I get closer to 2000 models, the duration for the query increases significantly.
With 2400 models I am already at a duration of 450ms.

I was expecting more of a memory problem, but from ~ 2000 objects onwards you can clearly see that the cpu is close to 100%. Network load drops, memory remains fairly constant. From this I deduce that my server, which executes the query, is overloading the cpu. (Of course, a lot is running in parallel there)

But I have only inserted a fraction of the models. So my import will never be finished.

[hmottestad]

Can you tell us a bit more about your setup?

• Java version
• Amount of RAM
• RDF4J version
• NativeStore or MemoryStore?
• Reasoning?
• What isolation level are you using?

Are you able use the explanation feature for SPARQL queries so we can see what's going on?

• https://rdf4j.org/documentation/programming/repository/#explaining-queries
• Preferably using both Explanation.Level.Timed and Explanation.Level.Unoptimized

[naturzukunft]

There are other parallel queries running against the same database! One complex construct query. See this thread.
I still do some optimisation. currently the query above "getLatestVersion" seems to be the biggest problem.

Measurement Point	#	Average	Min	Max	Total
LoaRdfPubAdapter:createPublication	2267	1,595.033	424.587	10,894.563	3,615,940.617
--------------------------------------------	------	-----------	---------	------------	---------------
LoaRdfPubAdapter:getConstructedPublication	1874	217.148	150.446	1,481.410	406,934.767
--------------------------------------------	------	-----------	---------	------------	---------------
LoaRdfPubAdapter:getLatestVersion	2267	1,060.500	147.258	10,179.626	2,404,154.529
--------------------------------------------	------	-----------	---------	------------	---------------
LoaRdfPubAdapter:postPublication	2267	354.374	255.835	1,772.660	803,366.303
--------------------------------------------	------	-----------	---------	------------	---------------

[hmottestad]

Looks like the query is scaling ok. The actual data that the query is joining doesn't seem to be increasing and the join order doesn't look like it should matter.

[hmottestad]

Could it be that the JVM is selecting the SerialGC because of the limited heap size? https://stackoverflow.com/questions/70664562/criteria-for-default-garbage-collector-hotspot-jvm-11-17

[hmottestad]

There are also a few cache size settings you can try to increase.

https://rdf4j.org/javadoc/3.7.7/org/eclipse/rdf4j/sail/nativerdf/NativeStore.html

[naturzukunft]

ok, i have continued debugging and testing. i have now outsourced the server part so that my laptop is relieved. in doing so, i have seen for the first time that the client builds up a lot of threads. i must take a closer look tomorrow. I have just seen that there are a lot of rdf4j-SharedHttpClientSessionManager-* threads. you will hear from me

[naturzukunft]

Shame on me. I found the problem. I was looking in completely the wrong place all along. The error was creating the SPARQLRepository on the client side. I created a new SPARQLRepository for each request. And i marked this code with a todo and a warning. But all this is of no use if you miss your hints.
Thanks again for your support!
I'll now cleanup and move the whole stuff to the servers.

Last test with running the client on my local machine:

Measurement Point	#	Average	Min	Max	Total
LoaRdfPubAdapter:createPublication	13792	274.594	173.967	948.103	3,787,206.738
--------------------------------------------	-------	---------	---------	---------	---------------
LoaRdfPubAdapter:getConstructedPublication	9092	89.409	65.313	700.371	812,909.246
--------------------------------------------	-------	---------	---------	---------	---------------
LoaRdfPubAdapter:getLatestVersion	13792	78.690	53.920	694.373	1,085,293.251
--------------------------------------------	-------	---------	---------	---------	---------------
LoaRdfPubAdapter:postPublication	13792	136.087	105.841	800.349	1,876,905.857
--------------------------------------------	-------	---------	---------	---------	---------------

DURATION: 559 seconds for 5000 kvmEntries.