Ontology Cache: Ignore third-party properties consistently

[tobiasschweizer]

resolves https://dasch.myjetbrains.com/youtrack/issue/DSP-175

additional improvements will be done in https://dasch.myjetbrains.com/youtrack/issue/DSP-237

[tobiasschweizer]

@subotic I need your help to set up the supported release of Knora on github-ci.

[tobiasschweizer]

@subotic I also have problems getting the zip file, giving up for today ...

[subotic]

Maybe this is something @benjamingeer could take a look at? knora-api should be running at that time. I don't know why the ziped file is not sent correctly. Shouldn't the curl be called with a flag to fail if something other than 200 is returned?

[subotic]

In one of my branches the curl is called with --fail and the error returned was 500 Internal Server Error: https://github.com/dasch-swiss/knora-api/pull/1644/checks?check_run_id=668345491#step:9:921

[benjamingeer]

@subotic

In one of my branches the curl is called with --fail and the error returned was 500 Internal Server Error

I would need to see the Knora stack trace.

[benjamingeer]

Maybe this is something @benjamingeer could take a look at? knora-api should be running at that time. I don't know why the ziped file is not sent correctly.

The only way to find out would be to print out the response from Knora as well as any stack trace on the Knora console.

[subotic]

@benjamingeer Hmm, yes, the stack trace would be useful. You are good with bash, could you maybe write a wrapper bash script for curl that calls make stack-logs-api-no-follow in the case of a non 200 status?

[tobiasschweizer]

@subotic Aside from the zip problem, could you help me set up a tagged Knora release?

[tobiasschweizer]

I tried to request the zip file locally and got

{"error":"akka.stream.BufferOverflowException: Exceeded configured max-open-requests value of [64]. This means that the request queue of this pool (HostConnectionPoolSetup(0.0.0.0,3333,ConnectionPoolSetup(ConnectionPoolSettings(15,0,3,64,1,Duration.Inf,100 milliseconds,2 minutes,30 seconds,ClientConnectionSettings(Some(User-Agent: akka-http/10.1.11),9999 milliseconds,479999 milliseconds,512,None,WebSocketSettings(,ping,Duration.Inf,akka.http.impl.settings.WebSocketSettingsImpl$$$Lambda$384/0x0000000800481040@66d39d4b),List(),ParserSettings(2048,16,1024,64,8192,64,8388608,8388608,256,2097152,Strict,RFC6265,true,Set(),Full,Error,Map(If-Range -> 0, If-Modified-Since -> 0, If-Unmodified-Since -> 0, default -> 12, Content-MD5 -> 0, Date -> 0, If-Match -> 0, If-None-Match -> 0, User-Agent -> 32),false,true,akka.util.ConstantFun$$$Lambda$260/0x00000008003c9040@5be5d970,akka.util.ConstantFun$$$Lambda$260/0x00000008003c9040@5be5d970,akka.util.ConstantFun$$$Lambda$261/0x00000008003ca040@9a5c703),None,TCPTransport),New,1 second),akka.http.scaladsl.HttpConnectionContext$@7310b19c,akka.event.MarkerLoggingAdapter@6c60acf6))) has completely filled up because the pool currently does not process requests fast enough to handle the incoming request load. Please retry the request later. See http://doc.akka.io/docs/akka-http/current/scala/http/client-side/pool-overflow.html for more information."}

I worked the second time I tried.

[tobiasschweizer]

@benjamingeer @subotic I gave it a lot of Valium (432e98c) and now the test passed.

[tobiasschweizer]

@flavens now third party props like rdfs:label are consistently excluded from the entity info on a ReadResource. This should make the error message in the GUI (UI lib) disappear. I built the lib locally with yalc and integrated it into the UI lib. It works for me. It would be great if you could confirm that it works correctly.

1. check out wip/DSP-175-third-party-props into your local git repo of JS lib
2. run npm run yalc-publish from the project root
3. from your UI lib repo run yalc add @knora/api and npm i
   (like the way we did before we had the RC on npm)

In a next step, we can do https://dasch.myjetbrains.com/youtrack/issue/DSP-142 to move the logic from the UI to the JS lib.

[flavens]

[...] It works for me. It would be great if you could confirm that it works correctly.

@tobiasschweizer I confirm it works, no more console.error displayed and no more third-party prop in the given entityInfo object (only ResourceProp and SystemProp)

[benjamingeer]

@subotic

You are good with bash

I'm just good at copying and pasting things from StackOverflow. :)

could you maybe write a wrapper bash script for curl that calls make stack-logs-api-no-follow in the case of a non 200 status?

Yes, but for now could we just add make stack-logs-api-no-follow to the GitHub action here?

@tobiasschweizer

Exceeded configured max-open-requests

Maybe the problem is that all the test data requests are being done concurrently, and there are too many of them for Knora's incoming connection pool. @subotic do you know the best way to deal with this in Akka?

[tobiasschweizer]

@subotic

You are good with bash

I'm just good at copying and pasting things from StackOverflow. :)

could you maybe write a wrapper bash script for curl that calls make stack-logs-api-no-follow in the case of a non 200 status?

Yes, but for now could we just add make stack-logs-api-no-follow to the GitHub action here?

@tobiasschweizer

Exceeded configured max-open-requests

Maybe the problem is that all the test data requests are being done concurrently, and there are too many of them for Knora's incoming connection pool. @subotic do you know the best way to deal with this in Akka?

It would be great if you could find out about this because it makes the test framework unstable. Maybe it's also a problem with the resources available on github-ci. I think you also experience problems for the knora-api tests, right?

[tobiasschweizer]

I might get away with my curl options:

0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0
136
0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0
137
0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0
138
0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0
139
Warning: Transient problem: HTTP error Will retry in 1 seconds. 5 retries
140
Warning: left.
141

142
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
143
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
144
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
145
0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
146
0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0
147
0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0
148
0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0
149
0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0
150
0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0
151
0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0
152
0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0
153
0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0
154
17 83704 17 14247 0 0 1179 0 0:01:10 0:00:12 0:00:58 2923
155
100 83704 100 83704 0 0 6930 0 0:00:12 0:00:12 --:--:-- 21617
156
http_codesleep 120

[benjamingeer]

Maybe it's also a problem with the resources available on github-ci. I think you also experience problems for the knora-api tests, right?

The problem we have in the knora-api tests is that GraphDB runs out of memory, that's something different.

@subotic I think that in this case, we could just do the requests one at a time, there's no reason to do them concurrently. What do you think is the easiest way to queue them?

[tobiasschweizer]

https://youtu.be/yfl6Lu3xQW0?t=8

got away with it ...

[tobiasschweizer]

@flavens Is this PR good to go?

[flavens]

LGTM

[subotic]

@benjamingeer I will create a separate test in knora-api for this. We can then move the discussion to there.

[tobiasschweizer]

@benjamingeer I will create a separate test in knora-api for this. We can then move the discussion to there.

sounds good, thanks