Implement smart IRIs #662

benjamingeer · 2017-11-07T14:38:46Z

Add a SmartIri class to simplify IRI inspection, validation, and conversion.
Rename methods in StringFormatter to clarify which ones inspect (is*), which ones validate (validate*), and which ones convert (to*).
Use SmartIri for the IRIs of ontologies and ontology entities in API v2 code.
Use SmartIri for all IRIs in KnarQL.
Don't use v2 ontology info classes in v1 code, because that would mean changing a lot of v1 code to use SmartIri. Instead, add v1 wrappers around v2 ontology info classes. In the wrappers, convert smart IRIs to string IRIs. Use only the wrappers in v1 code.
Refactor v2 ontology info classes so they can support both read and write operations using the wrapper pattern. The read wrappers can convert themselves from internal to external, and the wrapped classes can convert themselves from internal to external and vice versa.
Separate schema conversion from JSON-LD generation. When serving ontology info, first convert IRIs from internal to external as needed, generating a new ReadEntityDefinitionsV2 instance. Then convert that instance to JSON-LD.
For better performance, cache SmartIri instances representing Knora ontology entities..

…ases.

…D in ontology classes. - Use SmartIri in constant ontologies, too. - Fix a lot of compile errors (still many left).

- Separate ontology info schema conversion from JSON-LD generation.

# Conflicts: # webapi/src/main/scala/org/knora/webapi/util/KnoraIdUtil.scala # webapi/src/main/scala/org/knora/webapi/util/StringFormatter.scala # webapi/src/test/scala/org/knora/webapi/responders/v2/OntologyResponderV2Spec.scala # webapi/src/test/scala/org/knora/webapi/util/StringFormatterSpec.scala

- Remove IRI caching, because performance is good anyway.

- Put back IRI caching, but only for Knora definition IRIs. - I think OntologyV2R2RSpec test data is wrong (has DatatypeProperty where it should have ObjectProperty).

benjamingeer · 2017-11-10T22:35:26Z

@tobiasschweizer SearchRouteV2R2RSpec still has errors, and I need your help to understand why:

org.knora.webapi.NotImplementedException: Value type http://www.knora.org/ontology/knora-base#DateValue not supported in FilterExpression
	at org.knora.webapi.responders.v2.SearchResponderV2$AbstractExtendedSearchTransformer.transformFilterExpression(SearchResponderV2.scala:832)

Looking at that part of the code in SearchResponderV2, it's looking for things like OntologyConstants.KnoraBase.Date, which doesn't seem like it should exist. There are comments like this in OntologyConstants:

val Date: IRI = KnoraBasePrefixExpansion + "Date" // TODO: find a better solution for v2 simple

Could you explain? I'm guessing that you did this because it would have been more work to convert the IRI knora-api:Date to the IRI knora-base:DateValue. But this is done automatically now by SmartIri. So should this code in SearchResponderV2 now look for knora-base:DateValue?

- Cache some known non-Knora definition IRIs.

…bjects.

tobiasschweizer · 2017-11-13T09:27:38Z

The reason is that in a KnarQL query the user would use simple Iris like a knora-api:Date that will be converted to a knora-base:DateValue. The main rationale is hiding the value object complexity from the user (to her it is simply a date, not a value object). However, we could surely discuss if omitting Value makes a difference at all.

In SearchResponderV2 there are many places where Iris have to converted from external to internal (when looking at type annotations) or from internal to external (when comparing an already converted Iri to the given type annotations). It would make the code much clearer if smart Iris handled that automatically. We can also look at that later today.

tobiasschweizer · 2017-11-13T09:30:03Z

Looking at that part of the code in SearchResponderV2, it's looking for things like OntologyConstants.KnoraBase.Date, which doesn't seem like it should exist.

Yes, it was kind of a hack ;-)

benjamingeer · 2017-11-13T09:39:51Z

Maybe my question wasn’t clear. I’m asking why the search responder refers to an entity called http://www.knora.org/ontology/knora-base#Date instead of http://www.knora.org/ontology/knora-api/simple/v2#Date. There is no Date in knora-base, only in knora-api/simple/v2. Why does the search responder refer to a nonexistent Date entity in knora-base? Why doesn’t it use the Date entity that does exist in knora-api/simple/v2?

SmartIri can convert http://www.knora.org/ontology/knora-base#DateValue to http://www.knora.org/ontology/knora-api/simple/v2#Date. Perhaps that is what you need?

tobiasschweizer · 2017-11-13T09:51:23Z

Simply because the external Iri (type annotation) is converted to an internal one before looking at it, using a regex. And the regex just replaces a part of the Iri, but does not handle the missing Value:

https://github.com/dhlab-basel/Knora/blob/9236b0fca594dc035e39b31ae56dbb292cd2058f/webapi/src/main/scala/org/knora/webapi/responders/v2/SearchResponderV2.scala#L602

However, this is not really consistent behavior since there are also non Knora types like xsd:string that cannot be converted.

So I would be happy to change this.

benjamingeer · 2017-11-13T10:02:00Z

So I would be happy to change this.

OK, could you please do it on this branch? Then I think SearchRouteV2R2RSpec should pass.

I already changed the KnarQL parser and type inspector so they use SmartIri. Once you have a SmartIri, you can just call toOntologySchema on it to convert it from external to internal and vice versa.

tobiasschweizer · 2017-11-13T10:04:12Z

OK, could you please do it on this branch? Then I think SearchRouteV2R2RSpec should pass.

I will look at it :-)

- Don't parse new smart IRIs when it's not neceessary to do so.

…SmartIri. - Remove old API v2 plans doc, which has already been implemented and superseded. - Add route for getting ontology metadata per project. - Add R2R tests for getting ontology metadata. - Rename "namedgraph" to "allentities" in ontology routes. - Add KnoraContentV2 trait with toOntologySchema method.

benjamingeer · 2017-11-16T18:35:53Z

@tobiasschweizer I renamed the /ontologies/namedgraphs route to /ontologies/allentities. The route that just returns ontology metadata is now called /ontologies/metadata.

tobiasschweizer · 2017-11-17T07:47:53Z

@benjamingeer Thank you very much for the notification. I will make a new branch in SALSAH that will reflect this change and other changes on this branch that effect the routes or the messages.

Actually, we need to do coordinated merges: basically you need to open both pull requests and hit the merge buttons at the same time. I guess we need tentacles for this.

benjamingeer · 2017-11-17T14:45:18Z

@tobiasschweizer I started added some API v design docs, including a discussion of SmartIri, in docs/rst/knora-api-server/design-documentation/api-v2.rst. See what you think.

tobiasschweizer · 2017-11-20T08:06:58Z

Ok, I look at it now. Just making myself a cappuccino ...

tobiasschweizer · 2017-11-20T09:03:24Z

docs/rst/knora-api-server/design-documentation/api-v2.rst

+Each API response is represented by a class that extends ``KnoraResponseV2``, which
+has a method ``toJsonLDDocument`` that specifies the target schema. It is currently
+up to each route to determine what the appropriate response schema should be. Some routes
+will support only one respnse schema. Others will allow the client to choose, and there will


respnse -> response

tobiasschweizer · 2017-11-20T09:04:45Z

docs/rst/knora-api-server/design-documentation/api-v2.rst

+contains the value's IRI. When a value is created, it is wrapped in a ``CreateValueV2``, which has
+the resource IRI and the property IRI, but not the value IRI.
+
+A ``Read*`` wrapper can be wrapped in anotehr ``Read*`` wrapper; for example, a ``ReadResourceV2``


anotehr -> another

tobiasschweizer · 2017-11-20T09:56:30Z

I had a look at it and I think that it is very useful!

tobiasschweizer · 2017-11-20T10:23:54Z

I am adding more KnarQL R2R tests now to make sure that the changes on this branch do not have unintended effects.

tobiasschweizer · 2017-11-20T11:28:36Z

haha, found a bug in my code!!!!!!!!

…y be used for sorting

Benjamin Geer added 5 commits November 7, 2017 15:37

refactor (webapi): Start refactoring StringFormatter to clarify use c…

3e33ae2

…ases.

fix (webapi): Fix compile errors.

e55a6e0

refactor (webapi): Add SmartIri class to simplify IRI manipulation.

0b174d6

refactor (webapi): Separate IRI conversions from generation of JSON-L…

89bc46e

…D in ontology classes. - Use SmartIri in constant ontologies, too. - Fix a lot of compile errors (still many left).

refactor (webapi): Use SmartIri for ontology and entity IRIs in API v2.

f96e92f

- Separate ontology info schema conversion from JSON-LD generation.

benjamingeer changed the title ~~Ontology update infrastructure part 5~~ Implement smart IRis Nov 8, 2017

benjamingeer changed the title ~~Implement smart IRis~~ Implement smart IRIs Nov 8, 2017

benjamingeer mentioned this pull request Nov 9, 2017

Return more information about named graphs #651

Merged

Benjamin Geer added 6 commits November 10, 2017 12:22

feature (webapi): Cache SmartIri instances for better performance.

fda70ee

refactor (webapi): Simplify IRI parsing, removing regular expressions.

cdab454

- Remove IRI caching, because performance is good anyway.

fix (webapi): Fix bugs in IRI schema conversion.

07f7690

- Put back IRI caching, but only for Knora definition IRIs. - I think OntologyV2R2RSpec test data is wrong (has DatatypeProperty where it should have ObjectProperty).

fix (webapi): Fix more bugs in API schema conversions and in test data.

9bd8162

fix (webapi): Fix more parsing bugs and broken tests.

e58f609

benjamingeer requested a review from tobiasschweizer November 10, 2017 22:43

benjamingeer assigned benjamingeer and tobiasschweizer Nov 10, 2017

Benjamin Geer added 4 commits November 11, 2017 11:31

feature (webapi): Fix error-checking of IRIs in SearchParserV2.

8ea2713

- Cache some known non-Knora definition IRIs.

refactor (webapi): Use SmartIri for resource classes in search messages.

525e5d8

feature (webapi): Require API v2 simple IRIs in KnarQL.

bd0a971

fix (webapi): Allow non-Knora IRIs in KnarQL, but not ApiV2WithValueO…

5d412c8

…bjects.

Benjamin Geer added 2 commits November 13, 2017 18:04

feature (webapi): Data IRIs don't get an ontology schema.

2b0dd57

- Don't parse new smart IRIs when it's not neceessary to do so.

test (webapi): Start making tests more specific for StringFormatter.

3a34901

tobiasschweizer mentioned this pull request Nov 17, 2017

Adapt to changes in Knora's ontology route dhlab-basel/Salsah#103

Merged

tobiasschweizer reviewed Nov 20, 2017

View reviewed changes

Tobias Schweizer added 4 commits November 20, 2017 11:26

tests (KnarQL): get pages of a book ordered by their seqnum

15c29e0

test (KnarQL R2R): add JSON-LD test data for test

06d62c4

test (KnarQL R2R): add tests to get the next OFFSET

52f48bf

tests (KnarQL R2R): add JSON-LD test data

a5b5962

Tobias Schweizer and others added 9 commits November 20, 2017 13:40

fix (KnarQL query): always inster statement for start JDN sicne it ma…

cdf1f4b

…y be used for sorting

tests (KnarQL): add JSON-LD test data

0f4f757

tests (KnarQL): add JSON-LD test data

e742e18

tests (KnarQL): add JSON-LD test data

ddfd698

tests (KnarQL): add JSON-LD test data

426cb1b

tests (KnarQL): add JSON-LD test data

b0bd842

tests (KnarQL): add JSON-LD test data

98c0007

tests (KnarQL): add JSON-LD test data

cb401cb

docs (webapi): Fix typos.

287fb63

dasch-swiss deleted a comment from tobiasschweizer Nov 20, 2017

tobiasschweizer approved these changes Nov 20, 2017

View reviewed changes

benjamingeer merged commit cc53ef8 into develop Nov 20, 2017

benjamingeer deleted the wip/ontology-updates-5 branch November 20, 2017 16:10

subotic added this to the v1.1.0 milestone Feb 12, 2018

benjamingeer mentioned this pull request Apr 23, 2018

Ben's PR history #571

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement smart IRIs #662

Implement smart IRIs #662

benjamingeer commented Nov 7, 2017 •

edited

Loading

benjamingeer commented Nov 10, 2017

tobiasschweizer commented Nov 13, 2017 •

edited

Loading

tobiasschweizer commented Nov 13, 2017

benjamingeer commented Nov 13, 2017 •

edited

Loading

tobiasschweizer commented Nov 13, 2017

benjamingeer commented Nov 13, 2017

tobiasschweizer commented Nov 13, 2017

benjamingeer commented Nov 16, 2017

tobiasschweizer commented Nov 17, 2017

benjamingeer commented Nov 17, 2017

tobiasschweizer commented Nov 20, 2017

tobiasschweizer Nov 20, 2017

tobiasschweizer Nov 20, 2017

tobiasschweizer commented Nov 20, 2017

tobiasschweizer commented Nov 20, 2017

tobiasschweizer commented Nov 20, 2017

Implement smart IRIs #662

Implement smart IRIs #662

Conversation

benjamingeer commented Nov 7, 2017 • edited Loading

benjamingeer commented Nov 10, 2017

tobiasschweizer commented Nov 13, 2017 • edited Loading

tobiasschweizer commented Nov 13, 2017

benjamingeer commented Nov 13, 2017 • edited Loading

tobiasschweizer commented Nov 13, 2017

benjamingeer commented Nov 13, 2017

tobiasschweizer commented Nov 13, 2017

benjamingeer commented Nov 16, 2017

tobiasschweizer commented Nov 17, 2017

benjamingeer commented Nov 17, 2017

tobiasschweizer commented Nov 20, 2017

tobiasschweizer Nov 20, 2017

Choose a reason for hiding this comment

tobiasschweizer Nov 20, 2017

Choose a reason for hiding this comment

tobiasschweizer commented Nov 20, 2017

tobiasschweizer commented Nov 20, 2017

tobiasschweizer commented Nov 20, 2017

benjamingeer commented Nov 7, 2017 •

edited

Loading

tobiasschweizer commented Nov 13, 2017 •

edited

Loading

benjamingeer commented Nov 13, 2017 •

edited

Loading