Implement a DSL for API v2 extended search

[benjamingeer]

Since API v2 extended search needs to be implemented ASAP as part of #478, I'm going to try the approach I outlined here, i.e. implementing a simple DSL that the GUI can submit to the API, and that will be straightforward to translate to SPARQL. Tobias and I looked for something standard we could use, but didn't find anything that seemed suitable.

It should parse things like the following examples.

Find all the pages with sequence numbers less than or equal to 100, in books whose name contains either "Zeitglöcklein" and "Lebens" or "Itinerarius" and "peregrinarius":

B a incunabula:book .
B incunabula:pubdate ge(date("GREGORIAN:1490")) .
B incunabula:pubdate lt(date("GREGORIAN:1500")) .
P a incunabula:page .
P incunabula:isPartOf B .
P incunabula:pagenum le(int(100)) .

{
    B incunabula:title contains("Zeitglöcklein") .
    B incunabula:title contains("Lebens") .
} UNION {
    B incunabula:title contains("Itinerarius") .
    B incunabula:title contains("peregrinarius") .
}

Find a letter that has, either as its author or as its recipient, someone whose IAF identifier is either 123 or 456:

L a beol:letter .

{
    L beol:hasAuthor P1 .
} UNION {
    L beol:hasRecipient P1 .
} UNION {
    L beol:hasAuthor P2 .
} UNION {
    L beol:hasRecipient P2 .
}

P1 beol:hasIAFIdentifier "123" .
P1 beol:hasIAFIdentifier "456" .

Perhaps I can adjust this syntax a bit and get a SPARQL parser to parse it. Otherwise, I'll look into using a parser generator such as ANTLR.

[benjamingeer]

The client needs to be able to specify which data they want in the results, so we need something like a CONSTRUCT clause as well as a WHERE clause. So actually we can make this work rather like a SPARQL endpoint. The client can actually submit SPARQL, but the submitted SPARQL will be restricted:

• It will be allowed to use the simplified API ontologies, but not the internal ones or the ones that use value objects.
• To prevent excessively expensive queries, we'll only support relatively simple SPARQL syntax, e.g. without subqueries, OPTIONAL, or nested blocks.

The API server can parse this SPARQL using an existing SPARQL parser, validate it, and generate a real SPARQL query from it. The conversion will involve:

• Transforming patterns based on API ontologies into patterns based on internal ontologies.
• Adding additional statements for things like permissions.

So for example, if the client submits something like this:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX incunabula: <http://api.knora.org/ontology/incunabula/simple/v2#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> .

CONSTRUCT {
    ?book a ?bookType .
    ?book rdfs:label ?bookLabel .
    ?page a ?pageType .
    ?page rdfs:label ?pageLabel .
    ?page incunabula:isPartOf ?book .
} WHERE {
    ?book a incunabula:book .
    ?book rdfs:label ?bookLabel .
    ?book incunabula:publisher "Lienhart Ysenhut"^^xsd:string .
    ?book incunabula:pubdate ?pubdate .
    FILTER(?pubdate < "GREGORIAN:1500")
    ?page a incunabula:page .
    ?page rdfs:label ?pageLabel .
    ?page incunabula:partOf ?book .
    ?page incunabula:pagenum ?pagenum .
    ?page incunabula:seqnum ?seqnum .
    FILTER(?seqnum < 20)

    {
        ?page incunabula:pagenum "a7r"^^xsd:string .
    } UNION {
        ?page incunabula:pagenum "a8r"^^xsd:string .     
    }
}

This will be converted to something like this:

PREFIX knora-base: <http://www.knora.org/ontology/knora-base#>
PREFIX incunabula: <http://www.knora.org/ontology/incunabula#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> .

CONSTRUCT {
    ?book a ?bookType .
    ?book rdfs:label ?bookLabel .
    ?page a ?pageType .
    ?page rdfs:label ?pageLabel .
    ?page incunabula:isPartOf ?book .
} WHERE {
    ?book a incunabula:book .

    GRAPH <http://www.ontotext.com/explicit> {
        ?book a ?bookType .
    }

    ?book rdfs:label ?bookLabel .
    ?book incunabula:publisher ?publisher .
    ?publisher knora-base:valueHasString "Lienhart Ysenhut"^^xsd:string .
    ?book incunabula:pubdate ?pubdate .
    ?pubdate knora-base:valueHasStartJDN "2268924"^^xsd:integer .
    
    ?page a incunabula:page .

    GRAPH <http://www.ontotext.com/explicit> {
        ?page a ?pageType .
    }
    
    ?page rdfs:label ?pageLabel .
    ?page incunabula:partOf ?book .
    ?page incunabula:pagenum ?pagenum .
    ?page incunabula:seqnum ?seqnum .
    ?seqnum knora-base:valueHasInteger ?seqnumValue .
    FILTER(?seqnumValue < 20)

    {
        ?pagenum knora-base:valueHasString "a7r"^^xsd:string .
    } UNION {
        ?pagenum knora-base:valueHasString "a8r"^^xsd:string .        
    }
}

[benjamingeer]

In #478 there's now a utility called SearchParserV2 that parses simple CONSTRUCT queries.

[tobiasschweizer]

This is a simple sample query which returns the letters exchanged between two persons:

Please note that internal Iris are used (they still need to be changed to external Iris).

PREFIX beol: <http://www.knora.org/ontology/beol#>

CONSTRUCT {
    ?letter ?linkingProp1  ?person1 .
    ?letter ?linkingProp2  ?person2 .
} WHERE {
    ?letter a beol:letter .
    #?letter knora-api:isMainResource true .

    # Scheuchzer, Johann Jacob 1672-1733
    BIND(<http://rdfh.ch/beol/oU8fMNDJQ9SGblfBl5JamA> AS ?person1)
    
    ?letter ?linkingProp1  ?person1 .
    FILTER(?linkingProp1 = beol:hasAuthor || ?linkingProp1 = beol:hasRecipient )

    # Hermann, Jacob 1678-1733
    BIND(<http://rdfh.ch/beol/6edJwtTSR8yjAWnYmt6AtA> AS ?person2)
    
    ?letter ?linkingProp2 ?person2 .
    FILTER(?linkingProp2 = beol:hasAuthor || ?linkingProp2 = beol:hasRecipient )        
    
}

In Knora, this has to look like this:

PREFIX knora-base: <http://www.knora.org/ontology/knora-base#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX beol: <http://www.knora.org/ontology/beol#>

CONSTRUCT {

    # include this inferred information in the results, needed to identify resources
    ?resource a knora-base:Resource ;
        knora-base:isMainResource true ;
        rdfs:label ?label ;
        knora-base:attachedToUser ?resourceCreator ;
        knora-base:hasPermissions ?resourcePermissions ;
        knora-base:attachedToProject ?resourceProject ;
        a ?resourceType .

    ?resource knora-base:hasLinkTo ?person1 ;
        ?linkingProp1 ?person1 .

    ?resource ?linkValueProp1 ?linkValue1 .

    ?resource knora-base:hasValue ?linkValue1 .

    ?linkValue1 a ?valueObjectType1 ;
                     ?valueObjectProperty1 ?valueObjectValue1 .

    ?person1 a knora-base:Resource ;
        rdfs:label ?referredResourceLabel1 ;
        a ?referredResourceType1 ;
        knora-base:attachedToUser ?referredResourceCreator1 ;
        knora-base:hasPermissions ?referredResourcePermissions1 ;
        knora-base:attachedToProject ?referredResourceProject1 .

    ?resource knora-base:hasLinkTo ?person2 ;
        ?linkingProp2 ?person2 .

    ?resource ?linkValueProp2 ?linkValue2 .

    ?resource knora-base:hasValue ?linkValue2 .

    ?linkValue2 a ?valueObjectType2 ;
                     ?valueObjectProperty2 ?valueObjectValue2 .

    ?person2 a knora-base:Resource ;
        rdfs:label ?referredResourceLabel2 ;
        a ?referredResourceType2 ;
        knora-base:attachedToUser ?referredResourceCreator2 ;
        knora-base:hasPermissions ?referredResourcePermissions2 ;
        knora-base:attachedToProject ?referredResourceProject2 .

} WHERE {

    ?resource a knora-base:Resource ;
        knora-base:isDeleted false ;
        rdfs:label ?label ;
        knora-base:attachedToUser ?resourceCreator ;
        knora-base:hasPermissions ?resourcePermissions ;
        knora-base:attachedToProject ?resourceProject .

    # user provided
    ?resource a beol:letter .

    GRAPH <http://www.ontotext.com/explicit> {
        ?resource a ?resourceType .
    }

    # Scheuchzer, Johann Jacob 1672-1733
    BIND(<http://rdfh.ch/beol/oU8fMNDJQ9SGblfBl5JamA> AS ?person1)

    ?resource knora-base:hasLinkTo ?person1 .

    ?resource knora-base:hasValue ?linkValue1 .
    ?linkValue1 a knora-base:LinkValue ;
        rdf:subject ?resource ;
        rdf:object ?person1 .


    GRAPH <http://www.ontotext.com/explicit> {
        ?resource ?linkValueProp1 ?linkValue1 .

        ?linkValue1 a ?valueObjectType1 ;
                     ?valueObjectProperty1 ?valueObjectValue1 .
    }


    ?person1 a knora-base:Resource ;
        knora-base:isDeleted false .

    GRAPH <http://www.ontotext.com/explicit> {
        # user provided
        ?resource ?linkingProp1 ?person1 .
        FILTER(?linkingProp1 = beol:hasAuthor || ?linkingProp1 = beol:hasRecipient)

        ?person1 rdfs:label ?referredResourceLabel1 ;
            a ?referredResourceType1 ;
            knora-base:attachedToUser ?referredResourceCreator1 ;
            knora-base:hasPermissions ?referredResourcePermissions1 ;
            knora-base:attachedToProject ?referredResourceProject1 .
    }

    # Hermann, Jacob 1678-1733
    BIND(<http://rdfh.ch/beol/6edJwtTSR8yjAWnYmt6AtA> AS ?person2)

    ?resource knora-base:hasLinkTo ?person2 .

    ?resource knora-base:hasValue ?linkValue2 .
    ?linkValue2 a knora-base:LinkValue ;
        rdf:subject ?resource ;
        rdf:object ?person2 .


    GRAPH <http://www.ontotext.com/explicit> {
        ?resource ?linkValueProp2 ?linkValue2 .

        ?linkValue2 a ?valueObjectType2 ;
                     ?valueObjectProperty2 ?valueObjectValue2 .
    }


    ?person2 a knora-base:Resource ;
        knora-base:isDeleted false .

    GRAPH <http://www.ontotext.com/explicit> {
        # user provided
        ?resource ?linkingProp2 ?person2 .
        FILTER(?linkingProp2 = beol:hasAuthor || ?linkingProp2 = beol:hasRecipient)

        ?person2 rdfs:label ?referredResourceLabel2 ;
            a ?referredResourceType2 ;
            knora-base:attachedToUser ?referredResourceCreator2 ;
            knora-base:hasPermissions ?referredResourcePermissions2 ;
            knora-base:attachedToProject ?referredResourceProject2 .
    }

}

[tobiasschweizer]

@benjamingeer Do you think we could allow for the use of Boolean operators in the FILTER and Iris (non literals)?

Such as: FILTER(?linkingProp = beol:hasAuthor || ?linkingProp = beol:hasRecipient)

I want to get all the letters that refer to a person with one of two possible properties.

[benjamingeer]

Sure, I'll do this on Monday.

[tobiasschweizer]

Great, thx.!

By the way: I can run the query above (the longer one) and display the results with our standard v2 scala methods as JSON-LD :-)