Skip to content

Commit

Permalink
[refactor] Readme examples & arch, fixed URLs and tests, added test-data
Browse files Browse the repository at this point in the history
  • Loading branch information
amivanoff committed Aug 22, 2024
1 parent 24159ce commit af44812
Show file tree
Hide file tree
Showing 48 changed files with 5,059 additions and 650 deletions.
3 changes: 1 addition & 2 deletions .env.test
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,7 @@
REACT_EDITOR=code
EXTEND_ESLINT=true

#REACT_APP_SERVER_RDF_URL=https://expert.agentlab.ru/rdf4j-server
REACT_APP_SERVER_RDF_URL=https://rdf4j.agentlab.ru/rdf4j-server
REACT_APP_SERVER_RDF_URL=https://rdf4j.agentlab.eu/rdf4j-server

REACT_APP_RM_REPOSITORY_PARAM='{ "Repository ID": "reqs2" }'
REACT_APP_RM_REPOSITORY_TYPE=native-rdfs-dt
2 changes: 1 addition & 1 deletion .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ jobs:
run: pnpm lint
# Did not work without local files with test data
# - name: Test
# run: yarn test --ci --coverage --maxWorkers=2
# run: pnpm test --ci --coverage --maxWorkers=2
44 changes: 22 additions & 22 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,47 +11,41 @@
"editor.defaultFormatter": "esbenp.prettier-vscode",
"prettier.endOfLine": "lf",
"cSpell.words": [
"Coll's",
"Constr's",
"Constrs",
"Descr",
"Descrs",
"Elems",
"Individ",
"Inferencer",
"Interp",
"Intrnl",
"Modifiability",
"Objs",
"Rubberband",
"SHACL",
"SPDX",
"Schema's",
"Scroller",
"Sparqled",
"Subcat",
"addprops",
"agentlab",
"aldkg",
"antd",
"antv",
"bgps",
"clss",
"Coll's",
"collapsable",
"colls",
"cond",
"conds",
"Constr's",
"Constrs",
"dcterms",
"Descr",
"Descrs",
"dryrun",
"Elems",
"ents",
"Individ",
"Inferencer",
"inferencing",
"insertdelete",
"Interp",
"Intrnl",
"jsld",
"jsonld",
"jumpover",
"ldkg",
"middlewares",
"mktp",
"Modifiability",
"notexists",
"Objs",
"openrdf",
"oslc",
"pannable",
Expand All @@ -63,19 +57,25 @@
"remotedev",
"renumerate",
"reqs",
"Rubberband",
"Schema's",
"Scroller",
"SHACL",
"sider",
"sparql",
"Sparqled",
"sparqljs",
"sparqljsldclient",
"SPDX",
"spoc",
"strends",
"strstarts",
"Subcat",
"triplestore",
"undelegate",
"unscreen",
"uuidv",
"vocabs",
"триплов",
"триплы"
"vocabs"
],
"cSpell.language": "en,ru",
"terminal.integrated.scrollback": 10000,
Expand Down
130 changes: 111 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,34 @@
# SPARQL JSON Schema Linked Data Client

It is an intelligent SPARQL Client with SHACL Shapes support. You could use JSON objects and JSON Schemas to query SPARQL Endpoint, semi-similar to GraphQL style.
It is an intelligent SPARQL Client with object-oriented reactive blackboard. You could use JSON/JS objects and JSON Schemas to query SPARQL Endpoint, semi-similar to GraphQL.

Some core features requires:
You manipulate JS objects programmatically, client generates SPARQL Queries and interacts with SPARQL Endpoint. The Client retrieves SHACL Shapes to infer available class properties and allowed value types.

- RDF4J REST API in addition to standard SPARQL 1.1 Query and SPARQL 1.1 Update.
- SHACL Shapes for rdfs classes in server RDF repository.
![SPARQL JSLD Client Architecture Overview](/docs/ArchitectureOverview.png)

## Features

- Retrieves lazily SPARQL Prefixes (namespaces) from server (requires RDF4J REST API)
- Retrieves lazily SHACL Shapes from server, converts it into JSON Schemas and UI Schemas and caches them
- Uses JS objects and JSON Schemas (converted from shapes) to generate SPARQL Select Queries and SPARQL Update queries
- Converts SPARQL Results into JS objects for consumption in apps
- Have API server RDF repository creation and deletion (requires RDF4J REST API)
- Supports bulk-load data from local files to server RDF repository (requires RDF4J REST API)
- Retrieves SPARQL Prefixes (namespaces) from a server (requires RDF4J REST API)
- Retrieves RDFS classes and SHACL Shapes from a server, converts it into JSON Schemas and caches them
- Retrieves Linked Data metadata (class contexts, names and descriptions for class and property shapes) and caches them inside JSON Schemas
- Uses object-oriented "entity query" descriptions (Collection Constraints) and JSON Schemas with additional Linked Data metadata (converted from shapes) to generate SPARQL Select and SPARQL Update queries
- Converts SPARQL Results into JS objects and puts it into a blackboard
- Handles state changes and notifies all registered handler functions about specific property change
- Have an API for RDF repository creation and deletion on a server (requires RDF4J REST API)
- Supports bulk-load data from local files to RDF repository on a server (requires RDF4J REST API)
- Supports lazy loading and "load more" incremental page-based fetching
- Could be extended with specialized collection processors via `registerMstCollSchema()`

## Prerequisites

Client requires:

- RDF4J triplestore with RDF4J REST API extensions in addition to standard SPARQL 1.1 Query and SPARQL 1.1 Update.
- RDF triplestore should have SHACL Shapes for rdfs classes and properties (for base classes as well as subclasses used by this client).

## Usage

### RDF4J Repository Creation and Data Uploading (low level, without MST)
### RDF4J Repository Creation and Data Uploading (low level client API, without MST)

```typescript
import { SparqlClientImpl, uploadFiles } from '@agentlab/sparql-jsld-client';
Expand All @@ -29,27 +39,27 @@ await client.createRepositoryAndSetCurrent({ "Repository ID": "reqs2" }, 'native
const files = [
{
file: 'vocabs/rm.ttl',
baseURI: '<http://cpgu.kbpm.ru/ns/rm/rdf#>',
baseURI: '<https://agentlab.eu/ns/rm/rdf#>',
},
{
file: 'shapes/shacl/rm/rm-shapes.ttl',
baseURI: '<http://cpgu.kbpm.ru/ns/rm/rdf#>',
file: 'shapes/rm/rm-shapes.ttl',
baseURI: '<https://agentlab.eu/ns/rm/rdf#>',
},
{
file: 'data/cpgu/sample-collection.ttl',
baseURI: '<http://cpgu.kbpm.ru/ns/rm/reqs#>',
file: 'data/sample-collection.ttl',
baseURI: '<https://agentlab.eu/ns/rm/reqs#>',
},
];
await uploadFiles(client, files, './test/data/');
await uploadFiles(client, files, './test-data/');
```
### RDF4J Client SPARQL Queries (low level, without MST)
### RDF4J Client SPARQL Queries (low level client API, without MST)
```typescript
const results = await client.sparqlSelect('SELECT * WHERE ?s ?p ?o');
```
### Reactive Collections (with MST and SPARQL Queries Generation)
### Reactive Collections (high-level client API with MST and SPARQL Queries Generation)
```typescript
import { MstRepository, rootModelInitialState } from '@agentlab/sparql-jsld-client';
Expand Down Expand Up @@ -84,6 +94,88 @@ when(
);
```
### More complex entity query example
For example, an entity query to retrieve class shapes and property shapes:
```typescript
{
entConstrs: [
{
schema: 'sh:NodeShapeShape',
conditions: {
targetClass: 'rm:Artifact',
property: '?eIri1',
},
},
{
schema: 'sh:PropertyShapeShape',
},
],
orderBy: [{ expression: 'order1', descending: false }],
},
```
From this definition the SparqlGen will produce SPARQL Query like this:
```SPARQL
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX dcterms: <http://purl.org/dc/terms/>
PREFIX sh: <http://www.w3.org/ns/shacl#>
PREFIX rm: <https://agentlab.eu/ns/rm/rdf#>
CONSTRUCT {
?eIri0 rdf:type sh:NodeShape.
?eIri0 sh:property ?eIri1.
?eIri0 sh:targetClass rm:Artifact.
?eIri0 dcterms:title ?title0.
?eIri0 dcterms:description ?description0.
?eIri0 rm:inCreationMenu ?inCreationMenu0.
?eIri0 rm:defaultIndividNs ?defaultIndividNs0.
?eIri0 rm:defaultFormat ?defaultFormat0.
?eIri0 rm:iconReference ?iconReference0.

?eIri1 rdf:type sh:PropertyShape.
?eIri1 sh:name ?name1.
?eIri1 sh:description ?description1.
?eIri1 sh:path ?path1.
?eIri1 sh:order ?order1.
?eIri1 sh:datatype ?datatype1.
?eIri1 sh:minCount ?minCount1.
?eIri1 sh:maxCount ?maxCount1.
?eIri1 sh:class ?class1.
?eIri1 sh:nodeKind ?nodeKind1.
?eIri1 rm:shapeModifiability ?shapeModifiability1.
?eIri1 rm:valueModifiability ?valueModifiability1.
}
WHERE {
?eIri0 rdf:type sh:NodeShape;
sh:targetClass rm:Artifact.
OPTIONAL { ?eIri0 dcterms:title ?title0. }
OPTIONAL { ?eIri0 dcterms:description ?description0. }
OPTIONAL { ?eIri0 rm:inCreationMenu ?inCreationMenu0. }
OPTIONAL { ?eIri0 rm:defaultIndividNs ?defaultIndividNs0. }
OPTIONAL { ?eIri0 rm:defaultFormat ?defaultFormat0. }
OPTIONAL { ?eIri0 rm:iconReference ?iconReference0. }

OPTIONAL {
?eIri1 rdf:type sh:PropertyShape.
?eIri0 sh:property ?eIri1.
?eIri1 sh:path ?path1.
OPTIONAL { ?eIri1 sh:name ?name1. }
OPTIONAL { ?eIri1 sh:description ?description1. }
OPTIONAL { ?eIri1 sh:order ?order1. }
OPTIONAL { ?eIri1 sh:datatype ?datatype1. }
OPTIONAL { ?eIri1 sh:minCount ?minCount1. }
OPTIONAL { ?eIri1 sh:maxCount ?maxCount1. }
OPTIONAL { ?eIri1 sh:class ?class1. }
OPTIONAL { ?eIri1 sh:nodeKind ?nodeKind1. }
OPTIONAL { ?eIri1 rm:shapeModifiability ?shapeModifiability1. }
OPTIONAL { ?eIri1 rm:valueModifiability ?valueModifiability1. }
}
}
ORDER BY (?order1)
```
## Local Development
Use pnpm.
Expand Down
Binary file added docs/ArchitectureOverview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
27 changes: 17 additions & 10 deletions docs/Testcases Overall Structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,36 +7,43 @@ Suitable for quick test-based diagnostics in case you have many errors from seve

The list blow is sorted from simple unit tests to more and more complex integration tests (i.e. complexity increases):

## Without SPARQL Generation
## MST Model CRUD Operations and Rx (without SPARQL Generation and client-server I/O)
```bash
pnpm test MstModel.spec.ts
```

## RDF4J Client data uploading & low level textual SPARQL queries (without SPARQL Generation)
SparqlClientUpload.spec.ts -- REST rdf file upload
```bash
yarn test SparqlClientUpload.spec.ts
pnpm test SparqlClientUpload.spec.ts
```
SparqlClient.spec.ts -- namespaces and basic SPARQL Select with RDFS and DT reasoners
```bash
yarn test SparqlClient.spec.ts
pnpm test SparqlClient.spec.ts
```

## With SPARQL Generation
SparqlGen.spec.ts -- without: mobx, client-server I/O
## SPARQL Generation (without MST and client-server I/O)
SparqlGen.spec.ts -- without: MST, client-server I/O
```bash
yarn test SparqlGen.spec.ts
pnpm test SparqlGen.spec.ts
```

## SPARQL Generation with MST and client-server I/O
ArtifactShapeSchema.spec.ts -- initial shapes, server shapes (simple parentless and with inheritance)
```bash
yarn test ArtifactShapeSchema.spec.ts
pnpm test ArtifactShapeSchema.spec.ts
```
SimpleRetrieve.spec.ts -- upload all example files and all kind of selects with different conditions
```bash
yarn test SimpleRetrieve.spec.ts
pnpm test SimpleRetrieve.spec.ts
```
Artifact.spec.ts -- Entity deletion and creation
```bash
yarn test Artifact.spec.ts
pnpm test Artifact.spec.ts
```
ArtifactsInModule.spec.ts
```bash
yarn test ArtifactsInModule.spec.ts
pnpm test ArtifactsInModule.spec.ts
```

In case of errors we recommend to check and fix errors in this order.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@agentlab/sparql-jsld-client",
"version": "5.0.11",
"version": "5.0.12",
"description": "SPARQL JSON Schema Linked Data Client",
"license": "GPL-3.0",
"author": "Aleksei Ivanov <amivanoff@gmail.com>",
Expand Down
6 changes: 3 additions & 3 deletions src/SparqlClientImpl.ts
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,7 @@ export class SparqlClientImpl implements SparqlClient {
}

/**
* Удаляет все триплы в графе с заданным graph
* Delete all triples in a specified graph
* @param graph
*/
async clearGraph(graph = 'null'): Promise<any> {
Expand Down Expand Up @@ -339,8 +339,8 @@ export class SparqlClientImpl implements SparqlClient {
if (repType === 'virtuoso') {
repParam = {
...repParam,
'Default graph name': `http://cpgu.kbpm.ru/ns/rm/${repId}`,
'Inference RuleSet name': `http://cpgu.kbpm.ru/ns/rm/${repId}`,
'Default graph name': `https://agentlab.eu/ns/rm/${repId}`,
'Inference RuleSet name': `https://agentlab.eu/ns/rm/${repId}`,
"Use defGraph with SPARQL queries, if query default graph wasn't set": true,
};
}
Expand Down
6 changes: 3 additions & 3 deletions src/SparqlGen.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ export const factory: DataFactory = new DataFactory();

export function isUrl(str: string): boolean {
if (_isUrl(str) === true) return true;
if (str.startsWith('cpgu:///')) return true;
if (str.startsWith('clss:///')) return true;
return false;
}

Expand Down Expand Up @@ -835,8 +835,8 @@ export function getDataTriples(entConstr: EntConstrInternal): any[] {
return triples;
}

//TODO: упростить сигнатуру функции!
//Гармонизировать сигнатуру с другими функциями (getSimpleFilter, getDataTriples)
//TODO: Simplify function signature!
//Harmonize signature with other functions (getSimpleFilter, getDataTriples)
export function getExtendedFilter(
entConstr: EntConstrInternal,
filterKey: string,
Expand Down
4 changes: 2 additions & 2 deletions src/models/Model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@ export const rootModelInitialState: any = {
login: 'guest@example.com',
name: 'Guest',
},
processArea: 'projects:gishbbProject',
processArea: 'projects:defaultProject',
ns: {
current: {
rdf: 'http://www.w3.org/1999/02/22-rdf-syntax-ns#',
rdfs: 'http://www.w3.org/2000/01/rdf-schema#',
xsd: 'http://www.w3.org/2001/XMLSchema#',
sesame: 'http://www.openrdf.org/schema/sesame#', //rdf4j DirectType Inferencer sesame:directSubClassOf
rdf4j: 'http://rdf4j.org/schema/rdf4j#', //rdf4j Default Graph rdf4j:nil, rdf4j:SHACLShapeGraph
aldkg: 'https://agentlab.eu/ldkg/onto#',
aldkg: 'https://agentlab.eu/ns/ldkg#',
},
},
schemas: {
Expand Down
Loading

0 comments on commit af44812

Please sign in to comment.