New configuration file format to replace config.inc

[osma]

At the moment we use a PHP file called global.inc for global configuration settings. Most of them are set using define() statements. But this is not a very flexible format and causes difficulties in unit testing, because while the config.inc is parsed within the GlobalConfig constructor, its effects are global so there cannot be two configurations active within the same PHP environment.

We could introduce a new configuration file format for Skosmos 2.0 since it's a major version bump. There should be some migration mechanism, for example a script that converts an old configuration file to the new format.

Reasonable options for a new configuration file format:

• PHP .inc file that defines an array with settings
• YAML

Not so good options:

• JSON: doesn't support comments
• INI style: doesn't support arrays
• XML: nobody wants to use XML...

[kinow]

I'm used to YAML for several SpringBoot projects, Ansible, some AWS stuff, and even PHP Symfony projects. Happy to use .inc and JSON too.

But what I like the most about YAML are:

i) comments are easy
ii) it's not so hard to create some way to have files that include or inherit from other files. This way you can have something like global.yml with all the options, and a user.yml which overrides/adds options

There're also great libraries for YAML validation such as SnakeYAML and also for search/query like JMESPath (learned it after debugging an issue in aws-cli).

Just my 0.02 cents

[osma]

Thanks @kinow for your comments! Now that you mentioned comments, I think that's one good reason to prefer YAML over JSON. So the contenders are PHP .inc and YAML.

[osma]

@kinow Any pointers to a YAML based configuration file with an include mechanism?

[kinow]

I'm more familiar with Laravel, but lately have been learning Symfony at work. So the first one that came to my mind was Symfony YAML configuration: https://symfony.com/doc/current/service_container/import.html

Look at the import key used in their YAML file. I believe the logic behind it is done here https://github.com/symfony/symfony/blob/1fffb8554cf3a3cd1d800e66afbd00e63282123a/src/Symfony/Component/Routing/Loader/YamlFileLoader.php#L147

But shouldn't be very complicated. Here's another example, now a module just for configuration (never used it tho): https://github.com/awurth/PHP-Config

And this package https://packagist.org/packages/pragmarx/yaml has some interesting features that could be handy.

Alas I couldn't find a good library for PHP, generic enough so that it would be easy to integrate into Skosmos. I believe you will have to grab bits and pieces from the PHP Yaml module, plus examples/snippets from some libraries like these with compatible license, and create a Skosmos configuration parser :-/

Hope that helps

[nichtich]

How about config.ttl? For instance

<> skosmos:SERVICE_NAME "Skosmos" .

I'm a big fan of YAML but Skosmos is configured in Turtle syntax anyway so why add another syntax?

[kinow]

I'm OK with Turtle too.

but Skosmos is configured in Turtle syntax anyway so why add another syntax?

I believe this is partly right? You have the vocabulary config... but also a config.inc? Which points to the vocab config file location if I remember well...

I think this issue is to replace that other config file. But making everything Turtle could be an intetesting option. I had had to compare JSON, INI, XML, YAML, and other formats. But never compared any of these with Turtle. So definitely interested in what others think about it.

[osma]

Great idea @nichtich ! As @kinow said the original idea was to replace the current config.inc. But instead of directly replacing it with another format, we could merge the current config.inc and vocabularies.ttl configuration files into a single Turtle file that defines both the general settings and the vocabularies - a little bit like how Fuseki (especially version 1) is configured.

Then the configuration could also be kept in a triple store, as somebody requested a while ago.

Another idea would be to support owl:imports statements in the file that could be used as an include mechanism, to make it possible to split the configuration into several files or even just URIs/URLs that provide access to the configuration (though some caching mechanism would be necessary in that case, as the configuration is read on every request).

[kinow]

I think we still need to define how the configuration will be modelled with turtle then. Maybe something like

@prefix void: <http://rdfs.org/ns/void#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix dc: <http://purl.org/dc/terms/> .
@prefix skosmos: <http://purl.org/net/skosmos#> .
@prefix : <#> .

skosmos:Configuration a rdfs:Class .

:config a skosmos:Configuration ;
	:values
	    [ :name "VOCABULARIES_FILE" ; :value "vocabularies.ttl" ] ,
		[ :name "DEFAULT_ENDPOINT" ; :value "http://localhost:3030/ds/sparql" ] ,
		[ :name "DEFAULT_SPARQL_DIALECT" ; :value "Generic" ] ,
		[ :name "SPARQL_COLLATION_ENABLED" ; :value false ] ,
		[ :name "SPARQL_TIMEOUT" ; :value 20 ] ,
		[ :name "HTTP_TIMEOUT" ; :value 5 ] ,
		[ :name "SERVICE_NAME" ; :value "Skosmos" ] ,
		[ :name "BASE_HREF" ; :value "http://localhost/Skosmos/" ] ,
		[ :name "LANGUAGES" ;
			:values 
				[ :name "fi" ; :value "fi_FI.utf8" ] ,
				[ :name "sv" ; :value "sv_SE.utf8" ] ,
				[ :name "en" ; :value "en_GB.utf8" ]
			;
		] ,
		[ :name "SEARCH_RESULTS_SIZE" ; :value 20 ] ,
		[ :name "DEFAULT_TRANSITIVE_LIMIT" ; :value 1000 ] ,
		[ :name "LOG_CAUGHT_EXCEPTIONS" ; :value true ] ,
		[ :name "LOG_BROWSER_CONSOLE" ; :value false ] ,
		[ :name "LOG_FILE_NAME" ; :value "" ] ,
		[ :name "TEMPLATE_CACHE" ; :value "/tmp/skosmos-template-cache" ] ,
		[ :name "CUSTOM_CSS" ; :value "resource/css/stylesheet.css" ] ,
		[ :name "FEEDBACK_ADDRESS" ; :value "" ] ,
		[ :name "FEEDBACK_SENDER" ; :value "" ] ,
		[ :name "FEEDBACK_ENVELOPE_SENDER" ; :value "" ] ,
		[ :name "UI_LANGUAGE_DROPDOWN" ; :value false ] ,
		[ :name "UI_HONEYPOT_ENABLED" ; :value true ] ,
		[ :name "UI_HONEYPOT_TIME" ; :value 5 ]
.

Or we could have something like

skosmos:Configuration a rdfs:Class ;
	rdfs:label "A configuration source." ;
	rdfs:comment "All instances will be used for configuration. You are allowed to create multiple instances per logical sections (e.g. UI, System, etc)."
.

skosmos:VocabularyConfiguration a rdfs:Class ;
	rdfs:subClassOf skosmos:Configuration ;
	rdfs:label "The vocabulary configuration location." ;
	rdfs:comment "This file is loaded separately, and in a different manner." 
.

:vocabularyConfiguration a skosmos:VocabularyConfiguration ;
	skosmos:ConfigurationLocation <file://vocabularies.ttl>
.

:skosmosConfig a skosmos:Configuration ;
	skosmos:

...

And try to have subClasses as sections. Which I think would overcomplicate, and possibly reduce performance as PHP would have to re-evaluate it constantly, or cache it.

Not too fluent nor used to model many classes like these, so let me know if it is way off of a possible solution :-) no hard feelings.

Cheers
Bruno

ps: loading the first example, the following query would expand most blank nodes

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX skosmos: <http://purl.org/net/skosmos#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX owl: <http://www.w3.org/2002/07/owl#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX : <http://localhost:3030/ds2/data#> 

CONSTRUCT { ?values ?x ?y . ?values ?key ?value }
WHERE {
  ?x a skosmos:Configuration .
  ?x :values ?values .
  OPTIONAL {
	?values ?key ?value 
  }
}

The languages were actually a variable in the configuration file, holding a PHP array. We can try to expand it too, or just leave it to be done programmatically with EasyRDF...

[kinow]

Oh, and not too sure about importing vocabularies.ttl. Easier to see how it would work once the first part which loads the new ttl configuration is done, and then we check whether we are able & it makes sense to load another external file (and perhaps make it so it's generic and we can load others too?), or if maybe we should leave the parsing & the VocabularyConfig object as is.

[osma]

@kinow That's a good start! But I think it can be simplified.

First of all let's think of the whole Skosmos configuration (global + vocabularies) as a single RDF graph. We can later think about using owl:imports or some other mechanism to construct that graph from multiple files (or URLs) if necessary, but let's put that aside for a moment and assume it's just a single file with a well-known file name, say config.ttl. This means that the VOCABULARIES_FILE setting can be dropped.

For the other settings, most of them are just a single value, so the blank node with :name and :value indirection is not needed. Instead we could coin new properties for each setting, e.g. skosmos:defaultEndpoint and skosmos:serviceName.

There could be two RDFS/OWL classes, let's say skosmos:Configuration and skosmos:Vocabulary. The latter one already exists (note that there is currently no ontology that actually defines the skosmos: namespace, just conventions and documentation in the wiki).

[kinow]

@osma

So something more like this?

@prefix void: <http://rdfs.org/ns/void#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix owl: <http://www.w3.org/2002/07/owl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix dc: <http://purl.org/dc/terms/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix wv: <http://vocab.org/waiver/terms/norms> .        
@prefix sd: <http://www.w3.org/ns/sparql-service-description#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix skosmos: <http://purl.org/net/skosmos#> .
@prefix isothes: <http://purl.org/iso25964/skos-thes#> .
@prefix mdrtype: <http://publications.europa.eu/resource/authority/dataset-type/> .
@prefix : <#> .

# Skosmos main configuration

:fi a rdfs:Class ;
	:name "fi" ;
	:value "fi_FI.utf8"
.
:sv a rdfs:Class ;
	:name "sv" ;
	:value "sv_SE.utf8"
.
:en a rdfs:Class ;
	:name "en" ;
	:value "en_GB.utf8"
. 

:config a skosmos:Configuration ;
	skosmos:sparqlEndpoint "http://localhost:3030/ds/sparql" ;
	skosmos:sparqlDialect "Generic" ;
	skosmos:sparqlCollationEnabled true ;
	skosmos:sparqlTimeout 20 ;
	skosmos:httpTimeout 20 ;
	skosmos:serviceName "Skosmos" ;
	skosmos:baseHref "http://localhost/Skosmos/" ;
	skosmos:languages ( :fi :sv :en ) ;
	skosmos:searchResultsSize 20 ;
	skosmos:transitiveLimit 1000 ;
	skosmos:logCaughtExceptions true ;
	skosmos:logBrowserConsole false ;
	skosmos:logFileName "" ;
	skosmos:templateCache "/tmp/skosmos-template-cache" ;
	skosmos:customCss "resource/css/stylesheet.css" ;
	skosmos:feedbackAddress "" ;
	skosmos:feedbackSender "" ;
	skosmos:feedbackEnvelopeSender "" ;
	skosmos:uiLanguageDropdown false ;
	skosmos:uiHoneypotEnabled true ;
	skosmos:uiHoneypotTime 5
.

# Skosmos vocabularies

:ysa a skosmos:Vocabulary, void:Dataset ;
	dc:title "YSA - Yleinen suomalainen asiasanasto"@fi,
		"YSA - Allmän tesaurus på finska"@sv,
		"YSA - General Finnish thesaurus"@en ;
	dc:subject :cat_general ;
        dc:type mdrtype:THESAURUS ;
	void:uriSpace "http://www.yso.fi/onto/ysa/";
	skosmos:groupClass skos:Collection;
	skosmos:language "fi";
	skosmos:shortName "YSA";
  	skosmos:feedbackRecipient "vesa-posti@helsinki.fi" ;
	skosmos:showChangeList "true" ;
	void:dataDump <http://api.finto.fi/download/ysa/ysa-skos.ttl> ;
	void:sparqlEndpoint <http://api.dev.finto.fi/sparql> ;
	skosmos:sparqlGraph <http://www.yso.fi/onto/ysa/>
.

:categories a skos:ConceptScheme;
	skos:prefLabel "Skosmos Vocabulary Categories"@en
.

:cat_general a skos:Concept ;
	skos:topConceptOf :categories ;
	skos:inScheme :categories ;
	skos:prefLabel "Yleiskäsitteet"@fi, 
		"Allmänna begrepp"@sv,
		"General concepts"@en
.

mdrtype:THESAURUS a skos:Concept ;
	skos:prefLabel "Тезаурус"@bg, "Tezaurus"@cs, "Tesaurus"@da, "Thesaurus"@de, "Θησαυρός"@el, "Thesaurus"@en, "Tesaurus"@et, "Tesaurus"@fi, "Thésaurus"@fr, "Pojmovnik"@hr, "Tezaurusz"@hu, "Tesauro"@it, "Tēzaurs"@lv, "Tezauras"@lt, "Teżawru"@mt, "Thesaurus"@nl, "Tesaurus"@no, "Tezaurus"@pl, "Tesauro"@pt, "Tezaur"@ro, "Synonymický slovník"@sk, "Tezaver"@sl, "Tesauro"@es, "Tesaurus"@sv
.

mdrtype:ONTOLOGY a skos:Concept ;
	skos:prefLabel "Онтология"@bg, "Ontologie"@cs, "Ontologi"@da, "Ontologie"@de, "Οντολογία"@el, "Ontology"@en, "Ontoloogia"@et, "Ontologia"@fi, "Ontologie"@fr, "Ontologija"@hr, "Ontológia"@hu, "Ontologia"@it, "Ontoloģija"@lv, "Ontologija"@lt, "Ontoloġija"@mt, "Ontologie"@nl, "Ontologi"@no, "Struktura pojęciowa"@pl, "Ontologia"@pt, "Ontologie"@ro, "Ontológia"@sk, "Ontologija"@sl, "Ontología"@es, "Ontologi"@sv
.

[osma]

@kinow Exactly!

For the languages the order is significant so they should probably be expressed as a list. Other than that, looks good! And there are defaults for almost everything so real configurations don't need to be that verbose.

@nichtich Is this what you had in mind?

[kinow]

@osma updated the query... not so familiar with lists/collections in RDF/Turtle, so hopefully this version would return the languages in order?

Will wait a bit more to see what @nichtich thinks about it before starting to code. 🎉

By the way, in the beginning I wasn't sure how it would look... but now after looking at this version, and comparing with what I had in mind for YAML... I think I prefer this version :-)

[osma]

@kinow Actually EasyRdf doesn't contain a SPARQL engine, so if the configuration is stored in a file that's parsed by EasyRdf, the graph has to be navigated using the EasyRdf API, not SPARQL. VocabularyConfig already does that for vocabularies defined in vocabularies.ttl, the code in GlobalConfig that currently looks at PHP constants (except for the languages which are in a global variable) would need to be replaced with something similar that uses EasyRdf to read parts of the graph.

[kinow]

@osma that's what I had in mind too. My initial idea is to modify GlobalConfig.php, keeping all methods, but changing the constructor to handle a ... Resource? Still need to study the EasyRDF API and understand its objects relations to RDF.

Keeping the GlobalConfig object and its methods, I think the impact in the rest of the code would be minimum. Still need to consider whether the GlobalConfig object will parse/load the vocabularies, or if that'll be kept in the Model object.

Horrible weather here, and Queen's Birthday holiday next Monday. Might spend most of the time this coming long weekend having fun with Skosmos, RDF, Jena, EasyRDF, etc 🎉

[kinow]

And now we have some working code to review & discuss ! 🎉

#769

[osma]

#769 merged and mentioned this on the Upgrading page in the wiki. All done until 2.0 release time.