WIP: reorganization of owl files to streamline repo for import as submodule

[satra]

@cmaumet, @khelm, @dbkeator - please take a look

todo (before merge):

• nidm-result.owl appears to import several owl files, but the import directory contains ttl files
• should all imports be at the top level.
• reorganize nidm into a test folder
• redo the travis test script
• create an example folder (with similar organization as ontology) and generate nidm example documents

[khelm]

One question on this organization is: what happens if two different ontologies import different parts of a base ontology (and that are now in separate files)? For example, what if Results and Experiment use a set of terms from sio.owl that are non- or only partially-intersecting? Do you have an sio.owl file that contains all of the terms that are used in the union of the two? I realize that the relevant NIDM file would only have references to the terms imported for that ontology, but this seems less clean than having dedicated imports for each part of NIDM.

[satra]

@khelm - yes the idea would be a single sio import containing terms required by any sub ontology. i thought about separate imports, but i think having a consistent single file for external imports is perhaps reasonable at this point since we are likely to move to importing a single nidm.owl instead of any of the sub-nidm owl files in general.

once i hear back about why we have ttl files when the import says owl, i will combine the files into a single file.

[khelm]

We should also discuss the question of when we do the merging as Experiment, and it looks like Results, are currently in very active development and, I think at this stage, a single file it will be much harder to work on. We also have to carefully plan the merging so that the resulting graph is consistent and makes sense.

[satra]

@khelm - merges will happen at releases and through the python and javascript libraries. they won't happen continuously.

[khelm]

@satra - that sounds good, though I should still work with @cmaumet to figure out whether linking terms are needed between Experiment and Results and with your team to do the same for Workflows.

[satra]

@khelm - perhaps linking terms can go into nidm.owl

[khelm]

@satra - that makes sense to me.

[satra]

@cmaumet - can you please clarify the ttl/owl imports?

[cmaumet]

@satra: the import owl files are serialized in turtle. Is that not good?

[satra]

@cmaumet - all these imports point to an owl file extension, but only a few are owl files the rest are ttl files.

https://github.com/satra/nidm-specs/blob/2e31cebb8cf86283c5823c6c2d9b6e83d17fd0eb/ontology/result/nidm-result.owl#L23

[cmaumet]

@satra - I see! The matching between those purl URLs and the actual ttl files is done in the catalog file: https://github.com/incf-nidash/nidm-specs/blob/master/nidm/nidm-results/terms/catalog-v001.xml#L3-L11 (that is read and interpreted correctly by Protege). Does that help?

[satra]

thanks @cmaumet - how do you think we should organize? it may still be useful for protege to be able to read in that ontology folder.

[cmaumet]

@satra - if we keep the imports subfolders and the catalog files, Protege should still be able to read in the ontology files.

[satra]

i know this was marked as a WIP. but it really would like to merge this reorganization asap.

@cmaumet, @khelm - a review would be appreciated. also there is a cogpo_import.owl file which does not seem to be well formed rdf.

[cmaumet]

Hi @satra,

Here are a few suggestions:

• For simplicity, could we directly have the owl files inside "ontology" rather than in subfolders ("ontology/experiment" and "ontology/results")? The Protege catalog files could be merged and moved to "ontology" as well.
• To make each ontology file as short as possible (and therefore more easily understandable), could we avoid merging the import files and rather use a suffix to mention which ontology file use the import, e.g. iao_import_results?

The tests need to be updated to match with the new structure (currently tests are failing).

If the goal is still to streamline repo for import as submodule, there are also other files that we will have to reorganize, open questions for NIDM-Results:

• what shall we do of all the examples that are created and stored under "nidm-results/terms/examples" and then linked to in the owl files, e.g. https://github.com/incf-nidash/nidm-specs/blob/master/nidm/nidm-results/terms/nidm-results.owl#L3339?
• What do we do with the bigger examples, e.g. https://github.com/incf-nidash/nidm-specs/tree/master/nidm/nidm-results/spm/example001
• Where do we store the queries?

If we move queries and examples to other repos, how will we handle versioning, e.g. linking a particular set of examples/queries with a given version of the specs? What is your vision on this?

[khelm]

Hi @cmaumet - Could you please explain a bit more what you mean by

could we avoid merging the import files and rather use a suffix to mention which ontology file use the import

Also, if imports from the same ontology are used in different NIDM ontologies, how do we distinguish? Will we just have separate files named something like iao_nidm_exp_import and iao_nidm_exp_import? Or are you suggesting a single file that would supply each ontology?

[cmaumet]

@khelm, my suggestion was to move the import files as follows:

• nidm/nidm-experiment/imports/obi_import.ttl > ontology/imports/obi_import_experiment.ttl
• nidm/imports/obi_import.ttl (which are actually NIDM-Results imports) > ontology/imports/obi_import_results.ttl

And then in nidm-results.owl we would import obi_import_results.ttl and in nidm-experiment.owl we import obi_import_experiment.ttl.

The rationale was that if one component, e.g. NIDM-Results, needs a lot of external terms from a given ontoly (e.g. OBI), then people working on other components, e.g. NIDM-Experiment, would not see those terms. This is in an effort to reduce the complexity for someone new, trying to understand and jump in the development of one of the components. @khelm, @satra: what do you think?

[khelm]

Yes, that sounds good (and what I was trying to ask except for my cut and paste error above...)

[satra]

@cmaumet - take a look now?

[satra]

@khelm and @cmaumet - i would really like to merge this before we do too many other changes. i will rebase on master, but can you please let me know that this looks ok.

[cmaumet]

@satra - sorry I did not have time to review today... But will do soon! In the meantime, can you solve the test failure? How about answers to my comments above?

[khelm]

The cogpo_import.owl file is not actually imported into the nidm-experiment.owl at the moment. I was going to export that xml file as a turtle file when I got around to adding it in.

I'm ok with having:
/ontology
/ontology/imports
and a merged catalog file, since there shouldn't be any collisions if we follow the suggestion by @cmaumet to label the import files by which nidm section they are used. I agree that this arrangement is somewhat less complicated than having subfolders for each nidm section. So I vote +1 for the suggestion for organization by @cmaumet

I notice that in the nidm-experiment folder there are several other owl files for demographics and I don't know where these came from but they may be from old work by @nicholsn or @dbkeator . I assume that these would then be recast as imports were they to be used.