Specify extension and service registration API and model

[birkland]

• Enumerate all known extensions
• CRUD extension definitions and configuration
  • URIs exposed by extension, if applicable
  • Resources filtered by extension, if applicable - request/response to resource may be mutated by extension
  • Binding/selection criteria - which objects can the extension provide a service?
  • URI(s) for service implementing the extension
• See See SD&B POC page
• Could some of this be satisfied simply via LDP resources in the repository?  If so, much of this task may to specifying an ontology rather than an API per se.  Maintaining a list of services implementing an extension may be suited to an API, extension definitions could possibly be repository resources.

[ajs6f]

"Could some of this be satisfied simply via LDP resources in the repository?"

This. 100x this.

[acoburn]

Yes, use LDP!

[scossu]

👍 to LDP.

[scossu]

But 👎 to using the main Fedora repository to store API-X specific metadata. I think this information is better suited in a dedicated data store.

[ajs6f]

Why?

[scossu]

In my opinion, information meant to be stored, preserved and published should not be mixed with infrastructure-related information.

But actually I was not quite correct. I am not against using the same Fedora server, but if we want to use an LDP API for API-X specific metadata, at least there should be a dedicated root container in the Fedora repo that clearly separates the API-X stuff from the rest.

[ajs6f]

This is a long-running discussion in the Fedora community. Do aspects of an intellectual object like access policies or presentational form contribute so much to its long-term durability that they must be preserved alongside its most basic qualities? In any event, we don't want to prevent people from using one single repository by demanding a different kind of storage for non-transactional data. LDP is clearly the right choice here.

[scossu]

Oh, I am not against using LDP and in fact I put a +1 on it.

I think that API-X related metadata differ even from access policies in the fact that they are even more detached from the intellectual content. Therefore I am advocating for a clearly separated area to store API-X metadata, even if in practice it can (and should) be in the same repository.

[acoburn]

I agree with @ajs6f that the answer to the question of where API-X metadata is stored depends on how you define durability and the degree to which services on those objects are, themselves, durable.

In the SD&B POC I wrote, there are at least two types of metadata:

1. A definition of how a particular service binds to particular Fedora objects. I imagine this would be some sort of OWL graph, and storing this in Fedora could make a lot of sense.
2. A list of the instances of particular services (e.g. the validation service runs on http://host1.example.org/validation and http://host2.example.org/validation). This sort of metadata (as I imagined it) would be quite transitory. While one could store that in Fedora, I don't see that as a particularly good fit.

And a third that was outside the scope of my POC:

3. Metadata about the instances of API-X core, if running API-X in a clustered arrangement.

In a service-based architecture, I would expect certain things to change a lot (esp. the data for points 2 and 3), and I would think a separate database would be a better fit -- especially a clusterable, highly consistent database -- which is why I keep coming back to Zookeeper. For those things that don't change in the same way (where durability is more important -- esp. point 1), I could see how storing that in Fedora would be a good idea.

[birkland]

Just for some historical background, the approach Fedora <= 3 took with its 'disseminators' was to have everything in the repository, while at runtime relying on a database and caches that were derived from the content in the repository.

[ajs6f]

It's certainly true that we have various classes of information here, and they have various qualities of durability and deserve various levels of investment in persistence. I think the best approach is to be open about the possibilities. Some folks may want to incur the performance (and other) costs of keeping everything in the expensive-but-very-durable repository container. Others may not want to or may not be able to so do. Specifying to APIs and not specific components is the right thing to do here.

[birkland]

Just to update on where we are:

We're still trying to achieve consensus on this.

The API-X design proposal as it exists now has made a distinction between extensions and services (see the problem statement in extension definition design. In addition, extension definitions can define or reference OWL ontologies to achieve the end goal of describing the class of resources an extension binds to. In this light, API-X needs the following kinds of information:

• Descriptions of extensions
• Descriptions of services
• Ontologies
• List of service instances that happen to exist at the moment
• Other runtime information

Furthermore, the question has been raised (see #30) of whether API-X itself needs to be able to provide some durability e.g. for ontology resolution. I think the implication of this is that if API-X needs to provide durability for (insert kind of information here), then it would probably want to store content in the repository to do so.

Additionally, Amherst has been exploring the notion of having services respond to OPTIONS requests with an extension definition document.

[birkland]

LDP-based extension, service, and ontology registries are described in the design docs:

• LDP extension registry
  • Implementation is here
• LDP service registry
  • implementation is in a pull request
• Ontology registry
  • An implementation based on LDP, persisting ontologies as binary documents, is here

Furtheremore, the prospect of service instance registries based on technologies other than LDP are described in the docs, but no implementation of this exists at the moment