Replace interaction affordances in directory TD with a single link to its API endpoint

[benfrancis]

Fixes #172 and #178.

This is an alternative proposal for the Thing Description of a Directory which would make #158, #159 and #160 unnecessary.

The rationale is described in detail in #172 and #178, but it basically boils down to the fact that trying to describe the whole Directory Service API in a Thing Description (which are designed for describing devices, not general web services) isn't really working.

The proposal in this PR is to instead just provide a link to the top level endpoint of the Directory Service API and define the details of the API out of band in the WoT Discovery specification. The proposed link format uses the hosts link relation type defined in RFC6690 and a proposed new application/tdd+json content type.

    "links": [{
        "rel": "hosts",
        "href": "/things",
        "type": "application/tdd+json"
    }]

At the simplest level, the application/tdd+json content type could be defined as a JSON array of Thing Descriptions (application/td+json) and can be parsed as such. A Consumer implementing the full HTTP-based Directory Service API would also know how to create, update, delete and search the list of Things by implementing API defined in the prose of the specification, instead of parsing Forms in a Thing Description.

With this approach, the Thing Description of a directory is just used for discovering a directory (using the various introduction mechanisms in the WoT Discovery spec) and linking to its top level endpoint. The actual API of the directory web service would be defined in the normative text of the WoT Discovery specification, which may also provide a machine-readable OpenAPI specification as a convenience for developers.

---

Preview | Diff

[farshidtz]

The Notification and Search APIs got sacrificed in the process? If we are going for a simple link, I think it should the absolute URL of the Directory API root endpoint.

[benfrancis]

@farshidtz wrote:

The Notification and Search APIs got sacrificed in the process?

I was wondering about that too. In an HTTP-only API it might be feasible for a client to optionally ask for an event stream rather than a static JSON array from the same endpoint, using content negotiation. In other words:

Request:

GET /things
Accept: application/tdd+json

Response:

200 OK
Content-Type: application/tdd+json
[
 {...},
 {...}
]

Request:

GET /things
Accept: text/event-stream

Response:

200 OK
Content-Type: text/event-stream

event: thingCreated
data: {...}

event: thingUpdated
data: {...}

event: thingDeleted
data: {...}

This is probably a bit over-simplified as I'm not particularly familiar with how Server Sent Events work under the hood, but you get the idea.

It should be fine for JSONPath and XPath queries to use the same endpoint, by appending URI variables.

SPARQL is a bit trickier because of the fixed requirements around HTTP verbs and URI variables etc. and the potential for naming collisions, but it might be possible to further overload the same API endpoint for a SPARQL query by using a similar content negotiation approach as above. I haven't investigated that in detail, but I do note that the SPARQL specification actually recommends using HTTP content negotiation to request a particular response format.

e.g.

GET /things?query=...
Accept: application/sparql-results+json

POST /things
Content-type: application/x-www-form-urlencoded
Accept: application/sparql-results+json
...

POST /things
Content-Type: application/sparql-query
Accept: application/sparql-results+json
...

If we are going for a simple link, I think it should the absolute URL of the Directory API root endpoint.

You mean instead of specifying a base? I have no problem with that.

[farshidtz]

Not sure why we have to redesign the API to make the TD look better. A suitable API specification format should be capable of specifying a well-designed API.

If we are going for a simple link, I think it should the absolute URL of the Directory API root endpoint.

You mean instead of specifying a base? I have no problem with that.

Sorry I didn't see the base. No, I meant the link should be to the root endpoint of the API. Considering base as https://tdd.example.com, the endpoint of directory would be just /. /things is the endpoint of the Registration API only but the directory has other APIs.

Another link could point to the spec. E.g. without thinking too much about the relation types:

{
    "base": "https://tdd.example.com",
    "links": [{
        "rel": "hosts",
        "href": "/",
        "type": "application/tdd+json"
    },
    {
        "rel": "describedby",
        "href": "https://url-of-openapi-spec",
        "type": "application/vnd.oai.openapi;version=3.0"
    }],
    ...
}

I am not supporting nor am I against this change. For our use-cases, the OpenAPI spec is used to describe and consume the API. A TD for directory is still necessary to self-describe the directory or register directories inside a central one (federated directories @AndreaCimminoArriaga ), but that doesn't require specification of the APIs inside the TD and a simple link does the job.

[benfrancis]

@farshidtz wrote:

A suitable API specification format should be capable of specifying a well-designed API.

Yes, but WoT Thing Descriptions are not a general purpose API specification format for describing web services, they are designed for describing the affordances of WoT devices.

Not sure why we have to redesign the API to make the TD look better.

The purpose is not to make the TD "look better", it's to remove the API description from the TD since it turns out TDs aren't well suited to that job.

I meant the link should be to the root endpoint of the API. Considering base as https://tdd.example.com, the endpoint of directory would be just /. /things is the endpoint of the Registration API only but the directory has other APIs.

I agree that the link should ideally link to the top level endpoint of the Directory Service API, whatever the WoT Discovery specification defines that to be. A disadvantage of linking to a / endpoint with /td, /search, and /events sub-resources is that the application/tdd+json MIME type can no longer be parsed as a simple array of TDs and in fact it's not clear what fetching the resource at / would actually resolve to. That's unfortunate for simple client implementations which just want to fetch a list of TDs as they'd then need to implement at least parts of the full API specification to know where to find that list.

Maybe separate /td, /search, and /events endpoints aren't actually necessary. Simple consumers could just fetch the top level endpoint to get a list of TDs and more complex consumers could use URL variables and content negotiation for more complex features.

Another link could point to the spec. E.g. without thinking too much about the relation types

Sure, though I'm not sure what Consumers would do with that link?

I am not supporting nor am I against this change. For our use-cases, the OpenAPI spec is used to describe and consume the API. A TD for directory is still necessary to self-describe the directory or register directories inside a central one (federated directories @AndreaCimminoArriaga ), but that doesn't require specification of the APIs inside the TD and a simple link does the job.

👍

[AndreaCimminoArriaga]

First of all, I'm not against the proposal nor in favour, I'm positively not sure what is best. Nevertheless, let me add some comments:

Maybe separate /td, /search, and /events endpoints aren't actually necessary. Simple consumers could just fetch the top level endpoint to get a list of TDs and more complex consumers could use URL variables and content negotiation for more complex features.

1. This could work from the search point of view, we could handle the type of searches using arguments: xpath, jsonpath, and query (for SPARQL).

2. I kind of like this idea of not having separate /td, /search, and /event and interact with the directory through URI variables. On the one hand, I think allows to somehow simplify a bit the interactions with the directory especially with the more simple functionalities. On the other hand, I dislike the idea because I have the feeling (maybe I'm wrong) that the directory will have a too large list of URI variables and will hinder the usability of the complex interactions (like events). For these more complex interactions I have the feeling that it would make more sense to have separate endpoints (/td, /search, /envent) and less URI variables (or maybe more clustered/sorted/ordered under those endpoints). Again, not 100% sure what I like more.

SPARQL is a bit trickier because of the fixed requirements around HTTP verbs and URI variables etc. and the potential for naming collisions, but it might be possible to further overload the same API endpoint for a SPARQL query by using a similar content negotiation approach as above. I haven't investigated that in detail, but I do note that the SPARQL specification actually recommends using HTTP content negotiation to request a particular response format.

3. Here I think SPARQL could fit without problems in terms of URI, URI variables, etc. I think the example that @benfrancis show before proves how SPARQL could fit, however the GET endpoint should be GET /things?query=.... Also, as a reminder, recall that SPARQL has also CONSTRUCT and DESCRIBE queries which potentially return TDs (and thus may require process and understand the header application/tdd+json).

A TD for directory is still necessary to self-describe the directory or register directories inside a central one (federated directories @AndreaCimminoArriaga ), but that doesn't require specification of the APIs inside the TD and a simple link does the job.

4. Whatever is the result, I think directories should somehow keep track of other directories to enable the federation among directories. Now, in order to achieve this a simple link will do the job as @farshidtz mentioned.

Sure, though I'm not sure what Consumers would do with that link?

5. Several (interesting) things can be done, the simplest would be searching for other directories, the more complex would be performing a search over different directories using SPARQL federation. I understand the concern that WoT is not meant for APIs but for affordances of WoT devices. I think storing only the link could sort this out, in this way we are not actually describing the affordances but the directory is still discoverable. Otherwise, I have a suggestion: why don't we modify the entity DirectoryThing from the discovery ontology and instead, we create a Directory class that has no super-class and has some attributes like the link and maybe other small meta information that could be relevant for the discovery. In this way, we have Thing on the one hand, and separately, we have Directories which are not described as TDs.

[benfrancis]

1. I think the example that @benfrancis show before proves how SPARQL could fit, however the GET endpoint should be _GET /things?query=_

Ah, yes that was a typo sorry.

1. why don't we modify the entity DirectoryThing from the discovery ontology and instead, we create a Directory class that has no super-class and has some attributes like the link and maybe other small meta information that could be relevant for the discovery. In this way, we have Thing on the one hand, and separately, we have Directories which are not described as TDs.

For WebThings Gateway we'd ideally like to be able to expose a Thing Description for a gateway (the physical hub) which describes certain physical affordances (e.g. reboot, shutDown and startPairing) in addition to linking to the directory of things it hosts. In that case having ThingDirectory inherit from Thing would work quite well.

[mmccool]

The consensus seems (at this point) that we will NOT do this, but that a directory should return a concrete TD based on the (to-be) TM in the standard. So I propose closing this PR, but before I do would like to get confirmation on this assumption. Discussion in mtg held June 14 - consensus was to close, but to give it a week for any further comments, and if no objections in that time, to close.

[benfrancis]

No objections. I think this approach could have worked as an alternative, but the consensus seems to be to persevere with trying to describe the directory service API using forms.