how to extend the capabilities of a server?

[jvandegriff]

As people build implementations, there is renewed interested in how to add non-standard features to the server.

Telecon notes from 2017-05-16 have a discussion about this:
https://github.com/hapi-server/data-specification/wiki/telecon-notes

Everyone pretty much agrees that the HAPI endpoints should be kept strict (no extensions - any extra keywords is an error).

We would need a way for people to advertise extensions, presumably in the capabilities endpoints.

{
  "HAPI": "1.1",
  "status": {
    "code": 1200,  "message": "OK"
  },
  "outputFormats": [
    "csv", "binary", "json"
  ]
"extensions": {
    [
        {
            "endpoint": "averaged_data",   (name of endpoint)
            "description": "basic explanation of extension"
         },
         {
            "endpoint": "despiked_data",   (name of endpoint)
            "description": "basic explanation of extension"
          }
     ]
   }
}

The extension endpoint with no parameters IS a human readable landing page describing the parameters taken, and how to use the endpoint.

[rweigel]

There is an old wiki page that has some discussion of this: https://github.com/hapi-server/data-specification/wiki/extension-notes

[rweigel]

Here is an alternative that would not require the creation of new endpoints.

Example: A server provides time averaging. It can return the average, min, or max of parameters in non-overlapping windows with a given time width that can be 1 second, 1 minute, and 1 hour.

The request would have the form:

dataset=ID&parameters=IDs&start=START&stop=STOP&extension.window.width=PT1S&extension.window.operation=average

where the extension prefix communicates to someone who receives this link that they should not expect the URL parameter to work on any HAPI server.

Ideally, the server would also provide a schema so that a client can determine what the restrictions are on the extension parameters:

Capabilities response:

{
	"HAPI": "2.0",
	"status": { "code": 1200, "message": "OK"},
	"outputFormats": [ "csv", "binary", "json" ],
	"x_extensionSchema": {
		"window": {
			"type": "object",
			"items": {
				"required": ["width","operation"],
				"properties" {
					"width": {
						"description": "The width of the window to apply an operation."
						"type": "string"
						"enum": ["PT1S","PT1M","PT1H"]
					}
					"operation": {
						"description": "The calculation to be performed in each time window."
						"type": "string",
						"enum": ["average","max","min"]
					}
				}
			}
		}
	}
}	

This JSON schema can be used by a client who wishes to create a menu of options that can be selected for a request to the server.

[jvandegriff]

Many datasets have options, and current HAPI servers can only express these as a combinatorial explosion of datasets of slightly different names, each name representing one set of options. For basic differences like different time averages, it would be nice for clients to be able to recognize these relationships.

For the time averaging case, Autoplot can use this to request the correct resolution based on the duration of the time range requested.

The point here is that this linking of datasets of the the same data at different time resolutions could be done by offering a setting on the dataset, or by having different datasets that are linked via metadata (dataset X is the 5 minute resolution version of dataset Y, which has 1 hour resolution)

[jvandegriff]

At the 2019-07-15 telecon, it was pointed out that
a. add extensions carefully and only after significant discussion since extensions can change or undercut the original goal of a simplistic, lowest level, easy-to-implement server
b. extensions would always be optional
b. people may end up extending it anyway, so some guidelines on how to do it could be useful
d. extensions could help by allowing instrument teams or specific sub-communities to develop servers with lots of extra features, but that underneath, were fundamentally HAPI compliant (with good defaults automatically chosen for all the extra features)

As already discussed above in this issue, there are at least two kinds of (optional) extensions that could be added:

1. new endpoints
2. extra options to existing endpoints.

One approach (which involves not adding options) is to let any more complex processing (averaging, etc) be done by higher level services.

The effect of having many options when retrieving a dataset will make caching more difficult.

[jvandegriff]

related to issues #59 #70 #79 #78 #98 #100 #101 #102

All of these talk about extensions, or different REST-ful approaches that could advertise options or extensions.

[jbfaden]

"where" is another extension that comes up now and again. It would be nice to retrieve the Juno measurements only when RJ<10, for example, since this requires having the entire set available to extract the subset. (This is a good rule, where if an operator needs all the data is sent over either way, it shouldn't be a filter. For example, FFT Power shouldn't be done on the server side because we don't have reduction anyway.)