Improve server catalog access

[jeandet]

Hello,

I would like to use hapi-client in our project SciQLop/spwc to add more servers and maybe deprecate our direct REST API usage.
SPWC is for us an abstraction to access remote data with efficient multi layer caching system, it should also provide all the information to build a tree for each server like here.
I'm not sure if this is a client or server limitation or I missed something, but I see no way to generate such tree. First the fact that we need to do multiple requests per server to list datasets/parameters/parameters_desc makes it quite inefficient. Then it seems to miss information such as spacecraft/observatory name.

That would be nice to have the same level of information as we get here for AMDA:
http://amda.irap.omp.eu/AMDA//data/WSRESULT/obsdatatree_impex_20201118_AmdaLocalDataBaseParameters.xml

[rweigel]

This is a use-case that has been discussed before. My opinion was that there should be an automated process that generates the tree daily. Otherwise, many users will be writing code that generate the tree, which is inefficient as you mention.

Jeremy also needed this tree for building a menu similar to the one you linked to and has an automated process that walks all servers and pushes the results to https://github.com/hapi-server/servers/tree/master/index

So to generate the tree, you need to read https://github.com/hapi-server/servers/blob/master/index/servers.json and then read each of the associated URLs.

I'm of the opinion that there should be a single file that has everything so that you don't need to do 9 or so reads to build the tree. We have not yet come to a conclusion on this, but I think that your question will force us to finalize something and ideally you would provide input.

Regarding the spacecraft/observatory name, HAPI does not address this. HAPI has servers, datasets, and parameters. The datasets can be a spacecraft, a stand-alone instrument, a time series that is based on output from a simulation at many locations in space, or a list of 0s and 1s versus time to indicate whether an event occurred. Ideally, there would be a way to map from a HAPI ID to a spacecraft/observatory name. This is part of a larger and separate discussion, however. I'll post this as a separate issue.

[jbfaden]

This master list is updating automatically now, every day at 14:26 Central USA time.

[jeandet]

Thank you for your prompt reply :).
Those json files do the job for merging requests, maybe hapi-client could directly consume/expose them?

On observatory/spacecraft side, while I don't know yet the HAPI spec and how it deals with what information is mandatory or not, I would say that the more information (metadata) we can preserve from previous APIs or even source files the better it is.
That said, in our specific case our constraints/goals are:

• being able to build a hierarchical tree with mission/[spacecraft]/instrument/dataset/[component] where all part are human readable
• being able to search/filter tree with simple keywords (could be mission, instrument name, or any meta-data)
• being able to know what kind of data we are getting (scalar, vector, spectrogramme,...) to provide some contextual operations and for example, prevent user from plotting a spectrogramme over another
• being able to know data version for cache eviction (could be either version or last production date)

As a consumer of previous REST APIs and HAPI newcomer, I see HAPI as an opportunity to drop specific code for each server and provide a common minimal set of metadata.

[rweigel]

I'm not sure about using these files in hapi-client. In another post, you mentioned splitting the plotting part of hapi-client into a different package, and I agree. The plotting part was put in there to allow for testing the client as it was developed, but it has grown to the point where it should be separated out. It seems that there could be another package that allows users to search and presents them with a selection tree.

There is another project that will eventually use the list that Jeremy is creating (I'll call it the aggregated catalog). At present, http://hapi-server.org/servers/ builds things dynamically and this means that one can only search in a given drop-down list. With the aggregated catalog, it will be easy to search across everything.

[jbfaden]

Hello Alexis, I apologize for not really understating this ticket until now. I imagine we'll need some sort of hierarchy as the number of datasets grows. I have a couple of thoughts on this. My first inclination is to implicitly treat "/" as a delimiter character. In this case Autoplot would see these slashes and use a tree to allow selection of data. This is how we do things with the Das2Server API (which is a similar data API but was developed earlier). The Das2Server has had the problem that if things are rearranged, datasets lose their identity. Another way to handle this would be to have another tag in the catalog response which would develop a tree. This could use the mission/[spacecraft]/instrument/dataset/[component] model. Also software like Autoplot could combine datasets from different servers when they use the same heirarchy, making it easier to find things

[jeandet]

Hello @jbfaden thank you for your feedback, my turn to apologize :), I'm focused on something else for the next few weeks. I think a solution with an additional path tag might perfectly fit our needs since it just adds one tag and might give some flexibility about depth. For example it would work well with either single or multi spacecraft missions. Then I also think that / is ok since it is really common as path separator and nobody should use that in an instrument/mission/dataset name.
Do you think, it could be a mandatory field in datasets description, with some guidelines about how to fill it?

[jbfaden]

I have to apologize now, I missed the notification you'd responded and haven't been thinking about this. I plan on working more on Autoplot and HAPI in the next few months, so let me know if this is still something you are thinking about too.

This morning, reviewing your comments, I'm reminded of other systems which have a separate sort control. This is nice because sort is what you use when discovering data, which is separate from an id which is used to identify data. For example, you might want to move a dataset from one folder to a more appropriate one, and still have old references to the data work.

[jeandet]

@jbfaden , no problem, GH notifications are quite hard to sort/filter.
I'm still interested in supporting HAPI protocol in Speasy(formerly known as SPWC), actually I solved my problem with CDAWeb by "simply" parsing both this file and cdf masters from here.

Since our latest release we serve inventories through our cache server, here is an example of file we produce for CDAWeb inventory. We are able to produce a similar file with all supported servers where we keep the mission/spacecraft/instrument/dataset/parameter relationship. I don't really mind about the schema but this gives an example of the kind of information we would consume if we switch to HAPI.