Handling requests for most recent data

[rweigel]

Email sent to hapi-dev@groups.io

We are looking for feedback from HAPI data providers with servers where the dataset end date/time changes.

Eleco has an application where he wants to update a plot when new data becomes available (for INTERMAGNET now, but eventually for others). To do this, he needs to periodically request the info to get the current stop date and then make a data request if the stop date has changed. 

Based on the discussion at #97, we propose to allow "stop=stopDate" in the request URL in HAPI 3.3. If a server supports this keyword, it would be indicated somehow in the /capabilities response. This would support his application, where a user hits reload to update the plot. 

One issue this does not address is the case where the application wants to update the plot automatically. To do this, one would hit the server with stop=stopDate on a timer, which could lead to many useless requests. As a result, we propose that our documentation indicates that the server can set a HTTP header such as "expires" (but which header is debatable) that indicates to the client when it should make a new request.

If you have any feedback, please respond to this thread (ideally) or this email thread. You can also attend our weekly HAPI developers meeting for additional discussion.

[SimonFlower]

To ensure I understand, "stopDate" is a symbolic value meaning "the date/time of the latest sample in the data set". If this is the case, then stopDate would be inclusive, whereas other stop values in HAPI are exclusive - probably a minor semantic point, but maybe worth noting? I don't see any problem implementing this in the Intermagnet server, where the information on the date/time of the latest sample for any data set is easily available.

The "expires" header is more complex to implement. Data providers send data to Intermagnet on a schedule of their own choosing, so working out what to put in the "expire" timer isn't easy. Initially I would think about using a value equivalent to the sample period of the data, but this wouln't give clients any more information than they already have. To do anything better than this I'd need to do some calculation. I record the "lag" time of data sets, by which I mean the difference between the current time and the date/time of the latest sample. I could set the expire time to an average of the lag time over the past 24 hours or 7 days. I'd need to see how much calculation this would involve (would it be less than the resource needed to respond to repeated data requests!)? The lag time is recorded once a minute, so for higher sample rate data sets (e.g. 1Hz data) with low lag times, it might not be an accurate representation of the time between updates from data providers.

To give an idea of real-time performance in Intermagnet, I think the "best" insitutes are delivering data within about 3 minutes of recording. Quite a number deliver hourly. The Intermagnet requirement (set in the 1990s and not updated since) is for delivery within 72 hours. So there's quite a bit of variation. We have an aspiration to achieve lag times of 2 minutes for 1-minute data and 30 seconds for 1-second data.

As a side issue (that I noted in the earlier discussions) we are also implementing MQTT and Seedlink (a streaming transfer protocol from the Seismic community) for data provision. This work is part of trying to improve real-time performance in Intermagnet. At some point in the future I imagine we might think about making these data streams available to the community, to support real-time data applications, however for Intermagnet that's a little way off. Some individual institutes (e.g. NRCan and possible USGS) already make their Geomagnetic data available on public Seedlink servers.

[candeynasa]

The term StopDate for time of last data isn't obvious. Perhaps something like "LatestDataTime" or "CurrentEndTime" would be clearer.

[jbfaden]

Thanks for your feedback. Simon we hadn't considered that the stop time is exclusive, thanks for pointing that out. This seems like a good reason to consider Bobby's suggestion, something like "latest" would nod to the slightly different behavior.