how to handle datasets based on time ranges

[jbfaden]

previous title: consider metadata to hint that second column stop time should be included in data responses

Jon and I thought we might address the startTime/stopTime overlapping a visible interval would be handled with a property of stop time. For example the second parameter would have a "stopTime":true property, and the server should send any interval that overlaps with the requested interval. (Presently only the startTime, which would need to be in the interval, is considered.)

Suppose you have a table of instrument modes for the mission. For the first three years, mode A is used, and for the second four years, mode B is used. So if you requested the entire interval, you would get:

2019-01-01T00Z,2022-01-01T00Z,A
2022-01-01T00Z,2026-01-01T00Z,B

If the stop time column is marked as a stop time, then requesting start=2020-01-01T00:00 and stop=2020-01-02T00:00 would still return

2019-01-01T00Z,2022-01-01T00Z,A

because it spans the interval.

[jbfaden]

Or use ISO8601 time intervals, as a new data type:

2023-04-03T00:00/2023-04-07T00:00
2023-04-03T00:00/P1D

This would be type "isotimeInterval" instead of "isotime"...

[jbfaden]

/data?start=2023-04-03T00:00&stop=2023-04-04T00:00

/data?timeInterval=2023-04-03T00:00/P1D

/data?timeInterval=2023-04-03T00:00/P1D&extendInterval=true
/data?timeInterval=2023-04-03T00:00/P1D&mustContainStart=true&mustContainStop=true

includeOverlap
overlapLogic=true
eventLogic=true
events=true
eventRequest=true
intervalRequest=true

where(eventStop.gt(requestStart)) && where(eventStart.lt(requestStop))

[jbfaden]

Info response has something to indicate they are interval records:

{
    "intervalBasedRecords": true
    "timeStampLocation":"begin"
    "parameters": {
         "start":{ 

         }
         "stop": {
             
         }
         "file": {
         }

    }
}

[jvandegriff]

Conclusions from discussion today:

(this text could go in the spec to explain the new capabilities)

Starting with version 3.x, HAPI supports a standard way to express interval-based time series data using the "intervalBasedRecords": "true" attribute. Examples where this is important include event lists, data availability lists, and file listings. For this kind of data, each row in the HAPI dataset (rows are also called records) explicitly covers a range of time. The first column in a HAPI dataset must be a time column, and for non-interval-based data, that time value in each record represents the single time value associated with that record. For interval-based data, that first time column expresses the beginning of the time interval for the record, and the second column must also be a time column that represents the end time of the interval. The timeStampLocation attribute must be present with a value of begin to indicate that the values in the first time column are the beginning of the time interval for the record.

Here is an example of an info/ response for an interval-based dataset:

{
   "HAPI": "3.2",
   "status": { "code": 1200, "message": "OK"},
   "startDate": "1998-001Z",
   "stopDate" : "2017-100Z",
   "intervalBasedRecords": true,
   "timeStampLocation":"begin",
   "parameters": [
       { "name": "IntervalBeginTime","type": "isotime", "units": "UTC", "fill": null,"length": 24 },
       { "name": "IntervalEndTime","type": "isotime", "units": "UTC", "fill": null,"length": 24 },
       { "name": "solar_flare_ID", "type": "string", "units": dimensionless, "fill": null,
                "description": "identifier for a specific solar flare in this event list", "label": "flare ID"}
     ]
}

When a server returns records from an interval-based dataset, the default is to return only records in which the begin time of the data falls within the requested time range. This could miss any interval which begins before the requested start time, but then ends inside (or after) the requested interval. To support a user request for all intervals that have any overlap with the requested time range, a request parameter intervalRequest=true can be added to the data request. A full URL for such a request might look like this (line breaks are for clarity):

https:/server.com/hapi/data?
    dataset=CME_EVENT_LIST&
    start=2024-01-01T00:00Z&
    stop=2024-01-02T00:00Z
    &intervalRequest=true

If a CME event in this list started before 2024-01-01 but ended inside the interval (or even after), including
intervalRequest=true means the server must include that event in the server response, whereas if the data request did not
specifyintervalRequest=true, then the server would not return this event.

[jvandegriff]

We also decided that any kind of schema that is needed for the specific kinds of interval-based data (like a file listing) needs to be done separately, but is important, especially for file listings since for them to be useful, there needs to be a standard people can target.

But just being able to say that data is interval-based is already useful by itself, so the "intervalBasedRecords": true, designation does not need to be accompanied by any schema assignment (which we haven't figured out anyway).

[VoyagerPWS]

Forgive me if I haven't been following closely enough to make an intelligent comment, but here goes anyway. ;-)
IMHO, there are only two reliable ways to express an interval:

1. begin time (inclusive) and time span, like 2024-09-27T00:00Z P1D (well, it doesn't have to be begin time, but some time point)
2. begin time (inclusive) and end time (exclusive), like 2024-09-27 to 2024-09-28

Note that both of these specify the same interval and that there is absolutely no time within day 27 that can be used to indicate an end time.
Ideally, given data in an interval from t0 to t1 and a request in interval T0 to T1, the data should be returned if t1 > T0 and t0 < T1 (again, inclusive/exclusive times).
Finally, I note that no measurement that I can think of occurs at a point in time. Everything is a time interval.

[jbfaden]

ISO8601 is quite specific about time ranges, and you will be happy to know Larry that it is also exclusive, so 2024-01-01/2024-02-01 does not include the day 2024-02-01. (Also it's 2024-09-27T00:00Z/P1D, slash not a space.)