Proposal for new "files" endpoint

[jbfaden]

Bob, Jon, and Jeremy were talking about a file listing events. We think this might be best implemented with a new "files" endpoint.

See also #176 and #70

The virtue of a new end point is that it would allow for some slightly different behavior, such as:

• second column is always the interval stop time
• the additional columns have a subset of specifically-named columns:
  • filename is required
  • file size
  • file last modified
  • file last accessed
  • file provenance
  • DOI
  • number of records
  • number of pixels, etc
• time range would not be required, not specifying means all records are sent
• if the stop time is within the requested interval, it is sent
• only CSV and JSON responses are allowed

[jbfaden]

We immediately started to question if all this could be more easily supported using schemas,
where "file listing" schema would ensure that the output could be just as useful.

[jvandegriff]

need JSON schema for info/ response for a file listing schema (possible columns are: time, filename, file size in bytes?, last mod time, plus other optional items that are user defined? - hapi will ignore these; COLUMNS NAMES are specified in the JSON schema!!!

The info response needs to indicate that it is compliance with said schema

[jvandegriff]

tricky things like file size or mod time could be optional, but if present they have to have a specific column name and be before any of the user-defined columns (that would all have their own non-standard names)

[rweigel]

DOI added to Jeremy's list b/c we recommend in FAIR appendix using a file list when files or time spans have DOIs.

[jvandegriff]

See this issue and comment:

#224 (comment)

[jvandegriff]

When finished, add comment to modificationDate attribute to indicate that more granularity on mod times can be obtained from a file listing if present

[jvandegriff]

Discussion about the file listing content and which files get included.

If a file has a time range with a start time before the requested interval and an end time after the requested interval, HAPI will not include this file (because HAPI has a standard rule that it only returns records with a start time within the requested interval).

There was not full agreement on this, but people generally wanted it this way.

If we had different behavior, we would maybe need a different endpoint.

It would also be good to have a file listing that was known to conform to a specific schema. The schema can have simple but required columns (start time of data in file, stop time of data in file, filename), and then zero or more (in any order) of named columns for other possible file attributes (need to decide which limited attributes to allow - see list above)

Could use different keyword not he request to indicate we DO want the potentially overlapping first time. Use start* instead of start