Convention for communicating that dataset that is file listing associated with another dataset

[rweigel]

Do we want one server with datasetID that is numerical data and datasetID/files that is URLs and another that uses the convention datasetID and FilesForDatasetID?

Should we have a recomentation? Or would this be addressed by grouping/linking as discussed in #118?

[rweigel]

This applies to all issues tagged "association". It seems that RDF was invented to address associations. So I am planning on studying https://www.w3.org/TR/rdf11-primer/.

[rweigel]

Hi Rebecca,

We are about to attempt to tackle the problem of connecting datasets in the HAPI specification. A few of the things we want to be able to express:

• Another version of this dataset is at a different cadence or quality control level (e.g., L0, L1, ... in NASA terminology and preliminary, quasi-definitive, and definitive in magnetometer speak).

• This dataset contains only data when in burst mode. The nominal mode data are in another dataset.

• This dataset was used to generate an event list dataset

• The files used for this dataset over a given time range are available by requesting a hapi server using a different dataset ID with the same time range.

• This dataset is from a satellite that is part of a constellation (e.g., RBSP-A, RBSP-B) (SPASE does this, so we may not need it).

This seems to be an RDF use case. Do you have any suggestions on how we should proceed or know anyone with experience with this that could help?

---

I am not experienced in RDF, but Ryan (cc’d) is. Catherine, our digital librarian, is also experienced in metadata. In general, I recommend imitating or copying the DataCite schema (https://schema.datacite.org/meta/kernel-4.4/doc/DataCite-MetadataKernel_v4.4.pdf), especially for datasets, and then mapping/copying that to other schema (e.g. HAPI).

• The version issue can be taken care of with different version numbers, similar to software. See the Version field (clarify whether timestamps are for the start, middle or end of the interval #15 in table 2).
• Likely best in the description
• Can use the relatedItem field likely with Collection as the item type and IsOriginalForm as the relationship (see bottom of page 33 and table 9 in the appendix)
• The most useful option would be alternateIdentifier = the link to the HAPI dataset with some explanatory text in alternateIdentifierType.
• The international jury is out on the best way to indicate links to facilities/missions/observatories and instruments, with an answer hoped for in the next two years. In the meantime, the current RDA recommendations do accommodate for DOIs to be created for each mission/instrument, with some good arguments for using this approach, but there are some issues (see https://www.rd-alliance.org/system/files/pidinst-schema-1.0_Final.pdf, particularly Table 1 on p. 4, and https://github.com/rdawg-pidinst/schema/blob/master/schema-datacite.rst for a deeper dive). The best way to do this in DataCite appears to be the RelatedItem field (free text) with Physical Object as the object type and possibly with IsDerivedFrom as the relation.

Time series data is also a current topic in the ESIP SOSO group (second link). Some earth science groups have been using the approach linked below to get time series data in schema.org, too. Likely another good link to HAPI.

Rebecca

iodepo/odis-arch#125

https://github.com/ESIPFed/science-on-schema.org

[rweigel]

From Baptiste:

Yes, indeed that is a nice use case.

The first step would be to check what relations are needed (e.g.: build an information model / schema), and build a "graph" with nodes and relations.

E.g. (not formally in any language, but just to propose something to start with :-):

dataset from_observatory RBSP .
dataset has_distribution distrib0 .
dataset has_distribution distrib1 .
dataset is nominal_mode .
dataset see_also dataset_L0 .

distrib0 has_resolution 1 sec .
distrib0 has_hapi_server https://...hapiurl...
distrib0 has_hapi_dataset hapi_dataset_id0
distrib0 other_resolution distrib1 .

distrib1 has_resolution 10 sec .
distrib1 has_hapi_server https://...hapiurl…
distrib1 has_hapi_dataset hapi_dataset_id1
distrib1 other_resolution distrib0 .

dataset_b is burst_mode .
dataset_b is_supplement_to dataset .

Then, see if there are existing terms/relations already available in other schemas/ontologies.

For instance, the concept of "dataset" is rather well defined in DCAT (https://www.w3.org/TR/vocab-dcat-3/), and it allows to describe the "distribution" of the dataset.

In particular, see this:
https://www.w3.org/TR/vocab-dcat-3/#Class:Distribution
(which already has temporal resolution even

[rebeccaringuette]

The ESIP SOSO link I sent has a link to their living agenda, which has several useful links on how other sciences are approaching this using the schema.org structure.

[jvandegriff]

Before Wednesday's meeting, review info bout RDF and DCAT definitions.
See if Doug L has any input.

[dlindhol]

The building blocks of RDF are subject-predicate-object triples that each represent a single fact, as exemplified by Baptiste above. You can link the same subject (e.g. a dataset) to multiple objects via a meaningful predicate. This is similar to defining properties for an object but more loosely coupled. An object from one triple can be used as a subject for another triple thus allowing you to build a graph. If you only want to hang properties off of a dataset without deeper linking, RDF might be overkill. Though you could still take inspiration from various ontologies for naming things.

The "R" is for "resource" so think of each of those three triple components as resources with unique identifiers and well defined semantics. There is no limit on how you name these things but it is clearly more useful if you adopt a preexisting ontology (think schema). Schema.org and DCAT seem to be the most popular for dataset related metadata. Google dataset search claims to support both, though it seems like the emphasis is on schema.org. DataCite also seems like a reasonable way to link related resources. Maybe even SPASE? At LASP, we take most of our inspiration from DCAT. We've added our own concepts to better capture our needs. We then strive to be able to crosswalk our metadata to other ontologies/schema.

Another important part of RDF is to be able to share your metadata in a standard format. JSON-LD (for "linked data") seems to be a common option. If we embrace RDF here, we might want to rethink the "info" response.

[jvandegriff]

Ideas on dataset relationships

Cadence (this one is special so that it is machine interpretable)
This needs work to clarify to be specific enough to be machine usable, but not tangled in the weeds.
Needs some isomorphism amongst parameter names so that potting tools can easily switch between datasets and still plot the same parameters.
Key point: we want to support Eelco's use case of auto-selection of cadence by a timeline viewer (client-side plotting tool).
The right kind of descriptor could specify a linkage at the dataset level (dataset A is linked to dataset B by cadence) and if the datasets are not similar enough to do this (i.e., the parameters have different names, even though they are just different by cadence), then the linkage descriptor could specify connections between specific parameters.

Maybe have this in a separate endpoint for linkages or relationships? Linkages are a kind of overlay, but it would also be nice to see it in the info response.

Argument for external: while having it in info is convenient, we likely need to have an external place to manage the complexity of the different kinds of linkages.

Others could be denied, but then are not necessarily machine interpretable - up to people to use as needed
Calibration Level (NASA Level 0, Level 1)
Processing Version
Quality level
Transform (FFT, coordinate, statistical (min/max). background removal)

[rweigel]

/relations returns this information in some JSON from similar to

[
[server:dataset1, hasRelatedCadence, server:dataset2],
[server:dataset1, sameMissionName, server:dataset2],
[server:dataset1:parameter1, isATransformedVersionOf, server:dataset1:parameter2],
[server1:dataset1:parameter1, isCoordinateTransformOf, server2:dataset1:dataset3],
[server1:dataset1, isDifferentCalibrationLevelOf, server1:dataset4]
[server1:dataset5, isFileListingOf, server1:dataset1]
[dataset1, x_SameReviewerAs, dataset4]
]

We define a list of predicates. No need to specify reverse relationships. Look into RDF predicates for dataset relationships.

[rweigel]

Next task: Come up with JSON schema for above.

[eelcodoornbos]

To make full use of associations between datasets for interactive plotting, the association between datasets is a first step. But we need to also be able to have a meaningful mapping on the parameter level. For example, I currently work with high rate satellite datasets, where it is useful to also have a low rate dataset with the per orbit (or lower cadence) minimum, mean and maximum of some (but not all) of these parameters. The OMNI datasets also have different parameter names for the same observable at the 1min, 5min and 1hr cadences. Is this something that can be accomplished with RDF, schema.org and the like? I’ll have to look into it.

[BaptisteCecconi]

As mentioned by @dlindhol, and since we use JSON in the HAPI headers, opting for JSON-LD (or other linked-data flavour) is important for interoperability (as usual). I hope we don't reinvent yet another linked-data format.

We also should reuse predicate from existing ontologies so that our links are understandable by generic tools.

As a by-product, we would have a better FAIR score when assessing our products/services with FAIR assessment tools.