Provenance tracking using semantic web technologies?

[DamienIrving]

In recent years, Semantic Web technologies have been used to record the data processing steps involved in producing climate products (i.e. maps, plots or any other climate research outcome stored in a file).

While Python packages such as rook and ESMValTool simply define their own bespoke / narrow adaptation of the PROV / RDF data model to suit their own needs, there have been attempts to define a comprehensive ontology for climate products (e.g. Bedia et al 2019, Zhang et al., 2020).

As far as I can tell, the most widely used ontology is METACLIP (METAdata for CLImate Products; see their website, flyer and paper), which was initially developed for the Copernicus QA4Seas seasonal forecasting project and is now also being used for the VALUE downscaling initiative. The METACLIP developers work in R and have integrated their approach to provenance tracking into the climate4R package.

I'm wondering if there's any interest in trying to figure out how to incorporate provenance tracking into xarray? There isn't a Python implementation of METACLIP yet, but presumably one could implement the ontology using rdflib. @huard suggested cf-xarray might be a good place to start the conversation around this, but happy to move the discussion elsewhere if it would be more appropriate?

[dcherian]

@DamienIrving thanks for opening this thoughtful issue.

I'm wondering if there's any interest in trying to figure out how to incorporate provenance tracking into xarray?

Yes absolutely.

but happy to move the discussion elsewhere if it would be more appropriate?

I think this should be discussed in a new issue on the xarray github. It will get a lot more visibility and thoughtful input. Assuming that provenance could be tracked through a fancy attr, it will also tie in to the discussions at pydata/xarray#988, pydata/xarray#4896, and pydata/xarray#3891 (comment)

I think users would want to opt-in to provenance tracking at the xarray level rather than only when calling functions through the .cf namespace. But cf-xarray or some other package could provide a function that handles incoming provenance attributes and returns an appropriately merged set of attributes.

[huard]

@dcherian I agree there is value in getting input from the xarray crowd, and I agree provenance should be supported at the xarray level. However, I'm concerned that finding an all-purpose solution that works for every discipline will end up in an infinite github comment thread. I'd like to first define what the CF community wants, and then carry this proposal to xarray for feedback.

Based on discussions at the 2020 CF meeting and private discussions with provenance pros, here would be some development guidelines:

• provenance is logged in an external file, linked in the CF attributes (e.g. has_provenance = <link to provenance file>)
• provenance metadata is injected through external templates that can be modified by users. (https://lucmoreau.wordpress.com/2015/07/30/provtoolbox-tutorial-4-templates-for-provenance-part-1/)
• provenance information should be machine readable and use existing, standard, ontologies.

[dcherian]

I'm concerned that finding an all-purpose solution that works for every discipline will end up in an infinite github comment thread.

Fair point, but I think it would still be valuable to figure out how to do this technically. Xarray is unlikely to provide provenance handling directly so we'll have to figure out how to hook in to it. The biweekly xarray dev meeting is tomorrow (Wednesday @ 9.30am Mountain Time). @huard & @DamienIrving, if you can, I think it would be good for you to attend and raise this issue there. If not, the meeting repeats every two weeks... We can use it to drive the conversation forward.

I'd like to first define what the CF community wants,

👍 but the audience on this repo is really tiny ! Maybe https://discourse.pangeo.io/?

[huard]

Makes sense, will try those suggestions. Thanks !

[huard]

https://discourse.pangeo.io/t/tracking-provenance-in-xarray/1510

[DamienIrving]

Thanks, @dcherian. The xarray dev meeting is 1:30am Australian Eastern Standard Time so I won't be able to make it, but if you're available @huard that sounds like a good idea?

[huard]

Hi @DamienIrving
I attended the meeting and presented the issue. There's an open PR that could possibly be used a building block to track provenance: pydata/xarray#4896

[dcherian]

So... I'm thinking that cf_xarray can provide two functions: track_provenance and track_history that users could provide to that combine_attrs setting.

[jbusecke]

Just stumbled upon this on discourse, and I was wondering if xgcm (xgcm/xgcm#143) could piggy back on these efforts?

[dcherian]

Ah I guess another angle is how projects like xgcm can plug in to the provenance tracking infrastructure. xgcm could easily set the history attribute in each method today.

[huard]

@dcherian I like this idea. track_history would be the human-readable version, directly embedded in the object's attributes, and track_provenance the machine-readable version stored in an external resource (file or db).

@jbusecke I think the spirit is to have xarray expose a hook, and then let libraries configure those hooks to do the actual provenance tracking. I see no reason why xgcm couldn't implement its own provenance mechanism, built on or in parallel with cf-xarray using the same hook.

[jbusecke]

This sounds great. Please keep me posted if there is anything I can help with or test on the xgcm side.

[dcherian]

See #253. Opinions are welcome!