Usage of datestamp from Record during harvest using oai_dc format

[tcoupin]

When I setup a client using oai_dc format, the datestamp from Record is used as release time and is display above the citation:

<header>
<identifier>https://doi.org/10.15454/1TPZGG</identifier>
<datestamp>2020-10-08T13:01:11Z</datestamp>
<setSpec>Doi2Pmh</setSpec>
<setSpec>test-dduni</setSpec>
</header>

and result :

I think it is the good behavior because of the definition of datestamp tag: "the date of creation, modification or deletion of the record for the purpose". This is coherent with hosted datasets, this date is used as release time of datasetversion object and so for sorting dataset on display.

But when I harvest an other dataverse with this format, the datestamp is the export date of the record and this date can be updated without a new version of the dataset.

So I have 2 questions:

1- Why can the datestamp be updated without new version of the dataset?
2- Can I modify the code so that the datestamp will be the release time? A full haversting will be necessary for the client to rollback to the release time older than the current record datestamp

Which version of Dataverse are you using?

5.5 or 5.10.1

Any related open or closed issues to this bug report?

[poikilotherm]

Hi @tcoupin,

as I have my fingers dirty with XOAI and harvesting code right now, I tried to dig around the Dataverse OAI-PMH Client code to look if this might be a problem with the xoai-service-provider module. Turns out: it's not even used, as the complete parsing of OAI responses is done with a custom parser.

Digging deeper, I tried to find places where the <header><datestamp> field would be used. And what I found is not supporting your issue description: it seems to be ignored!

dataverse/src/main/java/edu/harvard/iq/dataverse/harvest/client/FastGetRecord.java

Lines 521 to 533 in e8435ac

	private void processHeader (XMLStreamReader xmlr) throws XMLStreamException {
	for (int event = xmlr.next(); event != XMLStreamConstants.END_DOCUMENT; event = xmlr.next()) {
	if (event == XMLStreamConstants.START_ELEMENT) {
	if (xmlr.getLocalName().equals("identifier")) {/*do nothing*/}
	else if (xmlr.getLocalName().equals("datestamp")) {/*do nothing -- ?*/}
	else if (xmlr.getLocalName().equals("setSpec")) {/*do nothing*/}
	} else if (event == XMLStreamConstants.END_ELEMENT) {
	if (xmlr.getLocalName().equals("header")) return;
	}
	}
	}

The import of the Dublin Core metadata written out as a file by FastGetRecord.harvestRecord() is quite complex. Maybe you can dig through this how the dates are mapped?

[poikilotherm]

Digging through the server code, I find that <datestamp> is created via harvest.server.xoai.Xitem.getDatestamp(), which requests OAIRecord.getLastUpdateTime(). This is a database entity, updated via the OAIRecordServiceBean.updateOaiRecordForDataset() with the current Date of method execution.

Possible ways to get there:

1. API trigger to /api/exportOAI/<setname>:
   Metadata.exportOaiSet() -> ... -> OAISetServiceBean.exportOaiSet() -> OAIRecordServiceBean.updateOaiRecords() -> OAIRecordServiceBean.updateOaiRecordForDataset()
2. UI interaction on the Sets page (creating, editing, ...):
   HarvestingSetsPage.runSetExport() -> OAISetServiceBean.exportOaiSet() -> OAIRecordServiceBean.updateOaiRecords() -> OAIRecordServiceBean.updateOaiRecordForDataset()
3. Export Timer Service:
   DataverseTimerServiceBean.handleTimeout() -> OAISetServiceBean.exportAllSets() -> OAISetServiceBean.exportOaiSet() -> OAIRecordServiceBean.updateOaiRecords() -> OAIRecordServiceBean.updateOaiRecordForDataset()

There is a check in place comparing the last export time of the record and the dataset, so the entry will only be updated if the last metadata export is newer than the OAI records last creation.

Of course this might be prone to a race condition or other error, where a dataset keeps getting exported etc. Someone would need to debug this more closely...

[tcoupin]

Hi @poikilotherm

In the client side:
the datestamp is extracted from the listidentifier response so it's not necessary to parse it again in the getrecord response:

dataverse/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java

Line 257 in e8435ac

Date dateStamp = h.getDatestamp();

The release date is finaly set in import service bean:

dataverse/src/main/java/edu/harvard/iq/dataverse/api/imports/ImportServiceBean.java

Line 278 in e8435ac

ds.getVersions().get(0).setReleaseTime(oaiDateStamp);

In the server side:
I found a lot of dataset where the datestamp does not match the last update date. Ex:

• 27/Oct/2021 on the ui : https://data.inrae.fr/dataset.xhtml?persistentId=doi:10.15454/58SYJV#datasetForm:tabView:versionsTab
• 6/May/2022 on the oai record : https://data.inrae.fr/oai?verb=GetRecord&metadataPrefix=oai_dc&identifier=doi:10.15454/58SYJV

With oai_dc format, there are only 2 dates: in the header and in date tag (not set in dataverse oai server responses). The date tag refers to the creation or publication date so it's not appropriate to use it to store the last modification date. Currently, the date is map to the publication date field of the dataset (https://github.com/IQSS/dataverse/blob/develop/src/main/resources/db/migration/afterMigrate__1-7256-upsert-referenceData.sql#L20)

I still don't unterdand why the datestamp is the date of the export and not the date of the last modification.

[poikilotherm]

You Sir @tcoupin are absolutely right about the client code using the header <datastamp>. Thanks for pointing that out! This part of the code is often confusing to read (which is why @landreev and me are trying to refactor it).

About the date missing from the oai_dc: it looks like this is a bug no one actually noticed for a LONG time.

Looking at DublinCoreExportUtil.createOAIDC() and the test class DublinCoreExportUtilTest the whole method was never covered, so no one noticed missing metadata. There are different dates that should be written, but this fails silently - empty elements are simply skipped.... Just - they should not be empty. xmlunit to the rescue... 🙈

And with regards to the <datestamp> getting updated, I bet there is a tricky to find race condition. @landreev looks like this whole metadata export thing needs love and tests. 🤯