Feature Request/Idea: Harvest metadata values that aren't from a list of controlled values

[DS-INRAE]

Overview of the Feature Request
When harvesting, the controlled values field (e.g. Subject) prove very limiting as the value expected for this field have to be in the list determined by the tsv.
It would be best if for harvested datasets these constraints were lifted.

What kind of user is the feature intended for?
(Example users roles: API User, Curator, Depositor, Guest, Superuser, Sysadmin)

What inspired the request?
Errors harvesting other data repositories. Example (with KindOfData field) :
<message>Exception processing getRecord(), oaiUrl=https://repo-int.ortolang.fr/api/oai, identifier=oai:ortolang.fr:e5bb1019-df55-41b8-bfb1-657dd9c37e3a, edu.harvard.iq.dataverse.api.imports.ImportException, Failed to import harvested dataset: class edu.harvard.iq.dataverse.util.json.ControlledVocabularyException (Value 'corpus' does not exist in type 'kindOfData')</message>

What existing behavior do you want changed?
Allow harvested datasets to have values out of the controlled list from the tsv.

Any brand new behavior do you want to add to Dataverse?
no

Any open or closed issues related to this feature request?
no ?

[pdurbin]

Related:

• API endpoint for creating a dataset by importing a DDI xml file fails when DDI file includes fields that Dataverse has a controlled vocabulary for #6559

[cmbz]

2023/12/19: Sized at 33. Some discussion about how best to implement the functionality. Plans to discuss this during a Tech Hours.

[landreev]

This is my recollection of the recent discussion of this at a tech hour:

• This is for harvesting only (i.e., a site should be able to configure this as an option - "accept values for such and such field even when they are not in the controlled vocab, when harvesting";
• We already have this notion, of relaxing our metadata requirements when importing harvested content - in the method ImportServiceBean.doImportHarvestedDataset(), some constraint violations are relaxed compared to "normal" imports;
• The way our metadata fields work, it should in fact be possible, to simply save an arbitrary field value for a field that has a controlled vocab., so this part should just work;
• Important to note that harvested metadata fields are only used for indexing in solr and nothing else.

[stevenwinship]

RE: > * The way our metadata fields work, it should in fact be possible, to simply save an arbitrary field value for a field that has a controlled vocab., so this part should just work;

I've tried this but one issue was that it created a new value that gets added to the UI dropdown list. The value wasn't added to the db table but it could be if needed. If we want to add it, it would require re-ordering the list or adding it to the end.
Another option is to simple remove the value and only save the values contained in the list. This keeps dv that is harvesting clean but then we loose the value for the field (possibly leaving it empty).

[qqmyers]

Did you save the value as a normal/non-controlled one or did you create a new controlled vocabulary value? I'd expect the latter to add a drop-down entry, but I think the hope was that the former would work.

[cmbz]

2024/02/22
Added to project; absence was an accidental oversight.