QDR/6886 external status labels

conf/solr/8.8.1/schema.xml

@@ -162,6 +162,7 @@
     <field name="dvSubject" type="string" stored="true" indexed="true" multiValued="true"/>
 
     <field name="publicationStatus" type="string" stored="true" indexed="true" multiValued="true"/>
+    <field name="externalStatus" type="string" stored="true" indexed="true" multiValued="false"/>
 
     <field name="subtreePaths" type="string" stored="true" indexed="true" multiValued="true"/>
 

doc/release-notes/7967-curation-labels.md

@@ -0,0 +1,19 @@
+### Curation Status Labels
+
+A new :AllowedCurationLabels setting allows a sysadmins to define one or more sets of labels that can be applied to a draft Dataset version via the user interface or API to indicate the status of the dataset with respect to a defined curation process.
+
+Labels are completely customizable (alphanumeric or spaces, up to 32 characters, e.g. "Author contacted", "Privacy Review", "Awaiting paper publication"). Superusers can select a specific set of labels, or disable this functionality per collection. Anyone who can publish a draft dataset (e.g. curators) can set/change/remove labels (from the set specified for the collection containing the dataset) via the user interface or via an API. The API also would allow external tools to search for, read and set labels on Datasets, providing an integration mechanism. Labels are visible on the Dataset page and in Dataverse collection listings/search results. Internally, the labels have no effect, and at publication, any existing label will be removed. A reporting API call allows admins to get a list of datasets and their curation statuses.
+
+The Solr schema must be updated as part of installing the release of Dataverse containing this feature for it to work.
+
+## Additional Release Steps
+
+1\. Replace Solr schema.xml to allow Curation Labels to be used
+
+-stop solr instance (usually service solr stop, depending on solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/5.7/installation/prerequisites.html#solr-init-script)
+
+-replace schema.xml
+
+cp /tmp/dvinstall/schema.xml /usr/local/solr/solr-8.8.1/server/solr/collection1/conf
+
+-start solr instance (usually service solr start, depending on solr/OS)

doc/sphinx-guides/source/admin/dataverses-datasets.rst

@@ -40,7 +40,7 @@ Recursively assigns the users and groups having a role(s),that are in the set co
 
     curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/$dataverse-alias/addRoleAssignmentsToChildren
 
-Configure a Dataverse Collection to store all new files in a specific file store
+Configure a Dataverse Collection to Store All New Files in a Specific File Store
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To direct new files (uploaded when datasets are created or edited) for all datasets in a given Dataverse collection, the store can be specified via the API as shown below, or by editing the 'General Information' for a Dataverse collection on the Dataverse collection page. Only accessible to superusers. ::
@@ -61,6 +61,36 @@ The available drivers can be listed with::
 
 (Individual datasets can be configured to use specific file stores as well. See the "Datasets" section below.)
 
+Configure a Dataverse Collection to Allow Use of a Given Curation Label Set
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Datasets within a given Dataverse collection can be annotated with a Curation Label to indicate the status of the dataset with respect to a defined curation process. Labels are completely customizable (alphanumeric or spaces, up to 32 characters, e.g. "Author contacted", "Privacy Review", "Awaiting paper publication").
+
+The label is applied to a draft Dataset version via the user interface or API and the available label sets are defined by :ref:`:AllowedCurationLabels <:AllowedCurationLabels>`. Internally, the labels have no effect, and at publication, any existing label will be removed. A reporting API call allows admins to get a list of datasets and their curation statuses.
+
+The label set used for a collection can be specified via the API as shown below, or by editing the 'General Information' for a Dataverse collection on the Dataverse collection page. Only accessible to superusers.
+
+The curationLabelSet to use within a given collection can be set by specifying its name using::
+
+    curl -H "X-Dataverse-key: $API_TOKEN" -X PUT -d $curationLabelSetName http://$SERVER/api/admin/dataverse/$dataverse-alias/curationLabelSet
+
+The reserved word "DISABLED" can be used to disable this feature within a given Dataverse collection. 
+
+The name of the current curationLabelSet can be seen using::
+
+    curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/$dataverse-alias/curationLabelSet
+
+and can be reset to the default (inherited from the parent collection or DISABLED for the root collection) with::
+
+    curl -H "X-Dataverse-key: $API_TOKEN" -X DELETE http://$SERVER/api/admin/dataverse/$dataverse-alias/curationLabelSet
+
+The available curation label sets can be listed with::
+
+    curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/curationLabelSet
+
+If the :AllowedCurationLabels setting has a value, one of the available choices will always be "DISABLED" which allows curation labels to be turned off for a given collection/dataset.
+
+Individual datasets can be configured to use specific curationLabelSets as well. See the "Datasets" section below.
 
 Datasets
 --------
@@ -134,7 +164,7 @@ Diagnose Constraint Violations Issues in Datasets
 
 To identify invalid data values in specific datasets (if, for example, an attempt to edit a dataset results in a ConstraintViolationException in the server log), or to check all the datasets in the Dataverse installation for constraint violations, see :ref:`Dataset Validation <dataset-validation-api>` in the :doc:`/api/native-api` section of the User Guide.
 
-Configure a Dataset to store all new files in a specific file store
+Configure a Dataset to Store All New Files in a Specific File Store
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Configure a dataset to use a specific file store (this API can only be used by a superuser) ::
@@ -153,4 +183,33 @@ The available drivers can be listed with::
 
     curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/storageDrivers
 
+Configure a Dataset to Allow Use of a Curation Label Set
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A dataset can be annotated with a Curation Label to indicate the status of the dataset with respect to a defined curation process. Labels are completely customizable (alphanumeric or spaces, up to 32 characters, e.g. "Author contacted", "Privacy Review", "Awaiting paper publication").
+
+The label is applied to a draft Dataset version via the user interface or API and the available label sets are defined by :ref:`:AllowedCurationLabels <:AllowedCurationLabels>`. Internally, the labels have no effect, and at publication, any existing label will be removed. A reporting API call allows admins to get a list of datasets and their curation statuses.
+
+The label set used for a dataset can be specified via the API as shown below. Only accessible to superusers.
+
+The curationLabelSet to use within a given dataset can be set by specifying its name using::
+
+    curl -H "X-Dataverse-key: $API_TOKEN" -X PUT -d $curationLabelSetName http://$SERVER/api/datasets/$dataset-id/curationLabelSet
+
+The reserved word "DISABLED" can be used to disable this feature within a given Dataverse collection. 
+
+The name of the current curationLabelSet can be seen using::
+
+    curl http://$SERVER/api/datasets/$dataset-id/curationLabelSet
+
+and can be reset to the default (inherited from the parent collection) with (only a superuser can do this) ::
+
+    curl -H "X-Dataverse-key: $API_TOKEN" -X DELETE http://$SERVER/api/datasets/$dataset-id/curationLabelSet
+
+The available curationLabelSets can be listed with::
+
+    curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/curationLabelSets
+
+If the :AllowedCurationLabels setting has a value, one of the available choices will always be "DISABLED" which allows curation labels to be turned off for a given collection/dataset.
 
+Collections can be configured to use specific curationLabelSets as well. See the "Dataverse Collections" section above.

doc/sphinx-guides/source/api/curation-labels.rst

@@ -0,0 +1,95 @@
+Dataset Curation Label API
+==========================
+
+When the :ref:`:AllowedCurationLabels <:AllowedCurationLabels>` setting has been used to define Curation Labels, this API can be used to set these labels on draft datasets. 
+Superusers can define which set of labels are allowed for a given datasets in a collection/an individual dataset using the api described in the :doc:`/admin/dataverses-datasets` section.
+The API here can be used by curators/those who have permission to publish the dataset to get/set/change/delete the label currently assigned to a draft dataset.
+
+This functionality is intended as a mechanism to integrate the Dataverse software with an external curation process/application: it is a way to make the state of a draft dataset, 
+as defined in the external process, visible within Dataverse. These labels have no other effect in Dataverse and are only visible to curators/those with permission to publish the dataset. 
+Any curation label assigned to a draft dataset will be removed upon publication.
+
+Get a Draft Dataset's Curation Label
+------------------------------------
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export DATASET_ID='12345'
+  export DATASET_PID='doi:10.5072/FK2A1B2C3'
+  export SERVER_URL=https://demo.dataverse.org
+
+  Example 1: Get the label using the DATASET ID
+
+    curl -H X-Dataverse-key:$API_TOKEN "$SERVER_URL/api/datasets/$DATASET_ID/curationStatus"
+
+  Example 2: Get the label using the DATASET PID
+
+    curl -H X-Dataverse-key:$API_TOKEN "$SERVER_URL/api/datasets/:persistentId/curationStatus?persistentId=$DATASET_PID"
+
+You should expect a 200 ("OK") response and the draft dataset's curation status label contained in a JSON 'data' object.
+
+
+Set a Draft Dataset's Curation Label
+------------------------------------
+
+To add a curation label for a draft Dataset, specify the Dataset ID (DATASET_ID) or Persistent identifier (DATASET_PID) and the label desired.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export DATASET_ID='12345'
+  export DATASET_PID='doi:10.5072/FK2A1B2C3'
+  export LABEL='Author contacted'
+  export SERVER_URL=https://demo.dataverse.org
+
+  Example: Add the label using the DATASET ID 
+
+    curl -X PUT -H X-Dataverse-key:$API_TOKEN "$SERVER_URL/api/datasets/$DATASET_ID/curationStatus?label=$LABEL"
+
+  Example 2: Add a description using the DATASET PID
+
+    curl -X PUT -H X-Dataverse-key:$API_TOKEN "$SERVER_URL/api/datasets/:persistentId/curationStatus?label=$LABEL&persistentId=$DATASET_PID"
+
+You should expect a 200 ("OK") response indicating that the label has been set. 403/Forbidden and 400/Bad Request responses are also possible, i.e. if you don't have permission to make this change or are trying to add a label that isn't in the allowed set or to add a label to a dataset with no draft version.
+
+
+Delete a Draft Dataset's Curation Label
+---------------------------------------
+
+To delete the curation label on a draft Dataset, specify the Dataset ID (DATASET_ID) or Persistent identifier (DATASET_PID).
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export DATASET_PID='doi:10.5072/FK2A1B2C3'
+  export SERVER_URL=https://demo.dataverse.org
+
+  Example: Delete the label using the DATASET PID
+
+    curl -X DELETE -H X-Dataverse-key:$API_TOKEN "$SERVER_URL/api/datasets/:persistentId/curationStatus?persistentId=$DATASET_PID"
+
+You should expect a 200 ("OK") response indicating the label has been removed.
+
+
+Get the Set of Allowed Labels for a Dataset
+-------------------------------------------
+
+To get the list of allowed curation labels allowed for a given Dataset
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export DATASET_ID='12345'
+  export DATASET_PID='doi:10.5072/FK2A1B2C3'
+  export SERVER_URL=https://demo.dataverse.org
+
+  Example 1: Get the label using the DATASET ID
+
+    curl -H X-Dataverse-key:$API_TOKEN "$SERVER_URL/api/datasets/$DATASET_ID/allowedCurationLabels"
+
+  Example 2: Get the label using the DATASET PID
+
+    curl -H X-Dataverse-key:$API_TOKEN "$SERVER_URL/api/datasets/:persistentId/allowedCurationLabels?persistentId=$DATASET_PID"
+
+You should expect a 200 ("OK") response with a comma-separated list of allowed labels contained in a JSON 'data' object.

doc/sphinx-guides/source/api/index.rst

@@ -20,5 +20,6 @@ API Guide
    sword
    client-libraries
    external-tools
+   curation-labels
    apps
    faq

doc/sphinx-guides/source/developers/index.rst

@@ -39,3 +39,4 @@ Developer Guide
    dataset-migration-api 
    workflows
    fontcustom
+

doc/sphinx-guides/source/installation/config.rst

@@ -2385,4 +2385,17 @@ Scripts that implement this association for specific service protocols are maint
 
 ``curl -X PUT --upload-file cvoc-conf.json http://localhost:8080/api/admin/settings/:CVocConf``
 
+.. _:AllowedCurationLabels:
+
+:AllowedCurationLabels
+++++++++++++++++++++++
+
+A JSON Object containing lists of allowed labels (up to 32 characters, spaces allowed) that can be set, via API or UI by users with the permission to publish a dataset. The set of labels allowed 
+for datasets can be selected by a superuser - via the Dataverse collection page (Edit/General Info) or set via API call. 
+The labels in a set should correspond to the states in an organization's curation process and are intended to help users/curators track the progress of a dataset through a defined curation process. 
+A dataset may only have one label at a time and if a label is set, it will be removed at publication time. 
+This functionality is disabled when this setting is empty/not set.
+Each set of labels is identified by a curationLabelSet name and a JSON Array of the labels allowed in that set.
+
+``curl -X PUT -d '{"Standard Process":["Author contacted", "Privacy Review", "Awaiting paper publication", "Final Approval"], "Alternate Process":["State 1","State 2","State 3"]}' http://localhost:8080/api/admin/settings/:AllowedCurationLabels``
 

src/main/java/edu/harvard/iq/dataverse/DatasetPage.java

@@ -107,6 +107,7 @@
 import edu.harvard.iq.dataverse.engine.command.impl.PublishDatasetResult;
 import edu.harvard.iq.dataverse.engine.command.impl.RestrictFileCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.ReturnDatasetToAuthorCommand;
+import edu.harvard.iq.dataverse.engine.command.impl.SetCurationStatusCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.SubmitDatasetForReviewCommand;
 import edu.harvard.iq.dataverse.externaltools.ExternalTool;
 import edu.harvard.iq.dataverse.externaltools.ExternalToolServiceBean;
@@ -5546,4 +5547,25 @@ public List<String> getVocabScripts() {
     public String getFieldLanguage(String languages) {
         return fieldService.getFieldLanguage(languages,session.getLocaleCode());
     }
+
+    public void setExternalStatus(String status) {
+        try {
+            dataset = commandEngine.submit(new SetCurationStatusCommand(dvRequestService.getDataverseRequest(), dataset, status));
+            workingVersion=dataset.getLatestVersion();
+            if (status == null || status.isEmpty()) {
+                JsfHelper.addInfoMessage(BundleUtil.getStringFromBundle("dataset.externalstatus.removed"));
+            } else {
+                JH.addMessage(FacesMessage.SEVERITY_INFO, BundleUtil.getStringFromBundle("dataset.externalstatus.header"), BundleUtil.getStringFromBundle("dataset.externalstatus.info", Arrays.asList(status)));
+            }
+
+        } catch (CommandException ex) {
+            String msg = BundleUtil.getStringFromBundle("dataset.externalstatus.cantchange");
+            logger.warning("Unable to change external status to " + status + " for dataset id " + dataset.getId() + ". Message to user: " + msg + " Exception: " + ex);
+            JsfHelper.addErrorMessage(msg);
+        }
+    }
+
+    public List<String> getAllowedExternalStatuses() {
+        return settingsWrapper.getAllowedExternalStatuses(dataset);
+    }
 }

src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java

@@ -198,6 +198,11 @@ public List<Long> findAllUnindexed() {
         return em.createQuery("SELECT o.id FROM Dataset o WHERE o.indexTime IS null ORDER BY o.id DESC", Long.class).getResultList();
     }
 
+    //Used in datasets listcurationstatus API
+    public List<Dataset> findAllUnpublished() {
+        return em.createQuery("SELECT object(o) FROM Dataset o, DvObject d WHERE d.id=o.id and d.publicationDate IS null ORDER BY o.id ASC", Dataset.class).getResultList();
+    }
+
     /**
      * For docs, see the equivalent method on the DataverseServiceBean.
      * @param numPartitions

src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java

@@ -171,6 +171,8 @@ public enum License {
     @OneToMany(mappedBy = "datasetVersion", cascade={CascadeType.REMOVE, CascadeType.MERGE, CascadeType.PERSIST})
     private List<WorkflowComment> workflowComments;
 
+    @Column(nullable=true)
+    private String externalStatusLabel;
 
     public Long getId() {
         return this.id;
@@ -1961,4 +1963,12 @@ public String getLocaleLastUpdateTime() {
         return DateUtil.formatDate(new Timestamp(lastUpdateTime.getTime()));
     }
 
+    public String getExternalStatusLabel() {
+        return externalStatusLabel;
+    }
+
+    public void setExternalStatusLabel(String externalStatusLabel) {
+        this.externalStatusLabel = externalStatusLabel;
+    }
+
 }

src/main/java/edu/harvard/iq/dataverse/DataversePage.java

@@ -36,6 +36,7 @@
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.HashMap;
+import java.util.HashSet;
 import java.util.Map;
 import java.util.Map.Entry;
 import java.util.Set;
@@ -1210,4 +1211,49 @@ public Set<Entry<String, String>> getMetadataLanguages() {
         return settingsWrapper.getMetadataLanguages(this.dataverse).entrySet();
     }
 
+    public Set<Entry<String, String>> getCurationLabelSetOptions() {
+        HashMap<String, String> setNames = new HashMap<String, String>();
+        Set<String> allowedSetNames = systemConfig.getCurationLabels().keySet();
+        if (allowedSetNames.size() > 0) {
+            // Add an entry for the default (inherited from an ancestor or the system
+            // default)
+            String inheritedLabelSet = getCurationLabelSetNameLabel();
+            if (!StringUtils.isBlank(inheritedLabelSet)) {
+                setNames.put(inheritedLabelSet,SystemConfig.DEFAULTCURATIONLABELSET);
+            }
+            // Add an entry for disabled
+            setNames.put(BundleUtil.getStringFromBundle("dataverse.curationLabels.disabled"), SystemConfig.CURATIONLABELSDISABLED);
+            allowedSetNames.forEach(name -> {
+                setNames.put(name, name);
+            });
+        }
+        return setNames.entrySet();
+    }
+
+    public String getCurationLabelSetNameLabel() {
+        Dataverse parent = dataverse.getOwner();
+        String setName = null;
+        boolean fromAncestor = false;
+        if (parent != null) {
+            setName = parent.getEffectiveCurationLabelSetName();
+            // recurse dataverse chain to root and if any have a curation label set name set,
+            // fromAncestor is true
+            while (parent != null) {
+                if (!parent.getCurationLabelSetName().equals(SystemConfig.DEFAULTCURATIONLABELSET)) {
+                    fromAncestor = true;
+                    break;
+                }
+                parent = parent.getOwner();
+            }
+        }
+        if (setName != null) {
+            if (fromAncestor) {
+                setName = setName + " " + BundleUtil.getStringFromBundle("dataverse.inherited");
+            } else {
+                setName = setName + " " + BundleUtil.getStringFromBundle("dataverse.default");
+            }
+        }
+        return setName;
+    }
+
 }