Merge branch 'develop' into harvest_exclude_invalid_tag

doc/release-notes/10708 - MDC Citation and DOI parsing improvements.md

@@ -1,3 +1,3 @@
 MDC Citation retrieval with the PID settings has been fixed.
-DOI parsing in Dataverse is case insensitive, improving interaction with services that may change the case.
+PID parsing in Dataverse is now case insensitive, improving interaction with services that may change the case of PIDs.
 Warnings related to managed/excluded PID lists for PID providers have been reduced

doc/release-notes/11012-get-dataverse-api-ext.md

@@ -0,0 +1 @@
+The JSON payload of the getDataverse endpoint has been extended to include properties isMetadataBlockRoot and isFacetRoot.

doc/sphinx-guides/source/api/native-api.rst

@@ -126,6 +126,8 @@ Same as in :ref:`create-dataverse-api`, the request JSON supports an optional ``
 
 To obtain an example of how these objects are included in the JSON file, download :download:`dataverse-complete-optional-params.json <../_static/api/dataverse-complete-optional-params.json>` file and modify it to suit your needs.
 
+See also :ref:`collection-attributes-api`.
+
 .. _view-dataverse:
 
 View a Dataverse Collection
@@ -1058,6 +1060,8 @@ The following attributes are supported:
 * ``affiliation`` Affiliation
 * ``filePIDsEnabled`` ("true" or "false") Restricted to use by superusers and only when the :ref:`:AllowEnablingFilePIDsPerCollection <:AllowEnablingFilePIDsPerCollection>` setting is true. Enables or disables registration of file-level PIDs in datasets within the collection (overriding the instance-wide setting).
 
+See also :ref:`update-dataverse-api`.
+
 .. _collection-storage-quotas:
 
 Update Collection Input Levels

doc/sphinx-guides/source/developers/making-releases.rst

@@ -10,10 +10,43 @@ Introduction
 
 This document is about releasing the main Dataverse app (https://github.com/IQSS/dataverse). See :doc:`making-library-releases` for how to release our various libraries. Other projects have their own release documentation.
 
-The steps below describe making both regular releases and hotfix releases.
-
 Below you'll see branches like "develop" and "master" mentioned. For more on our branching strategy, see :doc:`version-control`.
 
+Regular or Hotfix?
+------------------
+
+Early on, make sure it's clear what type of release this is. The steps below describe making both regular releases and hotfix releases.
+
+- regular
+
+  - e.g. 6.5 (minor)
+  - e.g. 7.0 (major)
+
+- hotfix
+
+  - e.g. 6.4.1 (patch)
+  - e.g. 7.0.1 (patch)
+
+Ensure Issues Have Been Created
+-------------------------------
+
+In advance of a release, GitHub issues should have been created already that capture certain steps. See https://github.com/IQSS/dataverse-pm/issues/335 for examples.
+
+Declare a Code Freeze
+---------------------
+
+The following steps are made more difficult if code is changing in the "develop" branch. Declare a code freeze until the release is out. Do not allow pull requests to be merged.
+
+Conduct Performance Testing
+---------------------------
+
+See :doc:`/qa/performance-tests` for details.
+
+Conduct Smoke Testing
+---------------------
+
+See :doc:`/qa/testing-approach` for details.
+
 .. _write-release-notes:
 
 Write Release Notes
@@ -23,26 +56,51 @@ Developers express the need for an addition to release notes by creating a "rele
 
 The task at or near release time is to collect these snippets into a single file.
 
-- Create an issue in GitHub to track the work of creating release notes for the upcoming release.
+- Find the issue in GitHub that tracks the work of creating release notes for the upcoming release.
 - Create a branch, add a .md file for the release (ex. 5.10.1 Release Notes) in ``/doc/release-notes`` and write the release notes, making sure to pull content from the release note snippets mentioned above. Snippets may not include any issue number or pull request number in the text so be sure copy the number from the filename of the snippet into the final release note.
 - Delete (``git rm``) the release note snippets as the content is added to the main release notes file.
 - Include instructions describing the steps required to upgrade the application from the previous version. These must be customized for release numbers and special circumstances such as changes to metadata blocks and infrastructure.
-- Take the release notes .md through the regular Code Review and QA process. That is, make a pull request.
+- Take the release notes .md through the regular Code Review and QA process. That is, make a pull request. Here's an example: https://github.com/IQSS/dataverse/pull/10866
 
-Create a GitHub Issue and Branch for the Release
-------------------------------------------------
+Upgrade Instructions for Internal
+---------------------------------
+
+To upgrade internal, go to /doc/release-notes, open the release-notes.md file for the current release and perform all the steps under "Upgrade Instructions".
+
+Deploy Release Candidate to Demo
+--------------------------------
+
+First, build the release candidate.
+
+ssh into the dataverse-internal server and undeploy the current war file.
+
+Go to https://jenkins.dataverse.org/job/IQSS_Dataverse_Internal/ and make the following adjustments to the config:
+
+- Repository URL: ``https://github.com/IQSS/dataverse.git``
+- Branch Specifier (blank for 'any'): ``*/develop``
+- Execute shell: Update version in filenames to ``dataverse-5.10.war`` (for example)
+
+Click "Save" then "Build Now".
+
+This will build the war file, and then automatically deploy it on dataverse-internal. Verify that the application has deployed successfully. 
+
+You can scp the war file to the demo server or download it from https://jenkins.dataverse.org/job/IQSS_Dataverse_Internal/ws/target/
+
+ssh into the demo server and follow the upgrade instructions in the release notes.
+
+Prepare Release Branch
+----------------------
+
+The release branch will have the final changes such as bumping the version number.
 
 Usually we branch from the "develop" branch to create the release branch. If we are creating a hotfix for a particular version (5.11, for example), we branch from the tag (e.g. ``v5.11``).
 
-Use the GitHub issue number and the release tag for the name of the branch. (e.g. ``8583-update-version-to-v5.10.1``
+Create a release branch named after the issue that tracks bumping the version with a descriptive name like "10852-bump-to-6.4" from https://github.com/IQSS/dataverse/pull/10871.
 
 **Note:** the changes below must be the very last commits merged into the develop branch before it is merged into master and tagged for the release!
 
 Make the following changes in the release branch.
 
-Bump Version Numbers and Prepare Container Tags
------------------------------------------------
-
 Increment the version number to the milestone (e.g. 5.10.1) in the following two files:
 
 - modules/dataverse-parent/pom.xml -> ``<properties>`` -> ``<revision>`` (e.g. `pom.xml commit <https://github.com/IQSS/dataverse/commit/3943aa0>`_)
@@ -58,14 +116,11 @@ Return to the parent pom and make the following change, which is necessary for p
 
 (Before you make this change the value should be ``${parsedVersion.majorVersion}.${parsedVersion.nextMinorVersion}``. Later on, after cutting a release, we'll change it back to that value.)
 
-Check in the Changes Above into a Release Branch and Merge It
--------------------------------------------------------------
-
 For a regular release, make the changes above in the release branch you created, make a pull request, and merge it into the "develop" branch. Like usual, you can safely delete the branch after the merge is complete.
 
 If you are making a hotfix release, make the pull request against the "master" branch. Do not delete the branch after merging because we will later merge it into the "develop" branch to pick up the hotfix. More on this later.
 
-Either way, as usual, you should ensure that all tests are passing. Please note that you will need to bump the version in `jenkins.yml <https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible/blob/develop/tests/group_vars/jenkins.yml>`_ in dataverse-ansible to get the tests to pass. Consider doing this before making the pull request. Alternatively, you can bump jenkins.yml after making the pull request and re-run the Jenkins job to make sure tests pass.
+Either way, as usual, you should ensure that all tests are passing. Please note that you will need to bump the version in `jenkins.yml <https://github.com/gdcc/dataverse-ansible/blob/develop/tests/group_vars/jenkins.yml>`_ in dataverse-ansible to get the tests to pass. Consider doing this before making the pull request. Alternatively, you can bump jenkins.yml after making the pull request and re-run the Jenkins job to make sure tests pass.
 
 Merge "develop" into "master"
 -----------------------------
@@ -94,7 +149,7 @@ After the "master" branch has been updated and the GitHub Action to build and pu
 
 To test these images against our API test suite, go to the "alpha" workflow at https://github.com/gdcc/api-test-runner/actions/workflows/alpha.yml and run it.
 
-If there are failures, additional dependencies or settings may have been added to the "develop" workflow. Copy them over and try again.
+Don't be surprised if there are failures. The test runner is a work in progress! Additional dependencies or settings may have been added to the "develop" workflow. Copy them over and try again.
 
 .. _build-guides:
 
@@ -186,11 +241,6 @@ Upload the following artifacts to the draft release you created:
   - metadata block tsv files
   - config files
 
-Deploy on Demo
---------------
-
-Now that you have the release ready to go, consider giving it one final test by deploying it on https://demo.dataverse.org. Note that this is also an opportunity to re-test the upgrade checklist as described in the release note.
-
 Publish the Release
 -------------------
 
@@ -228,7 +278,21 @@ Create a new branch (any name is fine but ``prepare-next-iteration`` is suggeste
 
 Now create a pull request and merge it.
 
-For more background, see :ref:`base-supported-image-tags`.
+For more background, see :ref:`base-supported-image-tags`. For an example, see https://github.com/IQSS/dataverse/pull/10896
+
+Deploy Final Release on Demo
+----------------------------
+
+Above you already did the hard work of deploying a release candidate to https://demo.dataverse.org. It should be relatively straightforward to undeploy the release candidate and deploy the final release.
+
+Update SchemaSpy
+----------------
+
+We maintain SchemaSpy at URLs like https://guides.dataverse.org/en/6.3/schemaspy/index.html
+
+Get the attention of the core team and ask someone to update it for the new release.
+
+Consider updating `the thread <https://groups.google.com/g/dataverse-community/c/f95DQU-wlVM/m/cvUp3E9OBgAJ>`_ on the mailing list once the update is in place.
 
 Add the Release to the Dataverse Roadmap
 ----------------------------------------

doc/sphinx-guides/source/installation/config.rst

@@ -236,6 +236,10 @@ Dataverse automatically manages assigning PIDs and making them findable when dat
 allow updating the PID target URLs and metadata of already-published datasets manually if needed <send-metadata-to-pid-provider>`, e.g. if a Dataverse instance is
 moved to a new URL or when the software is updated to generate additional metadata or address schema changes at the PID service.
 
+Note that while some forms of PIDs (Handles, PermaLinks) are technically case sensitive, common practice is to avoid creating PIDs that differ only by case.
+Dataverse treats PIDs of all types as case-insensitive (as DOIs are by definition). This means that Dataverse will find datasets (in search, to display dataset pages, etc.) 
+when the PIDs entered do not match the case of the original but will have a problem if two PIDs that differ only by case exist in one instance.
+
 Testing PID Providers
 +++++++++++++++++++++
 

docker/compose/demo/compose.yml

@@ -9,7 +9,7 @@ services:
     restart: on-failure
     user: payara
     environment:
-      DATAVERSE_SITEURL: "https://demo.example.org"
+      DATAVERSE_SITEURL: "http://${MACHINE_IP:-localhost}:8080"
       DATAVERSE_DB_HOST: postgres
       DATAVERSE_DB_PASSWORD: secret
       DATAVERSE_DB_USER: dataverse
@@ -74,6 +74,36 @@ services:
     volumes:
       - ./data/app/data:/dv
 
+  previewers-provider:
+    image: trivadis/dataverse-previewers-provider:latest
+    container_name: previewers-provider
+    hostname: previewers-provider
+    ports:
+      - 9000:9000
+    networks:
+      - dataverse      
+    environment:
+      - NGINX_HTTP_PORT=9000
+      - PREVIEWERS_PROVIDER_URL=http://${MACHINE_IP:-localhost}:9000
+      - VERSIONS="v1.4,betatest"
+
+  register-previewers:
+    container_name: register-previewers
+    hostname: register-previewers
+    image: trivadis/dataverse-deploy-previewers:latest
+    networks:
+      - dataverse    
+    environment:
+      - DATAVERSE_URL=http://dataverse:8080
+      - TIMEOUT=10m
+      - PREVIEWERS_PROVIDER_URL=http://${MACHINE_IP:-localhost}:9000
+      - INCLUDE_PREVIEWERS=text,html,pdf,csv,comma-separated-values,tsv,tab-separated-values,jpeg,png,gif,markdown,x-markdown
+      - EXCLUDE_PREVIEWERS=
+      - REMOVE_EXISTING=True
+    command:
+      - deploy
+    restart: no
+
   postgres:
     container_name: "postgres"
     hostname: postgres

src/main/java/edu/harvard/iq/dataverse/DvObject.java

@@ -27,17 +27,17 @@
     @NamedQuery(name = "DvObject.ownedObjectsById",
 			query="SELECT COUNT(obj) FROM DvObject obj WHERE obj.owner.id=:id"),
     @NamedQuery(name = "DvObject.findByGlobalId",
-            query = "SELECT o FROM DvObject o WHERE o.identifier=:identifier and o.authority=:authority and o.protocol=:protocol and o.dtype=:dtype"),
+            query = "SELECT o FROM DvObject o WHERE UPPER(o.identifier)=UPPER(:identifier) and o.authority=:authority and o.protocol=:protocol and o.dtype=:dtype"),
     @NamedQuery(name = "DvObject.findIdByGlobalId",
-            query = "SELECT o.id FROM DvObject o WHERE o.identifier=:identifier and o.authority=:authority and o.protocol=:protocol and o.dtype=:dtype"),
+            query = "SELECT o.id FROM DvObject o WHERE UPPER(o.identifier)=UPPER(:identifier) and o.authority=:authority and o.protocol=:protocol and o.dtype=:dtype"),
 
     @NamedQuery(name = "DvObject.findByAlternativeGlobalId",
             query = "SELECT o FROM DvObject o, AlternativePersistentIdentifier a  WHERE o.id = a.dvObject.id and a.identifier=:identifier and a.authority=:authority and a.protocol=:protocol and o.dtype=:dtype"),
     @NamedQuery(name = "DvObject.findIdByAlternativeGlobalId",
             query = "SELECT o.id FROM DvObject o, AlternativePersistentIdentifier a  WHERE o.id = a.dvObject.id and a.identifier=:identifier and a.authority=:authority and a.protocol=:protocol and o.dtype=:dtype"),
 
     @NamedQuery(name = "DvObject.findByProtocolIdentifierAuthority",
-            query = "SELECT o FROM DvObject o WHERE o.identifier=:identifier and o.authority=:authority and o.protocol=:protocol"),
+            query = "SELECT o FROM DvObject o WHERE UPPER(o.identifier)=UPPER(:identifier) and o.authority=:authority and o.protocol=:protocol"),
     @NamedQuery(name = "DvObject.findByOwnerId", 
                 query = "SELECT o FROM DvObject o WHERE o.owner.id=:ownerId  order by o.dtype desc, o.id"),
     @NamedQuery(name = "DvObject.findByAuthenticatedUserId", 
@@ -53,7 +53,8 @@
 @Table(indexes = {@Index(columnList="dtype")
 		, @Index(columnList="owner_id")
 		, @Index(columnList="creator_id")
-		, @Index(columnList="releaseuser_id")},
+		, @Index(columnList="releaseuser_id")
+        , @Index(columnList="authority,protocol, UPPER(identifier)", name="INDEX_DVOBJECT_authority_protocol_upper_identifier")},
 		uniqueConstraints = {@UniqueConstraint(columnNames = {"authority,protocol,identifier"}),@UniqueConstraint(columnNames = {"owner_id,storageidentifier"})})
 public abstract class DvObject extends DataverseEntity implements java.io.Serializable {
 

src/main/java/edu/harvard/iq/dataverse/api/imports/ImportServiceBean.java

@@ -313,7 +313,7 @@ public Dataset doImportHarvestedDataset(DataverseRequest dataverseRequest, Harve
                 // Creating a new dataset from scratch:
 
                 harvestedDataset = parser.parseDataset(obj);
-                
+
                 harvestedDataset.setHarvestedFrom(harvestingClient);
                 harvestedDataset.setHarvestIdentifier(harvestIdentifier);
 

src/main/java/edu/harvard/iq/dataverse/export/DDIExportServiceBean.java

@@ -98,8 +98,10 @@ public class DDIExportServiceBean {
     public static final String LEVEL_FILE = "file";
     public static final String NOTE_TYPE_UNF = "VDC:UNF";
     public static final String NOTE_TYPE_TAG = "DATAVERSE:TAG";
+    public static final String NOTE_TYPE_FILEDESCRIPTION = "DATAVERSE:FILEDESC";
     public static final String NOTE_SUBJECT_UNF = "Universal Numeric Fingerprint";
     public static final String NOTE_SUBJECT_TAG = "Data File Tag";
+    public static final String NOTE_SUBJECT_FILEDESCRIPTION = "DataFile Description"; 
 
     /*
      * Internal service objects:
@@ -742,11 +744,6 @@ private void createFileDscr(XMLStreamWriter xmlw, Set<String> excludedFieldSet,
                 xmlw.writeEndElement(); // fileName
             }
 
-            /*
-             xmlw.writeStartElement("fileCont");
-             xmlw.writeCharacters( df.getContentType() );
-             xmlw.writeEndElement(); // fileCont
-             */
             // dimensions
             if (checkField("dimensns", excludedFieldSet, includedFieldSet)) {
                 if (dt.getCaseQuantity() != null || dt.getVarQuantity() != null || dt.getRecordsPerCase() != null) {
@@ -801,26 +798,6 @@ private void createFileDscr(XMLStreamWriter xmlw, Set<String> excludedFieldSet,
             xmlw.writeEndElement(); // notes
         }
 
-        /*
-         xmlw.writeStartElement("notes");
-         writeAttribute( xmlw, "type", "vdc:category" );
-         xmlw.writeCharacters( fm.getCategory() );
-         xmlw.writeEndElement(); // notes
-         */
-        // A special note for LOCKSS crawlers indicating the restricted
-        // status of the file:
-
-        /*
-         if (tdf != null && isRestrictedFile(tdf)) {
-         xmlw.writeStartElement("notes");
-         writeAttribute( xmlw, "type", NOTE_TYPE_LOCKSS_CRAWL );
-         writeAttribute( xmlw, "level", LEVEL_FILE );
-         writeAttribute( xmlw, "subject", NOTE_SUBJECT_LOCKSS_PERM );
-         xmlw.writeCharacters( "restricted" );
-         xmlw.writeEndElement(); // notes
-
-         }
-         */
         if (checkField("tags", excludedFieldSet, includedFieldSet) && df.getTags() != null) {
             for (int i = 0; i < df.getTags().size(); i++) {
                 xmlw.writeStartElement("notes");
@@ -831,6 +808,17 @@ private void createFileDscr(XMLStreamWriter xmlw, Set<String> excludedFieldSet,
                 xmlw.writeEndElement(); // notes
             }
         }
+
+        // A dedicated node for the Description entry 
+        if (!StringUtilisEmpty(fm.getDescription())) {
+            xmlw.writeStartElement("notes");
+            xmlw.writeAttribute("level", LEVEL_FILE);
+            xmlw.writeAttribute("type", NOTE_TYPE_FILEDESCRIPTION);
+            xmlw.writeAttribute("subject", NOTE_SUBJECT_FILEDESCRIPTION);
+            xmlw.writeCharacters(fm.getDescription());
+            xmlw.writeEndElement(); // notes
+        }
+
         xmlw.writeEndElement(); // fileDscr   
     }