Merge branch 'develop' into 6505-optimize-zip-downloads

(fixed merge conflicts w/develop - mostly the POST handling added for the /api/access/datafiles/ API)

.gitignore

@@ -60,3 +60,8 @@ scripts/installer/default.config
 tests/node_modules
 tests/package-lock.json
 venv
+
+# from thumbnail tests in SearchIT
+scripts/search/data/binary/trees.png.thumb140
+src/main/webapp/resources/images/cc0.png.thumb140
+src/main/webapp/resources/images/dataverseproject.png.thumb140

doc/release-notes/4977-facet-sort-setting

@@ -0,0 +1,5 @@
+New setting for Dataverse Admins:
+
+The new :ChronologicalDateFacets setting. Default to true. 
+
+Facets with Date/Year are sorted chronologically by default, with the most recent value first. To have them sorted by number of hits, e.g. with the year with the most results first, set this to false 

doc/release-notes/5093-datacite.md

@@ -0,0 +1 @@
+If you are using DataCite as your DOI provider you must add a new JVM option called "doi.baseurlstringnext" with a value of "https://api.datacite.org" for production environments and "https://api.test.datacite.org" for test environments. More information about this JVM option can be found in the Installation Guide.

doc/release-notes/6684-buttons

@@ -0,0 +1 @@
+Note about how the buttons are the next steps towards the dataset and file redesign, mention previews on page, filtering/sorting options, tree view.

doc/release-notes/6895-payara-release-note

@@ -0,0 +1,5 @@
+##Payara 5
+
+A major upgrade of the application server will provide security updates, access to new features like MicroProfile Config API, and will enable upgrades to other core technologies.
+
+Note that moving from Glassfish to Payara will be required as part of the move to Dataverse 5.

doc/release-notes/6956-shib-attribute-character-set-conversion.md

@@ -0,0 +1,6 @@
+# Shibboleth attribute character set conversion
+
+By default, all attributes received from Shibboleth are converted from ISO-8859-1 to UTF-8.
+You can disable this behaviour by setting ShibAttributeCharacterSetConversionEnabled to false.
+
+For further documentation on this issue, see: doc/sphinx-guides/source/installation/config.rst

doc/release-notes/6959-html-codebook.md

@@ -0,0 +1 @@
+A small correction has been made to the stylesheet that generates the DDI HTML Codebook metadata export to remove a hard-coded reference to ICPSR. In order for previously published and exported datasets to reflect this correction, run ReExportAll as part of the upgrade process with `curl http://localhost:8080/api/admin/metadata/reExportAll`.

doc/release-notes/6970-ip-addresses-behind-proxy

@@ -0,0 +1,6 @@
+### Tracking users' IP addresses behind an address-masking proxy
+
+It is now possible to collect real user IP addresses in MDC logs 
+and/or set up an IP group on a system running behind a proxy/load balancer 
+that hides the addresses of incoming requests. See "Recording User IP 
+Addresses" in the Configuration section of the Installation Guide.

doc/sphinx-guides/source/admin/dataverses-datasets.rst

@@ -47,17 +47,17 @@ To direct new files (uploaded when datasets are created or edited) for all datas
  
     curl -H "X-Dataverse-key: $API_TOKEN" -X PUT -d $storageDriverLabel http://$SERVER/api/admin/dataverse/$dataverse-alias/storageDriver
     
-The current driver can be seen using:
+The current driver can be seen using::
 
     curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/$dataverse-alias/storageDriver
 
-and can be reset to the default store with:
+and can be reset to the default store with::
 
     curl -H "X-Dataverse-key: $API_TOKEN" -X DELETE http://$SERVER/api/admin/dataverse/$dataverse-alias/storageDriver
     
-The available drivers can be listed with:
+The available drivers can be listed with::
 
-    curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/storageDrivers
+    curl -H "X-Dataverse-key: $API_TOKEN" http://$SERVER/api/admin/dataverse/storageDrivers
     
 
 Datasets
@@ -115,6 +115,11 @@ Forces update to metadata provided to the PID provider of a published dataset. O
 
     curl -H "X-Dataverse-key: $API_TOKEN" -X POST http://$SERVER/api/datasets/$dataset-id/modifyRegistrationMetadata
 
+Check for Unreserved PIDs and Reserve Them
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+See :ref:`pids-api` in the API Guide for details.
+
 Make Metadata Updates Without Changing Dataset Version
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

doc/sphinx-guides/source/api/dataaccess.rst

@@ -58,15 +58,17 @@ the following parameter values are supported (for image and pdf files only):
 ==============  ===========
 Value           Description
 ==============  ===========
-true            Generates a thumbnail image, by rescaling to the default thumbnail size (64 pixels)
-``N``           Rescales the image to ``N`` pixels.
+true            Generates a thumbnail image by rescaling to the default thumbnail size (64 pixels wide).
+``N``           Rescales the image to ``N`` pixels wide. ``imageThumb=true`` and ``imageThumb=64`` are equivalent.
 ==============  ===========
 
 Multiple File ("bundle") download
 ---------------------------------
 
 ``/api/access/datafiles/$id1,$id2,...$idN``
 
+Alternate Form: POST to ``/api/access/datafiles`` with a ``fileIds`` input field containing the same comma separated list of file ids. This is most useful when your list of files surpasses the allowed URL length (varies but can be ~2000 characters).  
+
 Returns the files listed, zipped. 
 
 .. note:: If the request can only be completed partially - if only *some* of the requested files can be served (because of the permissions and/or size restrictions), the file MANIFEST.TXT included in the zipped bundle will have entries specifying the reasons the missing files could not be downloaded. IN THE FUTURE the API will return a 207 status code to indicate that the result was a partial success. (As of writing this - v.4.11 - this hasn't been implemented yet)

doc/sphinx-guides/source/api/native-api.rst

@@ -2519,6 +2519,98 @@ Each user can get a dump of their notifications by passing in their API token::
 
     curl -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/notifications/all
 
+.. _pids-api:
+
+PIDs
+----
+
+PIDs is short for Persistent IDentifiers. Examples include DOI or Handle. There are some additional PID operations listed in the :doc:`/admin/dataverses-datasets` section of the Admin Guide.
+
+Get Info on a PID
+~~~~~~~~~~~~~~~~~
+
+Get information on a PID, especially its "state" such as "draft" or "findable". Currently, this API only works on DataCite DOIs. A superuser API token is required.
+
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export PID=doi:10.70122/FK2/9BXT5O
+
+  curl -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/pids?persistentId=$PID
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx https://demo.dataverse.org/api/pids?persistentId=doi:10.70122/FK2/9BXT5O
+
+List Unreserved PIDs
+~~~~~~~~~~~~~~~~~~~~
+
+Get a list of PIDs that have not been reserved on the PID provider side. This can happen, for example, if a dataset is created while the PID provider is down. A superuser API token is required.
+
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+
+  curl -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/pids/unreserved
+
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx https://demo.dataverse.org/api/pids/unreserved
+
+Reserve a PID
+~~~~~~~~~~~~~
+
+Reserved a PID for a dataset. A superuser API token is required.
+
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export PID=doi:10.70122/FK2/9BXT5O
+
+  curl -H "X-Dataverse-key:$API_TOKEN" -X POST $SERVER_URL/api/pids/:persistentId/reserve?persistentId=$PID
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST https://demo.dataverse.org/api/pids/:persistentId/reserve?persistentId=doi:10.70122/FK2/9BXT5O
+
+Delete a PID
+~~~~~~~~~~~~
+
+Delete PID from DataCite (this is only possible for PIDs that are in the "draft" state) and within Dataverse, set ``globalidcreatetime`` to null and ``identifierregistered`` to false. A superuser API token is required.
+
+.. note:: See :ref:`curl-examples-and-environment-variables` if you are unfamiliar with the use of export below.
+
+.. code-block:: bash
+
+  export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+  export SERVER_URL=https://demo.dataverse.org
+  export PID=doi:10.70122/FK2/9BXT5O
+
+  curl -H "X-Dataverse-key:$API_TOKEN" -X DELETE $SERVER_URL/api/pids/:persistentId/delete?persistentId=$PID
+
+The fully expanded example above (without environment variables) looks like this:
+
+.. code-block:: bash
+
+  curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X DELETE https://demo.dataverse.org/api/pids/:persistentId/delete?persistentId=doi:10.70122/FK2/9BXT5O
+
+
 .. _admin:
 
 Admin