Merge branch 'develop' into sec38-div-issues

conf/docker-aio/0prep_deps.sh

@@ -4,10 +4,10 @@ if [ ! -d dv/deps ]; then
 fi
 wdir=`pwd`
 
-if [ ! -e dv/deps/payara-5.2021.4.zip ]; then
+if [ ! -e dv/deps/payara-5.2021.5.zip ]; then
 	echo "payara dependency prep"
 	# no more fiddly patching :)
-	wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.4/payara-5.2021.4.zip  -O dv/deps/payara-5.2021.4.zip
+	wget https://s3-eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/5.2021.5/payara-5.2021.5.zip  -O dv/deps/payara-5.2021.5.zip
 fi
 
 if [ ! -e dv/deps/solr-8.8.1dv.tgz ]; then

conf/docker-aio/c8.dockerfile

@@ -24,7 +24,7 @@ COPY disableipv6.conf /etc/sysctl.d/
 RUN rm /etc/httpd/conf/*
 COPY httpd.conf /etc/httpd/conf 
 RUN cd /opt ; tar zxf /tmp/dv/deps/solr-8.8.1dv.tgz 
-RUN cd /opt ; unzip /tmp/dv/deps/payara-5.2021.4.zip ; ln -s /opt/payara5 /opt/glassfish4
+RUN cd /opt ; unzip /tmp/dv/deps/payara-5.2021.5.zip ; ln -s /opt/payara5 /opt/glassfish4
 
 # this copy of domain.xml is the result of running `asadmin set server.monitoring-service.module-monitoring-levels.jvm=LOW` on a default glassfish installation (aka - enable the glassfish REST monitir endpoint for the jvm`
 # this dies under Java 11, do we keep it?

doc/release-notes/5.4-release-notes.md

@@ -274,6 +274,8 @@ Add the following JVM options to the `<config name="server-config"><java-config>
 
 `<jvm-options>[9|]--add-opens=java.logging/java.util.logging=ALL-UNNAMED</jvm-options>`  
 
+`<jvm-options>[1.8.0u251|]-Xbootclasspath/a:${com.sun.aas.installRoot}/lib/grizzly-npn-api.jar</jvm-options>`
+
 6\. Start Payara
 
 - `service payara start`

doc/release-notes/5.6-release-notes.md

@@ -0,0 +1,136 @@
+# Dataverse Software 5.6
+
+This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+### Anonymized Access in Support of Double Blind Review
+
+Dataverse installations can select whether or not to allow users to create anonymized private URLs and can control which specific identifying fields are anonymized. If this is enabled, users can create private URLs that do not expose identifying information about dataset depositors, allowing for double blind reviews of datasets in the Dataverse installation.
+
+### Guestbook Responses API
+
+A new API to retrieve Guestbook responses has been added. This makes it easier to retrieve the records for large guestbooks and also makes it easier to integrate with external systems.
+
+### Dataset Semantic API (Experimental)
+
+Dataset metadata can be retrieved, set, and updated using a new, flatter JSON-LD format - following the format of an OAI-ORE export (RDA-conformant Bags), allowing for easier transfer of metadata to/from other systems (i.e. without needing to know Dataverse's metadata block and field storage architecture). This new API also allows for the update of terms metadata (#5899).
+
+This development was supported by the [Research Data Alliance](https://rd-alliance.org), DANS, and Sciences PO and follows the recommendations from the [Research Data Repository Interoperability Working Group](http://dx.doi.org/10.15497/RDA00025).
+
+### Dataset Migration API (Experimental)
+
+Datasets can now be imported following the format of an OAI-ORE export (RDA-conformant Bags), allowing for easier migration from one Dataverse installation to another, and migration from other systems. This experimental, superuser only, endpoint also allows keeping the existing persistent identifier (where the authority and shoulder match those for which the software is configured) and publication dates.
+
+This development was supported by DANS and the [Research Data Alliance](https://rd-alliance.org) and follows the recommendations from the [Research Data Repository Interoperability Working Group](http://dx.doi.org/10.15497/RDA00025).
+
+### Direct Upload API Now Available for Adding Multiple Files Metadata to the Dataset
+
+Users can now add metadata of multiple files to the dataset once the files exists in the s3 bucket using the direct upload API. This makes direct uploads more efficient and reduces server load by only updating the dataset once instead of once per file. For more information, see the [Direct DataFile Upload/Replace API section](https://guides.dataverse.org/en/5.6/developers/s3-direct-upload-api.html) of the Dataverse Software Guides.
+
+## Major Use Cases
+
+Newly-supported major use cases in this release include:
+
+- Users can create Private URLs that anonymize dataset metadata, allowing for double blind peer review. (Issue #1724, PR #7908)
+- Users can download Guestbook records using a new API. (Issue #7767, PR #7931)
+- Users can update terms metadata using the new semantic API. (Issue #5899, PR #7414)
+- Users can retrieve, set, and update metadata using a new, flatter JSON-LD format. (Issue #6497, PR #7414)
+- Administrators can use the Traces API to retreive information about specific types of user activity  (Issue #7952, PR #7953)
+
+## Notes for Dataverse Installation Administrators
+
+### New Database Constraint
+
+A new DB Constraint has been added in this release. Full instructions on how to identify whether or not your database needs any cleanup before the upgrade can be found in the [Dataverse software GitHub repository](https://github.com/IQSS/dataverse/blob/develop/scripts/issues/7451/PRE-RELEASE-INFO.txt). This information was also emailed out to Dataverse installation contacts.
+
+### Payara 5.2021.5 (or Higher) Required
+
+Some changes in this release require an upgrade to Payara 5.2021.5 or higher. (See the upgrade section).
+
+Instructions on how to update can be found in the [Payara documentation](https://docs.payara.fish/community/docs/5.2021.4/documentation/user-guides/upgrade-payara.html) We've included the necessary steps below, but we recommend that you review the Payara upgrade instructions as it could be helpful during any troubleshooting.
+
+Installations upgrading from a previous Payara version shouldn't encounter a logging configuration bug in Payara-5.2021.5, but if your server.log fills with repeated notes about logging configuration and WELD complaints about loading beans, see the paragraph on `logging.properties` in the [installation guide](https://guides.dataverse.org/en/5.6/installation/installation-main.html#running-the-dataverse-software-installer)
+
+### Enhancement to DDI Metadata Exports
+
+To increase support for internationalization and to improve compliance with CESSDA requirements, DDI exports now have a holdings element with a URI attribute whose value is the URL form of the dataset PID.
+
+## New JVM Options and DB Settings
+
+:AnonymizedFieldTypeNames can be used to enable creation of anonymized Private URLs and to specify which fields will be anonymized.
+
+## Notes for Tool Developers and Integrators
+
+### Semantic API
+
+The new Semantic API is especially helpful in data migrations and getting metadata into a Dataverse installation. Learn more in the [Developers Guide](https://guides.dataverse.org/en/5.6/developers/).
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the [5.6 Milestone](https://github.com/IQSS/dataverse/milestone/97?closed=1) in Github.
+
+For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org.
+
+## Installation
+
+If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.6/installation/).
+
+## Upgrade Instructions
+
+0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.6.
+
+The steps below include a required upgrade to Payara 5.2021.5 or higher. (It is a simple matter of reusing your existing domain directory with the new distribution). But we also recommend that you review the Payara upgrade instructions as it could be helpful during any troubleshooting: [Payara documentation](https://docs.payara.fish/community/docs/5.2021.5/documentation/user-guides/upgrade-payara.html)
+
+If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user.  
+
+In the following commands we assume that Payara 5 is installed in `/usr/local/payara5`. If not, adjust as needed.
+
+`export PAYARA=/usr/local/payara5`
+
+(or `setenv PAYARA /usr/local/payara5` if you are using a `csh`-like shell)
+
+1\. Undeploy the previous version 
+
+- `$PAYARA/bin/asadmin list-applications`
+- `$PAYARA/bin/asadmin undeploy dataverse<-version>`
+
+2\. Stop Payara
+
+- `service payara stop`
+- `rm -rf $PAYARA/glassfish/domains/domain1/generated`
+
+3\. Move the current Payara directory out of the way
+
+- `mv $PAYARA $PAYARA.MOVED`
+
+4\. Download the new Payara version (5.2021.5+), and unzip it in its place
+
+5\. Replace the brand new payara/glassfish/domains/domain1 with your old, preserved `domain1`
+
+6\. In domain.xml, add/confirm the existence of the following JVM option to/in the `<config name="server-config"><java-config>` section:
+
+`<jvm-options>[1.8.0u251|]-Xbootclasspath/a:${com.sun.aas.installRoot}/lib/grizzly-npn-api.jar</jvm-options>`
+
+7\. Start Payara
+
+- `service payara start`
+
+8\. Deploy this version.
+
+- `$PAYARA/bin/asadmin deploy dataverse-5.6.war`
+
+9\. Restart payara
+
+- `service payara stop`
+- `service payara start`
+
+## Additional Release Steps
+
+**If your installation relies on the database-side stored procedure for generating sequential numeric identifiers:**
+
+Note that you can skip this step if your installation uses the default-style, randomly-generated six alphanumeric character-long identifiers for your datasets! This is the case with most Dataverse installations.
+
+The underlying database framework has been modified in this release, to make it easier for installations  to create custom procedures for generating identifier strings that suit their needs. Your current configuration will be automatically updated by the database upgrade (Flyway) script incorporated in the release. No manual configuration changes should be necessary. However, after the upgrade, we recommend that you confirm that your installation can still create new datasets, and that they are still assigned sequential numeric identifiers. In the unlikely chance that this is no longer working, please re-create the stored procedure following the steps described in the documentation for the `:IdentifierGenerationStyle` setting in the *Configuration* section of the Installation Guide for this release (v5.6).
+
+(Running the script supplied there will NOT overwrite the position on the sequence you are currently using!)

doc/release-notes/7786-ddi_changes.md

@@ -1,9 +1,9 @@
-Note: These notes cover several related PRs (#7984, #7958, #7959 respectively for the three bullets below.) If some are not merged before the next release, these notes should be adjusted.
+Note: These notes cover several related PRs (#7958, #7959 respectively for the three bullets below.) If some are not merged before the next release, these notes should be adjusted.
 
 ### Enhancements to DDI Metadata Exports
 
 Several changes have been made to the DDI exports to improve support for internationalization and to improve compliance with CESSDA requirements. These changes include:
-* Addition of a holdings element with a URI attribute whose value is the URL form of the dataset PID
+
 * Addition of xml:lang attributes specifying the dataset metadata language at the document level and for individual elements such as title and description
 * Specification of controlled vocabulary terms in duplicate elements in multiple languages (in the installation default langauge and, if different, the dataset metadata language)
 

doc/sphinx-guides/source/_static/api/dataset-migrate.jsonld

@@ -0,0 +1,43 @@
+{
+"citation:Depositor": "Admin, Dataverse",
+"Title": "Test Dataset",
+"Subject": "Computer and Information Science",
+"Creator": {
+  "author:Name": "Admin, Dataverse",
+  "author:Affiliation": "GDCC"
+},
+"Deposit Date": "2020-10-08",
+"citation:Distributor": {
+  "distributor:Name": "Demo Dataverse Repository",
+  "distributor:Affiliation": "Dataverse Community",
+  "distributor:Abbreviation": "GDCC",
+  "distributor:URL": "https://dataverse.org/global-dataverse-community-consortium"
+},
+"citation:Contact": {
+"datasetContact:Name": "Admin, Dataverse",
+"datasetContact:Affiliation": "GDCC",
+"datasetContact:E-mail": "admin@demo.dataverse.org"
+},
+"citation:Description": {
+  "dsDescription:Text": "A short description"
+},
+"@id": "doi:10.33564/FK27U7YBV",
+"schema:version": "1.0",
+"schema:license": "https://creativecommons.org/publicdomain/zero/1.0/",
+"schema:datePublished": "2021-07-21",
+"dvcore:fileTermsOfAccess": {
+"dvcore:fileRequestAccess": false
+},
+"@context": {
+  "Creator": "http://purl.org/dc/terms/creator",
+  "Deposit Date": "http://purl.org/dc/terms/dateSubmitted",
+  "Subject": "http://purl.org/dc/terms/subject",
+  "Title": "http://purl.org/dc/terms/title",
+  "author": "https://dataverse.org/schema/citation/author#",
+  "citation": "https://dataverse.org/schema/citation/",
+  "datasetContact": "https://dataverse.org/schema/citation/datasetContact#",
+  "distributor": "https://dataverse.org/schema/citation/distributor#",
+  "dsDescription": "https://dataverse.org/schema/citation/dsDescription#",
+  "dvcore": "https://dataverse.org/schema/core#",
+  "schema": "http://schema.org/"
+}}