Merge pull request #6369 from IQSS/develop

v4.18

README.md

@@ -17,7 +17,9 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 
 [![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg?raw=true "Dataverse Project")](http://dataverse.org)
 
-[![Build Status](https://travis-ci.org/IQSS/dataverse.svg?branch=develop)](https://travis-ci.org/IQSS/dataverse) [![Coverage Status](https://coveralls.io/repos/IQSS/dataverse/badge.svg?branch=develop&service=github)](https://coveralls.io/github/IQSS/dataverse?branch=develop)
+[![API Test Status](https://jenkins.dataverse.org/buildStatus/icon?job=IQSS-dataverse-develop&subject=API%20Test%20Status)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/)
+[![Unit Test Status](https://img.shields.io/travis/IQSS/dataverse?label=Unit%20Test%20Status)](https://travis-ci.org/IQSS/dataverse)
+[![Unit Test Coverage](https://img.shields.io/coveralls/github/IQSS/dataverse?label=Unit%20Test%20Coverage)](https://coveralls.io/github/IQSS/dataverse?branch=develop)
 
 [dataverse.org]: https://dataverse.org
 [demo.dataverse.org]: https://demo.dataverse.org

conf/docker-aio/configure_doi.bash

@@ -17,3 +17,8 @@ if [ ! -z "${doi_baseurl}" ]; then
         doi_baseurl_esc=`echo ${doi_baseurl} | sed -e 's/:/\\:/'`
         bin/asadmin create-jvm-options "\"-Ddoi.baseurlstring=${doi_baseurl_esc}\""
 fi
+if [ ! -z "${doi_mdcbaseurl}" ]; then
+        bin/asadmin delete-jvm-options "-Ddoi.mdcbaseurlstring=https\://api.test.datacite.org"
+        doi_mdcbaseurl_esc=`echo ${doi_mdcbaseurl} | sed -e 's/:/\\:/'`
+        bin/asadmin create-jvm-options "\"-Ddoi.mdcbaseurlstring=${doi_mdcbaseurl_esc}\""
+fi

conf/docker-aio/run-test-suite.sh

@@ -8,4 +8,4 @@ fi
 
 # Please note the "dataverse.test.baseurl" is set to run for "all-in-one" Docker environment.
 # TODO: Rather than hard-coding the list of "IT" classes here, add a profile to pom.xml.
-mvn test -Dtest=DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT -Ddataverse.test.baseurl=$dvurl
+mvn test -Dtest=DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT,AccessIT -Ddataverse.test.baseurl=$dvurl

doc/release-notes/4.17-release-notes.md

@@ -72,7 +72,7 @@ Second, file level tools that did not previously define a "contentType" are now
 
 The Dataverse team has been reaching out to tool makers about these breaking changes and getting various tools working in the https://github.com/IQSS/dataverse-ansible repo. Thank you for your patience as the dust settles around the external tool framework.
 
-For more information, check out new <a href=http://guides.dataverse.org/en/4.17/api/external-tools.html/>Building External Tools</a> section of the API Guide.
+For more information, check out new <a href=http://guides.dataverse.org/en/4.17/api/external-tools.html>Building External Tools</a> section of the API Guide.
 
 ## Complete List of Changes
 

doc/release-notes/4.18-release-notes.md

@@ -0,0 +1,118 @@
+# Dataverse 4.18
+
+This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+### File Page Previews and Previewers
+
+File-level External Tools can now be configured to display in a "Preview Mode" designed for embedding within the file landing page.
+
+While not technically part of this release, previewers have been made available for several common file types. The previewers support for spreadsheet, image, text, document, audio, video, html files and more. These previewers can be found in the <a href="https://github.com/QualitativeDataRepository/dataverse-previewers">Qualitative Data Repository Github Repository</a>.
+
+### Microsoft Login
+
+Users can now create Dataverse accounts and login using self-provisioned Microsoft accounts such as live.com and outlook.com. Users can also use Microsoft accounts managed by their institutions. This new feature not only makes it easier to log in to Dataverse but will also streamline the interaction between any external tools that utilize Azure services that require login.
+
+### Add Data and Host Dataverse
+
+More workflows to add data have been added across the UI, including a new button on the My Data tab of the Account page, as well as a link in the Dataverse navbar, which will display on every page. This will provider users much easier access to start depositing data. By default, the Host Dataverse will be the installation root dataverse for these new Add Data workflows, but there is now a dropdown component allowing creators to select a dataverse you have proper permissions to create a new dataverse or dataset in.
+
+### Primefaces 7
+
+Primefaces, the open source UI framework upon which the Dataverse front end is built, has been updated to the most recent version. This provides security updates and bug fixes and will also allow Dataverse developers to take advantage of new features and enhancements.
+
+### Integration Test Pipeline and Test Health Reporting
+
+As part of the Dataverse Community's ongoing efforts to provide more robust automated testing infrastructure, and in support of the project's desire to have the develop branch constantly in a "release ready" state, API-based integration tests are now run every time a branch is merged to develop. The status of the last test run is available as a badge at the bottom of the README.md file that serves as the homepage of Dataverse Github Repository.
+
+### Make Data Count Metrics Updates
+
+A new configuration option has been added that allows Make Data Count metrics to be collected, but not reflected in the front end. This option was designed to allow installations to collect and verify metrics for a period before turning on the display to users.
+
+### Search API Enhancements
+
+The Dataverse Search API will now display unpublished content when an API token is passed (and appropriate permissions exist).
+
+### Additional Dataset Author Identifiers
+
+The following dataset author identifiers are now supported:
+
+- DAI: https://en.wikipedia.org/wiki/Digital_Author_Identifier
+- ResearcherID: http://researcherid.com
+- ScopusID: https://www.scopus.com
+
+## Major Use Cases
+
+Newly-supported use cases in this release include:
+
+- Users can view previews of several common file types, eliminating the need to download or explore a file just to get a quick look.
+- Users can log in using self-provisioned Microsoft accounts and also can log in using Microsoft accounts managed by an organization.
+- Dataverse administrators can now revoke and regenerate API tokens with an API call.
+- Users will receive notifications when their ingests complete, and will be informed if the ingest was a success or failure.
+- Dataverse developers will receive feedback about the health of the develop branch after their pull request was merged.
+- Dataverse tool developers will be able to query the Dataverse API for unpublished data as well as published data.
+- Dataverse administrators will be able to collect Make Data Count metrics without turning on the display for users.
+- Users with a DAI, ResearcherID, or ScopusID and use these author identifiers in their datasets.
+
+## Notes for Dataverse Installation Administrators
+
+### API Token Management
+
+- You can now delete a user's API token, recreate a user's API token, and find a token's expiration date. See the <a href="http://guides.dataverse.org/en/4.18/api/native-api.html">Native API guide</a> for more information.
+
+### New JVM Options
+
+[:mdcbaseurlstring](http://guides.dataverse.org/en/4.18/installation/config.html#mdcbaseurlstring) allows dataverse administrators to use a test base URL for Make Data Count.
+
+### New Database Settings
+
+[:DisplayMDCMetrics](http://guides.dataverse.org/en/4.18/installation/config.html#DisplayMDCMetrics) can be set to false to disable display of MDC metrics.
+
+## Notes for Tool Developers and Integrators
+
+### Preview Mode
+
+Tool Developers can now add the `hasPreviewMode` parameter to their file level external tools. This setting provides an embedded, simplified view of the tool on the file pages for any installation that installs the tool. See <a href="http://guides.dataverse.org/en/4.18/api/external-tools.html">Building External Tools</a> for more information.
+
+### API Token Management
+
+If your tool writes content back to Dataverse, you can now take advantage of administrative endpoints that delete and re-create API tokens. You can also use an endpoint that provides the expiration date of a specific API token. See the <a href="http://guides.dataverse.org/en/4.18/api/native-api.html">Native API guide</a> for more information.
+
+### View Unpublished Data Using Search API
+
+If you pass a token, the search API output will include unpublished content.
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the <a href="https://github.com/IQSS/dataverse/milestone/85?closed=1">4.18 milestone</a> in Github.
+
+For help with upgrading, installing, or general questions please post to the <a href="https://groups.google.com/forum/#!forum/dataverse-community">Dataverse Google Group</a> or email support@dataverse.org.
+
+## Installation
+
+If this is a new installation, please see our <a href="http://guides.dataverse.org/en/4.18/installation/">Installation Guide</a>.
+
+## Upgrade
+
+1. Undeploy the previous version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin list-applications
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin undeploy dataverse
+
+2. Stop glassfish and remove the generated directory, start.
+
+- service glassfish stop
+- remove the generated directory: rm -rf &lt;glassfish install path&gt;glassfish4/glassfish/domains/domain1/generated
+- service glassfish start
+
+3. Deploy this version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin deploy &lt;path&gt;dataverse-4.18.war
+
+4. Restart glassfish.
+
+5. Update Citation Metadata Block
+
+- `wget https://github.com/IQSS/dataverse/releases/download/v4.18/citation.tsv`
+- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"`

doc/sphinx-guides/source/_static/installation/files/root/auth-providers/microsoft.json

@@ -0,0 +1,8 @@
+{
+    "id":"microsoft",
+    "factoryAlias":"oauth2",
+    "title":"Microsoft",
+    "subtitle":"",
+    "factoryData":"type: microsoft | userEndpoint: NONE | clientId: FIXME | clientSecret: FIXME",
+    "enabled":true
+}

doc/sphinx-guides/source/admin/external-tools.rst

@@ -37,6 +37,8 @@ In the curl command below, replace the placeholder "fabulousFileTool.json" place
 
   curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/externalTools --upload-file fabulousFileTool.json 
 
+Note that some tools will provide a preview mode, which provides an embedded, simplified view of the tool on the file pages of your installation. This is controlled by the `hasPreviewMode` parameter. 
+
 Listing All External Tools in Dataverse
 +++++++++++++++++++++++++++++++++++++++
 
@@ -76,10 +78,15 @@ Once you have added an external tool to your installation of Dataverse, you will
 File Level Explore Tools
 ++++++++++++++++++++++++
 
-File level explore tools are specific to the file type (content type or MIME type) of the file. For example, there is a tool for exploring PDF files in the "File Previewers" set of tools.
+File level explore tools are specific to the file type (content type or MIME type) of the file. For example, Data Explorer is tool for exploring tabular data files.
 
 An "Explore" button will appear (on both the dataset page and the file landing page) for files that match the type that the tool has been built for. When there are multiple explore tools for a filetype, the button becomes a dropdown.
 
+File Level Preview Tools
+++++++++++++++++++++++++
+
+File level explore tools can be set up to display in preview mode, which is a simplified view of an explore tool designed specifically for embedding in the file page. 
+
 File Level Configure Tools
 ++++++++++++++++++++++++++
 

doc/sphinx-guides/source/admin/integrations.rst

@@ -90,7 +90,7 @@ Dataverse supports a protocol called OAI-PMH that facilitates harvesting dataset
 SHARE
 +++++
 
-`SHARE <http://www.share-research.org>`_ is building a free, open, data set about research and scholarly activities across their life cycle. It's possible to add and installation of Dataverse as one of the `sources <https://share.osf.io/sources>`_ they include if you contact the SHARE team.
+`SHARE <http://www.share-research.org>`_ is building a free, open, data set about research and scholarly activities across their life cycle. It's possible to add an installation of Dataverse as one of the `sources <https://share.osf.io/sources>`_ they include if you contact the SHARE team.
 
 Research Data Preservation
 --------------------------

doc/sphinx-guides/source/admin/make-data-count.rst

@@ -63,8 +63,15 @@ Enable Logging for Make Data Count
 
 To make Dataverse log dataset usage (views and downloads) for Make Data Count, you must set the ``:MDCLogPath`` database setting. See :ref:`MDCLogPath` for details.
 
+If you wish to start logging in advance of setting up other components, or wish to log without display MDC metrics for any other reason, you can set the optional ``:DisplayMDCMetrics`` database setting to false. See :ref:`DisplayMDCMetrics` for details.
+
 After you have your first day of logs, you can process them the next day.
 
+Enable or Disable Display of Make Data Count Metrics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, when MDC logging is enabled (when ``:MDCLogPath`` is set), Dataverse will display MDC metrics instead of it's internal (legacy) metrics. You can avoid this (e.g. to collect MDC metrics for some period of time before starting to display them) by setting ``:DisplayMDCMetrics`` to false.
+
 Configure Counter Processor
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -128,11 +135,15 @@ Once you are satisfied with your testing, you should contact support@datacite.or
 - Views ("investigations" in COUNTER)
 - Downloads ("requests" in COUNTER)
 
+.. _MDC-updateCitationsForDataset:
+
 Configuring Dataverse for Make Data Count Citations
 ---------------------------------------------------
 
 Please note: as explained in the note above about limitations, this feature is not available to installations of Dataverse that use Handles.
 
+To configure Dataverse to pull citations from the test vs. production DataCite server see :ref:`doi.mdcbaseurlstring` in the Installation Guide.
+
 Please note that in the curl example, Bash environment variables are used with the idea that you can set a few environment variables and copy and paste the examples as is. For example, "$DOI" could become "doi:10.5072/FK2/BL2IBM" by issuing the following export command from Bash:
 
 ``export DOI="doi:10.5072/FK2/BL2IBM"``

doc/sphinx-guides/source/admin/troubleshooting.rst

@@ -32,12 +32,28 @@ A User Needs Their Account to Be Converted From Institutional (Shibboleth), ORCI
 
 See :ref:`converting-shibboleth-users-to-local` and :ref:`converting-oauth-users-to-local`.
 
+.. _troubleshooting-glassfish:
+
 Glassfish
 ---------
 
-``server.log`` is the main place to look when you encounter problems. Hopefully an error message has been logged. If there's a stack trace, it may be of interest to developers, especially they can trace line numbers back to a tagged version.
+.. _glassfish-log:
+
+Finding the Glassfish Log File
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``/usr/local/glassfish4/glassfish/domains/domain1/logs/server.log`` is the main place to look when you encounter problems (assuming you installed Glassfish in the default directory). Hopefully an error message has been logged. If there's a stack trace, it may be of interest to developers, especially they can trace line numbers back to a tagged version or commit. Send more of the stack trace (the entire file if possible) to developers who can help (see "Getting Help", below) and be sure to say which version of Dataverse you are running.
+
+.. _increase-glassfish-logging:
+
+Increasing Glassfish Logging
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-For debugging purposes, you may find it helpful to increase logging levels as mentioned in the :doc:`/developers/debugging` section of the Developer Guide.
+For debugging purposes, you may find it helpful to temporarily increase logging levels. Here's an example of increasing logging for the Java class behind the "datasets" API endpoints:
+
+``./asadmin set-log-levels edu.harvard.iq.dataverse.api.Datasets=FINE``
+
+For more on setting log levels, see the :doc:`/developers/debugging` section of the Developer Guide.
 
 Our guides focus on using the command line to manage Glassfish but you might be interested in an admin GUI at http://localhost:4848
 
@@ -100,3 +116,8 @@ Many Files with a File Type of "Unknown", "Application", or "Binary"
 --------------------------------------------------------------------
 
 From the home page of a Dataverse installation you can get a count of files by file type by clicking "Files" and then scrolling down to "File Type". If you see a lot of files that are "Unknown", "Application", or "Binary" you can have Dataverse attempt to redetect the file type by using the :ref:`Redetect File Type <redetect-file-type>` API endpoint.
+
+Getting Help
+------------
+
+If the troubleshooting advice above didn't help, contact any of the support channels mentioned in the :ref:`support` section of the Installation Guide.

doc/sphinx-guides/source/api/external-tools.rst

@@ -29,7 +29,13 @@ Note: This is the same list that appears in the :doc:`/admin/external-tools` sec
 How External Tools Are Presented to Users
 -----------------------------------------
 
-In short, an external tool appears under an "Explore" or "Configure" button either on a dataset landing page or a file landing page. See also the :ref:`testing-external-tools` section of the Admin Guide for some perspective on how installations of Dataverse will expect to test your tool before announcing it to their users.
+An external tool can appear in Dataverse in one of three ways:
+
+- under an "Explore" or "Configure" button either on a dataset landing page
+- under an "Explore" or "Configure" button on a file landing page
+- as an embedded preview on the file landing page
+
+See also the :ref:`testing-external-tools` section of the Admin Guide for some perspective on how installations of Dataverse will expect to test your tool before announcing it to their users.
 
 Creating an External Tool Manifest
 ----------------------------------
@@ -75,6 +81,8 @@ Terminology
     type                         Whether the external tool is an **explore** tool or a **configure** tool. Configure tools require an API token because they make changes to data files (files within datasets). Configure tools are currently not supported at the dataset level (no "Configure" button appears in the GUI for datasets).
 
     toolUrl                      The **base URL** of the tool before query parameters are added.
+
+    hasPreviewMode               A boolean that indicates whether tool has a preview mode which can be embedded in the File Page. Since this view is designed for embedding within Dataverse, the preview mode for a tool will typically be a view without headers or other options that may be included with a tool that is designed to be launched in a new window. Sometimes, a tool will exist solely to preview files in Dataverse and the preview mode will be the same as the regular view. 
 
     contentType                  File level tools operate on a specific **file type** (content type or MIME type such as "application/pdf") and this must be specified. Dataset level tools do not use contentType.
 

doc/sphinx-guides/source/api/getting-started.rst

@@ -94,6 +94,13 @@ A quick example search for the word "data" is https://demo.dataverse.org/api/sea
 
 See the :doc:`search` section for details.
 
+Finding Recently Published Dataverses, Datasets, and Files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+See :ref:`search-date-range`.
+
+It's also possible to find recently published datasets via OAI-PMH.
+
 Downloading Files
 ~~~~~~~~~~~~~~~~~