Skip to content

Commit

Permalink
Merge branch 'develop' into 4250-ingest-failed IQSS#4250
Browse files Browse the repository at this point in the history
  • Loading branch information
pdurbin committed Nov 17, 2017
2 parents 268c3dc + b4cea62 commit dcfcbaf
Show file tree
Hide file tree
Showing 32 changed files with 955 additions and 335 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
"factoryAlias":"oauth2",
"title":"ORCID",
"subtitle":"",
"factoryData":"type: orcid | userEndpoint: https://api.sandbox.orcid.org/v1.2/{ORCID}/orcid-profile | clientId: FIXME | clientSecret: FIXME",
"factoryData":"type: orcid | userEndpoint: https://api.sandbox.orcid.org/v2.0/{ORCID}/record | clientId: FIXME | clientSecret: FIXME",
"enabled":true
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@
"factoryAlias":"oauth2",
"title":"ORCID",
"subtitle":"",
"factoryData":"type: orcid | userEndpoint: https://api.orcid.org/v1.2/{ORCID}/orcid-profile | clientId: FIXME | clientSecret: FIXME",
"factoryData":"type: orcid | userEndpoint: https://api.orcid.org/v2.0/{ORCID}/record | clientId: FIXME | clientSecret: FIXME",
"enabled":true
}
4 changes: 2 additions & 2 deletions doc/sphinx-guides/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,9 @@
# built documents.
#
# The short X.Y version.
version = '4.8.2'
version = '4.8.3'
# The full version, including alpha/beta/rc tags.
release = '4.8.2'
release = '4.8.3'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
Expand Down
98 changes: 97 additions & 1 deletion doc/sphinx-guides/source/developers/big-data-support.rst
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ Configuring the RSAL Mock

Info for configuring the RSAL Mock: https://github.com/sbgrid/rsal/tree/master/mocks

Also, to configure Dataverse to use the new workflow you must do the following:
Also, to configure Dataverse to use the new workflow you must do the following (see also the section below on workflows):

1. Configure the RSAL URL:

Expand Down Expand Up @@ -160,3 +160,99 @@ To specify replication sites that appear in rsync URLs:
In the GUI, this is called "Local Access". It's where you can compute on files on your cluster.

``curl http://localhost:8080/api/admin/settings/:LocalDataAccessPath -X PUT -d "/programs/datagrid"``

Workflows
---------

Dataverse can perform two sequences of actions when datasets are published: one prior to publishing (marked by a ``PrePublishDataset`` trigger), and one after the publication has succeeded (``PostPublishDataset``). The pre-publish workflow is useful for having an external system prepare a dataset for being publicly accessed (a possibly lengthy activity that requires moving files around, uploading videos to a streaming server, etc.), or to start an approval process. A post-publish workflow might be used for sending notifications about the newly published dataset.

Workflow steps are created using *step providers*. Dataverse ships with an internal step provider that offers some basic functionality, and with the ability to load 3rd party step providers. This allows installations to implement functionality they need without changing the Dataverse source code.

Steps can be internal (say, writing some data to the log) or external. External steps involve Dataverse sending a request to an external system, and waiting for the system to reply. The wait period is arbitrary, and so allows the external system unbounded operation time. This is useful, e.g., for steps that require human intervension, such as manual approval of a dataset publication.

The external system reports the step result back to dataverse, by sending a HTTP ``POST`` command to ``api/workflows/{invocation-id}``. The body of the request is passed to the paused step for further processing.

If a step in a workflow fails, Dataverse make an effort to roll back all the steps that preceeded it. Some actions, such as writing to the log, cannot be rolled back. If such an action has a public external effect (e.g. send an EMail to a mailing list) it is advisable to put it in the post-release workflow.

.. tip::
For invoking external systems using a REST api, Dataverse's internal step
provider offers a step for sending and receiving customizable HTTP requests.
It's called *http/sr*, and is detailed below.

Administration
~~~~~~~~~~~~~~

A Dataverse instance stores a set of workflows in its database. Workflows can be managed using the ``api/admin/workflows/`` endpoints of the :doc:`/api/native-api`. Sample workflow files are available in ``scripts/api/data/workflows``.

At the moment, defining a workflow for each trigger is done for the entire instance, using the endpoint ``api/admin/workflows/default/«trigger type»``.

In order to prevent unauthorized resuming of workflows, Dataverse maintains a "white list" of IP addresses from which resume requests are honored. This list is maintained using the ``/api/admin/workflows/ip-whitelist`` endpoint of the :doc:`/api/native-api`. By default, Dataverse honors resume requests from localhost only (``127.0.0.1;::1``), so set-ups that use a single server work with no additional configuration.


Available Steps
~~~~~~~~~~~~~~~

Dataverse has an internal step provider, whose id is ``:internal``. It offers the following steps:

log
+++

A step that writes data about the current workflow invocation to the instance log. It also writes the messages in its ``parameters`` map.

.. code:: json
{
"provider":":internal",
"stepType":"log",
"parameters": {
"aMessage": "message content",
"anotherMessage": "message content, too"
}
}
pause
+++++

A step that pauses the workflow. The workflow is paused until a POST request is sent to ``/api/workflows/{invocation-id}``.

.. code:: json
{
"provider":":internal",
"stepType":"pause"
}
http/sr
+++++++

A step that sends a HTTP request to an external system, and then waits for a response. The response has to match a regular expression specified in the step parameters. The url, content type, and message body can use data from the workflow context, using a simple markup language. This step has specific parameters for rollback.

.. code:: json
{
"provider":":internal",
"stepType":"http/sr",
"parameters": {
"url":"http://localhost:5050/dump/${invocationId}",
"method":"POST",
"contentType":"text/plain",
"body":"START RELEASE ${dataset.id} as ${dataset.displayName}",
"expectedResponse":"OK.*",
"rollbackUrl":"http://localhost:5050/dump/${invocationId}",
"rollbackMethod":"DELETE ${dataset.id}"
}
}
Available variables are:

* ``invocationId``
* ``dataset.id``
* ``dataset.identifier``
* ``dataset.globalId``
* ``dataset.displayName``
* ``dataset.citation``
* ``minorVersion``
* ``majorVersion``
* ``releaseStatus``
9 changes: 4 additions & 5 deletions doc/sphinx-guides/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,23 +3,22 @@
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Dataverse 4.8.2 Guides
Dataverse 4.8.3 Guides
======================

These guides are for the most recent version of Dataverse. For the guides for **version 4.8.1** please go `here <http://guides.dataverse.org/en/4.8.1/>`_.
These guides are for the most recent version of Dataverse. For the guides for **version 4.8.2** please go `here <http://guides.dataverse.org/en/4.8.2/>`_.

.. toctree::
:glob:
:titlesonly:
:maxdepth: 2

user/index
installation/index
admin/index
api/index
installation/index
developers/index
style/index
admin/index
workflows

How the Guides Are Organized
=============================
Expand Down
92 changes: 0 additions & 92 deletions doc/sphinx-guides/source/workflows.rst

This file was deleted.

2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

<groupId>edu.harvard.iq</groupId>
<artifactId>dataverse</artifactId>
<version>4.8.2</version>
<version>4.8.3</version>
<packaging>war</packaging>

<name>dataverse</name>
Expand Down
8 changes: 0 additions & 8 deletions scripts/api/data/authentication-providers/base-oauth2.json

This file was deleted.

6 changes: 3 additions & 3 deletions scripts/api/data/authentication-providers/orcid-sandbox.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"id":"orcid-sandbox",
"id":"orcid-v2-sandbox",
"factoryAlias":"oauth2",
"title":"ORCID Sandbox",
"subtitle":"ORCiD - sandbox",
"factoryData":"type: orcid | userEndpoint: https://api.sandbox.orcid.org/v1.2/{ORCID}/orcid-profile | clientId: APP-HIV99BRM37FSWPH6 | clientSecret: ee844b70-f223-4f15-9b6f-4991bf8ed7f0",
"subtitle":"ORCiD - sandbox (v2)",
"factoryData":"type: orcid | userEndpoint: https://api.sandbox.orcid.org/v2.0/{ORCID}/person | clientId: APP-HIV99BRM37FSWPH6 | clientSecret: ee844b70-f223-4f15-9b6f-4991bf8ed7f0",
"enabled":true
}
4 changes: 2 additions & 2 deletions scripts/installer/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,10 @@ ${GLASSFISH_SETUP_SCRIPT}: glassfish-setup.sh
/bin/cp glassfish-setup.sh ${INSTALLER_ZIP_DIR}


${POSTGRES_DRIVERS}: pgdriver/postgresql-8.4-703.jdbc4.jar pgdriver/postgresql-9.0-802.jdbc4.jar pgdriver/postgresql-9.1-902.jdbc4.jar
${POSTGRES_DRIVERS}: pgdriver/postgresql-8.4-703.jdbc4.jar pgdriver/postgresql-9.0-802.jdbc4.jar pgdriver/postgresql-9.1-902.jdbc4.jar pgdriver/postgresql-9.2-1004.jdbc4.jar pgdriver/postgresql-9.3-1104.jdbc4.jar pgdriver/postgresql-9.4.1212.jar pgdriver/postgresql-42.1.4.jar
@echo copying postgres drviers
@mkdir -p ${POSTGRES_DRIVERS}
/bin/cp pgdriver/postgresql-8.4-703.jdbc4.jar pgdriver/postgresql-9.0-802.jdbc4.jar pgdriver/postgresql-9.1-902.jdbc4.jar ${INSTALLER_ZIP_DIR}/pgdriver
/bin/cp pgdriver/postgresql-8.4-703.jdbc4.jar pgdriver/postgresql-9.0-802.jdbc4.jar pgdriver/postgresql-9.1-902.jdbc4.jar pgdriver/postgresql-9.2-1004.jdbc4.jar pgdriver/postgresql-9.3-1104.jdbc4.jar pgdriver/postgresql-9.4.1212.jar pgdriver/postgresql-42.1.4.jar ${INSTALLER_ZIP_DIR}/pgdriver

${API_SCRIPTS}: ../api/setup-datasetfields.sh ../api/setup-users.sh ../api/setup-dvs.sh ../api/setup-identity-providers.sh ../api/setup-all.sh ../api/post-install-api-block.sh ../api/setup-builtin-roles.sh ../api/data
@echo copying api scripts
Expand Down
4 changes: 3 additions & 1 deletion src/main/java/Bundle.properties
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,7 @@ login.error=Error validating the username, email address, or password. Please tr
user.error.cannotChangePassword=Sorry, your password cannot be changed. Please contact your system administrator.
user.error.wrongPassword=Sorry, wrong password.
login.button=Log In with {0}
login.button.orcid=Create or Connect your ORCID
# authentication providers
auth.providers.title=Other options
auth.providers.tip=You can convert a Dataverse account to use one of the options above. <a href="{0}/{1}/user/account.html" target="_blank">Learn more</a>.
Expand All @@ -259,6 +260,7 @@ auth.providers.persistentUserIdName.orcid=ORCID iD
auth.providers.persistentUserIdName.github=ID
auth.providers.persistentUserIdTooltip.orcid=ORCID provides a persistent digital identifier that distinguishes you from other researchers.
auth.providers.persistentUserIdTooltip.github=GitHub assigns a unique number to every user.
auth.providers.orcid.insufficientScope=Dataverse was not granted the permission to read user data from ORCID.
# Friendly AuthenticationProvider names
authenticationProvider.name.builtin=Dataverse
authenticationProvider.name.null=(provider is unknown)
Expand Down Expand Up @@ -331,7 +333,7 @@ oauth2.convertAccount.success=Your Dataverse account is now associated with your

# oauth2/callback.xhtml
oauth2.callback.page.title=OAuth Callback
oauth2.callback.message=<strong>OAuth2 Error</strong> - Sorry, the identification process did not succeed.
oauth2.callback.message=<strong>Authentication Error</strong> - Dataverse could not authenticate your ORCID login. Please make sure you authorize your ORCID account to connect with Dataverse. For more details about the information being requested, see the <a href="{0}/{1}/user/account.html#orcid-log-in" title="ORCID Log In - Dataverse User Guide" target="_blank">User Guide</a>.

# tab on dataverseuser.xhtml
apitoken.title=API Token
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/edu/harvard/iq/dataverse/LoginPage.java
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ public String login() {
}
authReq.setIpAddress( dvRequestService.getDataverseRequest().getSourceAddress() );
try {
AuthenticatedUser r = authSvc.authenticate(credentialsAuthProviderId, authReq);
AuthenticatedUser r = authSvc.getCreateAuthenticatedUser(credentialsAuthProviderId, authReq);
logger.log(Level.FINE, "User authenticated: {0}", r.getEmail());
session.setUser(r);

Expand Down
8 changes: 6 additions & 2 deletions src/main/java/edu/harvard/iq/dataverse/api/Admin.java
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,6 @@
import edu.harvard.iq.dataverse.ingest.IngestServiceBean;
import edu.harvard.iq.dataverse.userdata.UserListMaker;
import edu.harvard.iq.dataverse.userdata.UserListResult;
import edu.harvard.iq.dataverse.util.StringUtil;
import java.math.BigDecimal;
import java.util.Date;
import java.util.ResourceBundle;
import javax.inject.Inject;
Expand Down Expand Up @@ -1002,4 +1000,10 @@ public Response validatePassword(String password) {
.add("errors", errorArray)
);
}

@GET
@Path("/isOrcid")
public Response isOrcidEnabled() {
return authSvc.isOrcidEnabled() ? ok("Orcid is enabled") : ok("no orcid for you.");
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ public class AuthenticatedUserDisplayInfo extends RoleAssigneeDisplayInfo {
private String firstName;
private String position;

/**
/*
* @todo Shouldn't we persist the displayName too? It still exists on the
* authenticateduser table.
*/
Expand Down
Loading

0 comments on commit dcfcbaf

Please sign in to comment.