Merge pull request #5 from IQSS/develop

Merge upstream to 2043-split-gb-table

.gitignore

@@ -52,3 +52,10 @@ scripts/installer/default.config
 .idea
 **/*.iml
 /bin/
+
+# do not track Visual Studio Code files
+.vscode
+
+# ignore UI testing related files
+tests/node_modules
+tests/package-lock.json

.travis.yml.future

@@ -0,0 +1,42 @@
+services:
+  - docker
+
+jobs:
+  include:
+    # Execute java unit- and integration tests
+    - stage: test
+      language: java
+      jdk:
+        - oraclejdk8
+      script: mvn -DcompilerArgument=-Xlint:unchecked test -P all-unit-tests
+      after_success: mvn jacoco:report coveralls:report
+
+    # Execute Cypress for UI testing
+    # see https://docs.cypress.io/guides/guides/continuous-integration.html
+    - stage: test
+      language: node_js
+      node_js:
+        - "10"
+      addons:
+        apt:
+          packages:
+            # Ubuntu 16+ does not install this dependency by default, so we need to install it ourselves
+            - libgconf-2-4
+      cache:
+        # Caches $HOME/.npm when npm ci is default script command
+        # Caches node_modules in all other cases
+        npm: true
+        directories:
+          # we also need to cache folder with Cypress binary
+          - ~/.cache
+          # we want to cache the Glassfish and Solr dependencies as well
+          - conf/docker-aio/dv/deps
+      before_install:
+        - cd tests
+      install:
+        - npm ci
+      before_script:
+        - ./run_docker_dataverse.sh
+      script:
+        # --key needs to be injected using CYPRESS_RECORD_KEY to keep it secret
+        - $(npm bin)/cypress run --record

conf/solr/7.3.1/schema.xml

@@ -145,6 +145,7 @@
     <field name="dvObjectType" type="string" stored="true" indexed="true" multiValued="false"/>
     <field name="metadataSource" type="string" stored="true" indexed="true" multiValued="false"/>
     <field name="isHarvested" type="boolean" stored="true" indexed="true" multiValued="false"/>
+    <field name="fileDeleted" type="boolean" stored="true" indexed="true" multiValued="false"/>
 
     <field name="dvName" type="text_en" stored="true" indexed="true" multiValued="false"/>
     <field name="dvAlias" type="text_en" stored="true" indexed="true" multiValued="false"/>

doc/release-notes/5584-dataset-page-solr.md

@@ -0,0 +1 @@
+* Improved search on files within a dataset plus search facets are added to the dataset page. A solr schema upgrade and a full reindex are required to fully take advantage of this functionality. 

doc/sphinx-guides/source/admin/solr-search-index.rst

@@ -1,7 +1,7 @@
 Solr Search Index
 =================
 
-Dataverse requires Solr to be operational at all times. If you stop Solr, you should see a error about this on the home page, which is powered by the search index Solr provides. You can set up Solr by following the steps in our Installation Guide's :doc:`/installation/prerequisites` and :doc:`/installation/config` sections explaining how to configure it. This section you're reading now is about the care and feeding of the search index. PostgreSQL is the "source of truth" and the Dataverse application will copy data from PostgreSQL into Solr. For this reason, the search index can be rebuilt at any time. Depending on the amount of data you have, this can be a slow process. You are encouraged to experiment with production data to get a sense of how long a full reindexing will take.
+Dataverse requires Solr to be operational at all times. If you stop Solr, you should see a error about this on the root dataverse page, which is powered by the search index Solr provides. You can set up Solr by following the steps in our Installation Guide's :doc:`/installation/prerequisites` and :doc:`/installation/config` sections explaining how to configure it. This section you're reading now is about the care and feeding of the search index. PostgreSQL is the "source of truth" and the Dataverse application will copy data from PostgreSQL into Solr. For this reason, the search index can be rebuilt at any time. Depending on the amount of data you have, this can be a slow process. You are encouraged to experiment with production data to get a sense of how long a full reindexing will take.
 
 .. contents:: Contents:
 	:local:
@@ -17,7 +17,7 @@ Clear and Reindex
 Clearing Data from Solr
 ~~~~~~~~~~~~~~~~~~~~~~~
 
-Please note that the moment you issue this command, it will appear to end users looking at the home page that all data is gone! This is because the home page is powered by the search index.
+Please note that the moment you issue this command, it will appear to end users looking at the root dataverse page that all data is gone! This is because the root dataverse page is powered by the search index.
 
 ``curl http://localhost:8080/api/admin/index/clear``
 

doc/sphinx-guides/source/api/client-libraries.rst

@@ -11,9 +11,12 @@ Because Dataverse is a SWORD server, additional client libraries exist for Java,
 Python
 ------
 
-https://github.com/IQSS/dataverse-client-python is the offical Python package for Dataverse APIs.
+There are two Python modules for interacting with Dataverse APIs.
+
+`pyDataverse <https://github.com/AUSSDA/pyDataverse>`_ had its initial release in 2019 and can be installed with ``pip install pyDataverse``. The module is developed by `Stefan Kasberger <http://stefankasberger.at>`_ from `AUSSDA - The Austrian Social Science Data Archive <https://aussda.at>`_.  
+
+`dataverse-client-python <https://github.com/IQSS/dataverse-client-python>`_ had its initial release in 2015. `Robert Liebowitz <https://github.com/rliebz>`_ created this library while at the `Center for Open Science (COS) <https://centerforopenscience.org>`_ and the COS uses it to integrate the `Open Science Framework (OSF) <https://osf.io>`_ with Dataverse via an add-on which itself is open source and listed on the :doc:`/api/apps` page.
 
-`Robert Liebowitz <https://github.com/rliebz>`_ created this library while at the `Center for Open Science (COS) <https://centerforopenscience.org>`_ and the COS uses it to integrate the `Open Science Framework (OSF) <https://osf.io>`_ with Dataverse via an add-on which itself is open source and listed on the :doc:`/api/apps` page.
 
 R
 -

doc/sphinx-guides/source/api/intro.rst

@@ -47,6 +47,6 @@ Rather than using a production installation of Dataverse, API users are welcome
 Support
 -------
 
-If you are using the APIs for an installation of Dataverse hosted by your institution, you may want to reach out to the team that supports it. At the top of the Dataverse installation's home page, there should be a form you can fill out by clicking the "Support" link.
+If you are using the APIs for an installation of Dataverse hosted by your institution, you may want to reach out to the team that supports it. In the header at the top of the site, there should be a form you can fill out by clicking the "Support" link.
 
 If you are having trouble with http://demo.dataverse.org or have questions about the APIs, please feel free to reach out to the Dataverse community via https://groups.google.com/forum/#!forum/dataverse-community .

doc/sphinx-guides/source/developers/deployment.rst

@@ -80,19 +80,16 @@ You can skip this step if you're fine with the values in the "main.yml" file in
 Download and Run the "Create Instance" Script
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Once you have done the configuration above, you are ready to try running the "create instance" script to spin up Dataverse in AWS.
+Once you have done the configuration above, you are ready to try running the "ec2-create-instance.sh" script to spin up Dataverse in AWS.
 
 Download :download:`ec2-create-instance.sh <../../../../scripts/installer/ec2-create-instance.sh>` and put it somewhere reasonable. For the purpose of these instructions we'll assume it's in the "Downloads" directory in your home directory.
 
-You need to decide which branch you'd like to deploy to AWS. Select a branch from https://github.com/IQSS/dataverse/branches/all such as "develop" and pass it to the script with ``-b`` as in the following example. (Branches such as "master" and "develop" are described in the :doc:`version-control` section.)
+ec2-create-instance accepts a number few command-line switches:
 
-``bash ~/Downloads/ec2-create-instance.sh -b develop``
-
-You must specify the branch with ``-b`` but you can also specify a non-IQSS git repo URL with ``-r`` as in the following example.
-
-``bash ~/Downloads/ec2-create-instance.sh -b develop -r https://github.com/scholarsportal/dataverse.git``
-
-If you configured an Ansible file above and want to make use of it, add ``-g main.yml`` (or whatever you named your file) as in the following example.
+* -r: GitHub Repository URL (defaults to https://github.com/IQSS/dataverse.git)
+* -b: branch to build (defaults to develop)
+* -p: pemfile directory (defaults to $HOME)
+* -g: Ansible GroupVars file (if you wish to override role defaults)
 
 ``bash ~/Downloads/ec2-create-instance.sh -b develop -r https://github.com/scholarsportal/dataverse.git -g main.yml``
 
@@ -101,7 +98,7 @@ Now you will need to wait around 15 minutes until the deployment is finished. Ev
 Caveats
 ~~~~~~~
 
-Please note that while the script should work fine on newish branches, older branches that have different dependencies such as an older version of Solr are now expected to yield a working Dataverse installation. Your mileage may vary.
+Please note that while the script should work fine on newish branches, older branches that have different dependencies such as an older version of Solr may not produce a working Dataverse installation. Your mileage may vary.
 
 ----
 

doc/sphinx-guides/source/developers/testing.rst

@@ -276,7 +276,7 @@ Future Work on Unit Tests
 Future Work on Integration Tests
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- Automate testing of the Python client: https://github.com/IQSS/dataverse-client-python/issues/10
+- Automate testing of dataverse-client-python: https://github.com/IQSS/dataverse-client-python/issues/10
 - Work with @leeper on testing the R client: https://github.com/IQSS/dataverse-client-r
 - Review and attempt to implement "API Test Checklist" from @kcondon at https://docs.google.com/document/d/199Oq1YwQ4pYCguaeW48bIN28QAitSk63NbPYxJHCCAE/edit?usp=sharing
 - Attempt to use @openscholar approach for running integration tests using Travis https://github.com/openscholar/openscholar/blob/SCHOLAR-3.x/.travis.yml (probably requires using Ubuntu rather than CentOS)

doc/sphinx-guides/source/developers/troubleshooting.rst

@@ -14,7 +14,7 @@ For unknown reasons, Netbeans will sometimes change the following line under ``s
 
 ``<context-root>/</context-root>``
 
-Sometimes Netbeans will change ``/`` to ``/dataverse``. Sometimes it will delete the line entirely. Either way, you will see very strange behavior when attempting to click around Dataverse in a browser. The home page will load but icons will be missing. Any other page will fail to load entirely and you'll see a Glassfish error.
+Sometimes Netbeans will change ``/`` to ``/dataverse``. Sometimes it will delete the line entirely. Either way, you will see very strange behavior when attempting to click around Dataverse in a browser. The homepage will load but icons will be missing. Any other page will fail to load entirely and you'll see a Glassfish error.
 
 The solution is to put the file back to how it was before Netbeans touched it. If anyone knows of an open Netbeans bug about this, please let us know.
 

doc/sphinx-guides/source/installation/prerequisites.rst

@@ -224,7 +224,9 @@ On operating systems which use systemd such as RHEL or CentOS 7, you may then ad
 
         # sudo prlimit --pid pid --nofile=65000:65000
 
-Solr launches asynchronously and attempts to use the ``lsof`` binary to watch for its own availability. Installation of this package isn't required but will prevent a warning in the log at startup.
+Solr launches asynchronously and attempts to use the ``lsof`` binary to watch for its own availability. Installation of this package isn't required but will prevent a warning in the log at startup::
+
+	# yum install lsof
 
 Finally, you may start Solr and create the core that will be used to manage search information::
 
@@ -245,7 +247,7 @@ For systems running systemd, as root, download :download:`solr.service<../_stati
 For systems using init.d, download this :download:`Solr init script <../_static/installation/files/etc/init.d/solr>` and place it in ``/tmp``. Then start Solr and configure it to start at boot with the following commands::
 
         cp /tmp/solr /etc/init.d
-        service solr start
+        systemctl restart solr.service
         chkconfig solr on
 
 Securing Solr
@@ -259,7 +261,12 @@ jq
 Installing jq
 =============
 
-``jq`` is a command line tool for parsing JSON output that is used by the Dataverse installation script. https://stedolan.github.io/jq explains various ways of installing it, but a relatively straightforward method is described below. Please note that you must download the 64- or 32-bit version based on your architecture. In the example below, the 64-bit version is installed. We confirm it's executable and in our ``$PATH`` by checking the version (1.4 or higher should be fine):: 
+``jq`` is a command line tool for parsing JSON output that is used by the Dataverse installation script. It is available in the EPEL repository::
+
+	# yum install epel-release
+	# yum install jq
+
+or you may install it manually::
 
         # cd /usr/bin
         # wget http://stedolan.github.io/jq/download/linux64/jq
@@ -370,11 +377,12 @@ for the daemon (:fixedwidthplain:`/etc/init.d/rserve`), so that it
 gets started automatically when the system boots.  This is an
 :fixedwidthplain:`init.d`-style startup file. If this is a
 RedHat/CentOS 7 system, you may want to use the
-:fixedwidthplain:`systemctl`-style file
-:fixedwidthplain:`rserve.service` instead. (Copy it into the
-:fixedwidthplain:`/usr/lib/systemd/system/` directory)
-
+:download:`rserve.service<../../../../scripts/r/rserve/rserve.service>`
+systemd unit file instead. Copy it into the /usr/lib/systemd/system/ directory, then::
 
+	# systemctl daemon-reload
+	# systemctl enable rserve
+	# systemctl start rserve
 
 Note that the setup will also set the Rserve password to
 ":fixedwidthplain:`rserve`".  Rserve daemon runs under a