Deprecate rsync: remove from guides, mark code as deprecated (#10620)

* remove rsync docs #8985

* add deprecation notices to Java code for rsync #8985

* delete deprecated config options from guides #8985

* whoops, put :UploadMethods doc ref back #8985

* restore :PublicInstall to non-deprecated #8985

doc/release-notes/8985-deprecate-rsync.md

@@ -0,0 +1,8 @@
+Support for rsync has been deprecated. Information has been removed from the guides for rsync and related software such as Data Capture Module (DCM) and Repository Storage Abstraction Layer (RSAL). You can still find this information in [older versions](https://guides.dataverse.org/en/6.2/developers/big-data-support.html#data-capture-module-dcm) of the guides.
+
+The following related database settings have been deprecated as well:
+
+- :DataCaptureModuleUrl
+- :DownloadMethods
+- :LocalDataAccessPath
+- :RepositoryStorageAbstractionLayerUrl

doc/sphinx-guides/source/api/native-api.rst

@@ -2315,7 +2315,7 @@ The fully expanded example above (without environment variables) looks like this
 
   curl "https://demo.dataverse.org/api/datasets/24/locks?type=Ingest"
 
-Currently implemented lock types are ``Ingest``, ``Workflow``, ``InReview``, ``DcmUpload``, ``finalizePublication``, ``EditInProgress`` and ``FileValidationFailed``.
+Currently implemented lock types are ``Ingest``, ``Workflow``, ``InReview``, ``DcmUpload`` (deprecated), ``finalizePublication``, ``EditInProgress`` and ``FileValidationFailed``.
 
 The API will output the list of locks, for example:: 
 
@@ -2406,7 +2406,7 @@ Use the following API to list ALL the locks on all the datasets in your installa
 The listing can be filtered by specific lock type **and/or** user, using the following *optional* query parameters:
 
 * ``userIdentifier`` - To list the locks owned by a specific user
-* ``type`` - To list the locks of the type specified. If the supplied value does not match a known lock type, the API will return an error and a list of valid lock types. As of writing this, the implemented lock types are ``Ingest``, ``Workflow``, ``InReview``, ``DcmUpload``, ``finalizePublication``, ``EditInProgress`` and ``FileValidationFailed``.
+* ``type`` - To list the locks of the type specified. If the supplied value does not match a known lock type, the API will return an error and a list of valid lock types. As of writing this, the implemented lock types are ``Ingest``, ``Workflow``, ``InReview``, ``DcmUpload`` (deprecated), ``finalizePublication``, ``EditInProgress`` and ``FileValidationFailed``.
 
 For example:
 
@@ -3192,7 +3192,7 @@ Note: you can use the combination of cURL's ``-J`` (``--remote-header-name``) an
 Restrict Files
 ~~~~~~~~~~~~~~
 
-Restrict or unrestrict an existing file where ``id`` is the database id of the file or ``pid`` is the persistent id (DOI or Handle) of the file to restrict. Note that some Dataverse installations do not allow the ability to restrict files.
+Restrict or unrestrict an existing file where ``id`` is the database id of the file or ``pid`` is the persistent id (DOI or Handle) of the file to restrict. Note that some Dataverse installations do not allow the ability to restrict files (see :ref:`:PublicInstall`).
 
 A curl example using an ``id``
 

doc/sphinx-guides/source/developers/big-data-support.rst

@@ -1,7 +1,7 @@
 Big Data Support
 ================
 
-Big data support includes some highly experimental options. Eventually more of this content will move to the Installation Guide.
+Big data support includes some experimental options. Eventually more of this content will move to the Installation Guide.
 
 .. contents:: |toctitle|
         :local:
@@ -187,179 +187,3 @@ As described in that document, Globus transfers can be initiated by choosing the
 An overview of the control and data transfer interactions between components was presented at the 2022 Dataverse Community Meeting and can be viewed in the `Integrations and Tools Session Video <https://youtu.be/3ek7F_Dxcjk?t=5289>`_ around the 1 hr 28 min mark.
 
 See also :ref:`Globus settings <:GlobusSettings>`.
-
-Data Capture Module (DCM)
--------------------------
-
-Please note: The DCM feature is deprecated.
-
-Data Capture Module (DCM) is an experimental component that allows users to upload large datasets via rsync over ssh.
-
-DCM was developed and tested using Glassfish but these docs have been updated with references to Payara.
-
-Install a DCM
-~~~~~~~~~~~~~
-
-Installation instructions can be found at https://github.com/sbgrid/data-capture-module/blob/master/doc/installation.md. Note that shared storage (posix or AWS S3) between your Dataverse installation and your DCM is required. You cannot use a DCM with Swift at this point in time.
-
-.. FIXME: Explain what ``dataverse.files.dcm-s3-bucket-name`` is for and what it has to do with ``dataverse.files.s3.bucket-name``.
-
-Once you have installed a DCM, you will need to configure two database settings on the Dataverse installation side. These settings are documented in the :doc:`/installation/config` section of the Installation Guide:
-
-- ``:DataCaptureModuleUrl`` should be set to the URL of a DCM you installed.
-- ``:UploadMethods`` should include ``dcm/rsync+ssh``.
-
-This will allow your Dataverse installation to communicate with your DCM, so that your Dataverse installation can download rsync scripts for your users.
-
-Downloading rsync scripts via Your Dataverse Installation's API
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The rsync script can be downloaded from your Dataverse installation via API using an authorized API token. In the curl example below, substitute ``$PERSISTENT_ID`` with a DOI or Handle:
-
-``curl -H "X-Dataverse-key: $API_TOKEN" $DV_BASE_URL/api/datasets/:persistentId/dataCaptureModule/rsync?persistentId=$PERSISTENT_ID``
-
-How a DCM reports checksum success or failure to your Dataverse Installation
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Once the user uploads files to a DCM, that DCM will perform checksum validation and report to your Dataverse installation the results of that validation. The DCM must be configured to pass the API token of a superuser. The implementation details, which are subject to change, are below.
-
-The JSON that a DCM sends to your Dataverse installation on successful checksum validation looks something like the contents of :download:`checksumValidationSuccess.json <../_static/installation/files/root/big-data-support/checksumValidationSuccess.json>` below:
-
-.. literalinclude:: ../_static/installation/files/root/big-data-support/checksumValidationSuccess.json
-   :language: json
-
-- ``status`` - The valid strings to send are ``validation passed`` and ``validation failed``.
-- ``uploadFolder`` - This is the directory on disk where your Dataverse installation should attempt to find the files that a DCM has moved into place. There should always be a ``files.sha`` file and a least one data file. ``files.sha`` is a manifest of all the data files and their checksums. The ``uploadFolder`` directory is inside the directory where data is stored for the dataset and may have the same name as the "identifier" of the persistent id (DOI or Handle). For example, you would send ``"uploadFolder": "DNXV2H"`` in the JSON file when the absolute path to this directory is ``/usr/local/payara6/glassfish/domains/domain1/files/10.5072/FK2/DNXV2H/DNXV2H``.
-- ``totalSize`` - Your Dataverse installation will use this value to represent the total size in bytes of all the files in the "package" that's created. If 360 data files and one ``files.sha`` manifest file are in the ``uploadFolder``, this value is the sum of the 360 data files.
-
-
-Here's the syntax for sending the JSON.
-
-``curl -H "X-Dataverse-key: $API_TOKEN" -X POST -H 'Content-type: application/json' --upload-file checksumValidationSuccess.json $DV_BASE_URL/api/datasets/:persistentId/dataCaptureModule/checksumValidation?persistentId=$PERSISTENT_ID``
-
-
-Steps to set up a DCM mock for Development
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-See instructions at https://github.com/sbgrid/data-capture-module/blob/master/doc/mock.md
-
-
-Add Dataverse Installation settings to use mock (same as using DCM, noted above):
-
-- ``curl http://localhost:8080/api/admin/settings/:DataCaptureModuleUrl -X PUT -d "http://localhost:5000"``
-- ``curl http://localhost:8080/api/admin/settings/:UploadMethods -X PUT -d "dcm/rsync+ssh"``
-
-At this point you should be able to download a placeholder rsync script. Your Dataverse installation is then waiting for news from the DCM about if checksum validation has succeeded or not. First, you have to put files in place, which is usually the job of the DCM. You should substitute "X1METO" for the "identifier" of the dataset you create. You must also use the proper path for where you store files in your dev environment.
-
-- ``mkdir /usr/local/payara6/glassfish/domains/domain1/files/10.5072/FK2/X1METO``
-- ``mkdir /usr/local/payara6/glassfish/domains/domain1/files/10.5072/FK2/X1METO/X1METO``
-- ``cd /usr/local/payara6/glassfish/domains/domain1/files/10.5072/FK2/X1METO/X1METO``
-- ``echo "hello" > file1.txt``
-- ``shasum file1.txt > files.sha``
-
-
-
-Now the files are in place and you need to send JSON to your Dataverse installation with a success or failure message as described above. Make a copy of ``doc/sphinx-guides/source/_static/installation/files/root/big-data-support/checksumValidationSuccess.json`` and put the identifier in place such as "X1METO" under "uploadFolder"). Then use curl as described above to send the JSON.
-
-Troubleshooting
-~~~~~~~~~~~~~~~
-
-The following low level command should only be used when troubleshooting the "import" code a DCM uses but is documented here for completeness.
-
-``curl -H "X-Dataverse-key: $API_TOKEN" -X POST "$DV_BASE_URL/api/batch/jobs/import/datasets/files/$DATASET_DB_ID?uploadFolder=$UPLOAD_FOLDER&totalSize=$TOTAL_SIZE"``
-
-Repository Storage Abstraction Layer (RSAL)
--------------------------------------------
-
-Please note: The RSAL feature is deprecated.
-
-Steps to set up a DCM via Docker for Development
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-See https://github.com/IQSS/dataverse/blob/develop/conf/docker-dcm/readme.md
-
-Using the RSAL Docker Containers
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-- Create a dataset (either with the procedure mentioned in DCM Docker Containers, or another process)
-- Publish the dataset (from the client container): ``cd /mnt; ./publish_major.bash ${database_id}``
-- Run the RSAL component of the workflow (from the host): ``docker exec -it rsalsrv /opt/rsal/scn/pub.py``
-- If desired, from the client container you can download the dataset following the instructions in the dataset access section of the dataset page.
-
-Configuring the RSAL Mock
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Info for configuring the RSAL Mock: https://github.com/sbgrid/rsal/tree/master/mocks
-
-Also, to configure your Dataverse installation to use the new workflow you must do the following (see also the :doc:`workflows` section):
-
-1. Configure the RSAL URL:
-
-``curl -X PUT -d 'http://<myipaddr>:5050' http://localhost:8080/api/admin/settings/:RepositoryStorageAbstractionLayerUrl``
-
-2. Update workflow json with correct URL information:
-
-Edit internal-httpSR-workflow.json and replace url and rollbackUrl to be the url of your RSAL mock.
-
-3. Create the workflow:
-
-``curl http://localhost:8080/api/admin/workflows -X POST --data-binary @internal-httpSR-workflow.json -H "Content-type: application/json"``
-
-4. List available workflows:
-
-``curl http://localhost:8080/api/admin/workflows``
-
-5. Set the workflow (id) as the default workflow for the appropriate trigger:
-
-``curl http://localhost:8080/api/admin/workflows/default/PrePublishDataset -X PUT -d 2``
-
-6. Check that the trigger has the appropriate default workflow set:
-
-``curl http://localhost:8080/api/admin/workflows/default/PrePublishDataset``
-
-7. Add RSAL to whitelist
-
-8. When finished testing, unset the workflow:
-
-``curl -X DELETE http://localhost:8080/api/admin/workflows/default/PrePublishDataset``
-
-Configuring download via rsync
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-In order to see the rsync URLs, you must run this command:
-
-``curl -X PUT -d 'rsal/rsync' http://localhost:8080/api/admin/settings/:DownloadMethods``
-
-..  TODO: Document these in the Installation Guide once they're final.
-
-To specify replication sites that appear in rsync URLs:
-
-Download :download:`add-storage-site.json <../../../../scripts/api/data/storageSites/add-storage-site.json>` and adjust it to meet your needs. The file should look something like this:
-
-.. literalinclude:: ../../../../scripts/api/data/storageSites/add-storage-site.json
-
-Then add the storage site using curl:
-
-``curl -H "Content-type:application/json" -X POST http://localhost:8080/api/admin/storageSites --upload-file add-storage-site.json``
-
-You make a storage site the primary site by passing "true". Pass "false" to make it not the primary site. (id "1" in the example):
-
-``curl -X PUT -d true http://localhost:8080/api/admin/storageSites/1/primaryStorage``
-
-You can delete a storage site like this (id "1" in the example):
-
-``curl -X DELETE http://localhost:8080/api/admin/storageSites/1``
-
-You can view a single storage site like this: (id "1" in the example):
-
-``curl http://localhost:8080/api/admin/storageSites/1``
-
-You can view all storage site like this:
-
-``curl http://localhost:8080/api/admin/storageSites``
-
-In the GUI, this is called "Local Access". It's where you can compute on files on your cluster.
-
-``curl http://localhost:8080/api/admin/settings/:LocalDataAccessPath -X PUT -d "/programs/datagrid"``
-
-

doc/sphinx-guides/source/installation/config.rst

@@ -4218,20 +4218,6 @@ This is useful for specific cases where an installation's files are stored in pu
 
 ``curl -X PUT -d true http://localhost:8080/api/admin/settings/:PublicInstall``
 
-:DataCaptureModuleUrl
-+++++++++++++++++++++
-
-The URL for your Data Capture Module (DCM) installation. This component is experimental and can be downloaded from https://github.com/sbgrid/data-capture-module .
-
-``curl -X PUT -d 'https://dcm.example.edu' http://localhost:8080/api/admin/settings/:DataCaptureModuleUrl``
-
-:RepositoryStorageAbstractionLayerUrl
-+++++++++++++++++++++++++++++++++++++
-
-The URL for your Repository Storage Abstraction Layer (RSAL) installation. This component is experimental and can be downloaded from https://github.com/sbgrid/rsal .
-
-``curl -X PUT -d 'https://rsal.example.edu' http://localhost:8080/api/admin/settings/:RepositoryStorageAbstractionLayerUrl``
-
 .. _:UploadMethods:
 
 :UploadMethods
@@ -4241,23 +4227,15 @@ This setting controls which upload methods are available to users of your Datave
 
 - ``native/http``: Corresponds to "Upload with HTTP via your browser" and APIs that use HTTP (SWORD and native).
 - ``dvwebloader``: Corresponds to :ref:`folder-upload`. Note that ``dataverse.files.<id>.upload-redirect`` must be set to "true" on an S3 store for this method to show up in the UI. In addition, :ref:`:WebloaderUrl` must be set. CORS allowed on the S3 bucket. See :ref:`cors-s3-bucket`.
-- ``dcm/rsync+ssh``: Corresponds to "Upload with rsync+ssh via Data Capture Module (DCM)". A lot of setup is required, as explained in the :doc:`/developers/big-data-support` section of the Developer Guide.
 
 Out of the box only ``native/http`` is enabled and will work without further configuration. To add multiple upload method, separate them using a comma like this:
 
-``curl -X PUT -d 'native/http,dcm/rsync+ssh' http://localhost:8080/api/admin/settings/:UploadMethods``
+``curl -X PUT -d 'native/http,dvwebloader' http://localhost:8080/api/admin/settings/:UploadMethods``
 
 You'll always want at least one upload method, so the easiest way to remove one of them is to simply ``PUT`` just the one you want, like this:
 
 ``curl -X PUT -d 'native/http' http://localhost:8080/api/admin/settings/:UploadMethods``
 
-:DownloadMethods
-++++++++++++++++
-
-This setting is experimental and related to Repository Storage Abstraction Layer (RSAL).
-
-``curl -X PUT -d 'rsal/rsync' http://localhost:8080/api/admin/settings/:DownloadMethods``
-
 :GuestbookResponsesPageDisplayLimit
 +++++++++++++++++++++++++++++++++++