Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GDCC/8750 DRS archiver #8752

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
de7a914
DRS Archiver and documentation, required pom updates
qqmyers May 26, 2022
cf265ad
try 3 space indent
qqmyers May 26, 2022
a81517f
alternate sublist char
qqmyers May 26, 2022
0d9c4ac
Merge remote-tracking branch 'IQSS/develop' into GDCC/8750-DRS_Archiver
qqmyers May 27, 2022
42ddd69
blank line before list
qqmyers Jun 2, 2022
3d39a5e
fix reference
qqmyers Jun 2, 2022
4338417
Merge remote-tracking branch 'IQSS/develop' into GDCC/8750-DRS_Archiver
qqmyers Jun 26, 2022
a793d1b
Updates from HDC3 branch
qqmyers Jun 26, 2022
72ae051
Merge branch 'GDCC/8749-S3Archiver' into GDCC/8750-DRS_Archiver
qqmyers Jul 21, 2022
59a39f3
update constant names
qqmyers Jul 21, 2022
d142868
fix merge issue
qqmyers Jul 25, 2022
583420f
Merge branch 'GDCC/8749-S3Archiver' into GDCC/8750-DRS_Archiver
qqmyers Jul 25, 2022
7dc92e7
Apply suggestions from code review
qqmyers Jul 25, 2022
47a7eb3
param name change per review
qqmyers Jul 25, 2022
1cee369
Merge branch 'GDCC/8750-DRS_Archiver' of https://github.com/GlobalDat…
qqmyers Jul 25, 2022
f954a88
Merge branch 'GDCC/8749-S3Archiver' into GDCC/8750-DRS_Archiver
qqmyers Jul 28, 2022
e03c362
remove DRS Archiver info from guide
qqmyers Jul 28, 2022
062d982
info -> fine
qqmyers Jul 28, 2022
a4cd742
remove redundant/obsolete comment/log stmt
qqmyers Jul 28, 2022
9516634
Merge branch 'GDCC/8749-S3Archiver' into GDCC/8750-DRS_Archiver
qqmyers Aug 3, 2022
a1a1504
Merge remote-tracking branch 'IQSS/develop' into
qqmyers Aug 3, 2022
6f72207
Merge branch 'GDCC/8749-S3Archiver' into GDCC/8750-DRS_Archiver
qqmyers Aug 9, 2022
c6ca334
single quotes in curl
qqmyers Aug 9, 2022
f326a02
restore archival status support
qqmyers Aug 9, 2022
45c967a
Merge remote-tracking branch 'IQSS/develop' into GDCC/8750-DRS_Archiver
qqmyers Aug 10, 2022
1ddbe38
Merge remote-tracking branch 'IQSS/develop' into GDCC/8750-DRS_Archiver
qqmyers Aug 12, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/sphinx-guides/source/installation/config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1081,7 +1081,7 @@ These archival Bags include all of the files and metadata in a given dataset ver

The Dataverse Software offers an internal archive workflow which may be configured as a PostPublication workflow via an admin API call to manually submit previously published Datasets and prior versions to a configured archive such as Chronopolis. The workflow creates a `JSON-LD <http://www.openarchives.org/ore/0.9/jsonld>`_ serialized `OAI-ORE <https://www.openarchives.org/ore/>`_ map file, which is also available as a metadata export format in the Dataverse Software web interface.

At present, archiving classes include the DuraCloudSubmitToArchiveCommand, LocalSubmitToArchiveCommand, GoogleCloudSubmitToArchive, and S3SubmitToArchiveCommand , which all extend the AbstractSubmitToArchiveCommand and use the configurable mechanisms discussed below.
At present, archiving classes include the DuraCloudSubmitToArchiveCommand, LocalSubmitToArchiveCommand, GoogleCloudSubmitToArchive, and S3SubmitToArchiveCommand , which all extend the AbstractSubmitToArchiveCommand and use the configurable mechanisms discussed below. (A DRSSubmitToArchiveCommand, which works with Harvard's DRS also exists and, while specific to DRS, is a useful example of how Archivers can support single-version-only semantics and support archiving only from specified collections (with collection specific parameters)).

All current options support the archival status APIs and the same status is available in the dataset page version table (for contributors/those who could view the unpublished dataset, with more detail available to superusers).

Expand Down Expand Up @@ -1185,7 +1185,7 @@ The S3 Archiver defines one custom setting, a required :S3ArchiverConfig. It can

The credentials for your S3 account, can be stored in a profile in a standard credentials file (e.g. ~/.aws/credentials) referenced via "profile" key in the :S3ArchiverConfig setting (will default to the default entry), or can via MicroProfile settings as described for S3 stores (dataverse.s3archiver.access-key and dataverse.s3archiver.secret-key)

The :S3ArchiverConfig setting is a json object that must include an "s3_bucket_name" and may include additional S3-related parameters as described for S3 Stores, including "profile", "connection-pool-size","custom-endpoint-url", "custom-endpoint-region", "path-style-access", "payload-signing", and "chunked-encoding".
The :S3ArchiverConfig setting is a JSON object that must include an "s3_bucket_name" and may include additional S3-related parameters as described for S3 Stores, including "profile", "connection-pool-size","custom-endpoint-url", "custom-endpoint-region", "path-style-access", "payload-signing", and "chunked-encoding".

\:S3ArchiverConfig - minimally includes the name of the bucket to use. For example:

Expand Down
18 changes: 15 additions & 3 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
-->
<!-- TODO: Housekeeping is utterly needed. -->
<dependencies>

<!-- This dependency ensures the lib is on the classpath. That way, anything coming from
SLF4J should flow through our usual ways of using java.util.logging.
(Which makes it also configurable using those mechanisms.) No direct use intended. -->
Expand Down Expand Up @@ -357,7 +357,7 @@
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.9</version>
<version>1.15</version>
</dependency>
<!-- JavaSwift/JOSS: for accessing OpenStack cloud storage -->
<dependency>
Expand Down Expand Up @@ -516,7 +516,19 @@
<artifactId>google-cloud-storage</artifactId>
<!-- no version here as managed in <dependencyManagement> of parent for convergence! -->
</dependency>

<!-- JWT support, currently in DRSSubmitToArchiveCommand-->
<!-- https://mvnrepository.com/artifact/com.auth0/java-jwt -->
<dependency>
<groupId>com.auth0</groupId>
<artifactId>java-jwt</artifactId>
<version>3.19.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/io.github.erdtman/java-json-canonicalization -->
<dependency>
<groupId>io.github.erdtman</groupId>
<artifactId>java-json-canonicalization</artifactId>
<version>1.1</version>
</dependency>

<!-- TESTING DEPENDENCIES -->
<dependency>
Expand Down
Loading