Merge remote-tracking branch 'IQSS/develop' into IQSS/9506-thumbnail-…

…tracking

.gitattributes

@@ -0,0 +1,4 @@
+# https://www.git-scm.com/docs/gitattributes
+
+# This set mandatory LF line endings for .sh files preventing from windows users to having to change the value of their git config --global core.autocrlf to 'false' or 'input'
+*.sh text eol=lf

.github/ISSUE_TEMPLATE/bug_report.md

@@ -3,7 +3,7 @@ name: Bug report
 about: Did you encounter something unexpected or incorrect in the Dataverse software?
   We'd like to hear about it!
 title: ''
-labels: ''
+labels: 'Type: Bug'
 assignees: ''
 
 ---

.github/ISSUE_TEMPLATE/feature_request.md

@@ -2,7 +2,7 @@
 name: Feature request
 about: Suggest an idea or new feature for the Dataverse software!
 title: 'Feature Request/Idea:'
-labels: ''
+labels: 'Type: Feature'
 assignees: ''
 
 ---

README.md

@@ -1,7 +1,7 @@
 Dataverse®  
 ===============
 
-Dataverse is an [open source][] software platform for sharing, finding, citing, and preserving research data (developed by the [Data Science and Products team](http://www.iq.harvard.edu/people/people/data-science-products) at the [Institute for Quantitative Social Science](http://iq.harvard.edu/) and the [Dataverse community][]).
+Dataverse is an [open source][] software platform for sharing, finding, citing, and preserving research data (developed by the [Dataverse team](https://dataverse.org/about) at the [Institute for Quantitative Social Science](https://iq.harvard.edu/) and the [Dataverse community][]).
 
 [dataverse.org][] is our home on the web and shows a map of Dataverse installations around the world, a list of [features][], [integrations][] that have been made possible through [REST APIs][], our development [roadmap][], and more.
 
@@ -15,7 +15,7 @@ We love contributors! Please see our [Contributing Guide][] for ways you can hel
 
 Dataverse is a trademark of President and Fellows of Harvard College and is registered in the United States.
 
-[![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg?raw=true "Dataverse Project")](http://dataverse.org)
+[![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg "Dataverse Project")](http://dataverse.org)
 
 [![API Test Status](https://jenkins.dataverse.org/buildStatus/icon?job=IQSS-dataverse-develop&subject=API%20Test%20Status)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/)
 [![API Test Coverage](https://img.shields.io/jenkins/coverage/jacoco?jobUrl=https%3A%2F%2Fjenkins.dataverse.org%2Fjob%2FIQSS-dataverse-develop&label=API%20Test%20Coverage)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/ws/target/coverage-it/index.html)
@@ -26,15 +26,15 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 [dataverse.org]: https://dataverse.org
 [demo.dataverse.org]: https://demo.dataverse.org
 [Dataverse community]: https://dataverse.org/developers
-[Installation Guide]: http://guides.dataverse.org/en/latest/installation/index.html
+[Installation Guide]: https://guides.dataverse.org/en/latest/installation/index.html
 [latest release]: https://github.com/IQSS/dataverse/releases
 [features]: https://dataverse.org/software-features
 [roadmap]: https://www.iq.harvard.edu/roadmap-dataverse-project
 [integrations]: https://dataverse.org/integrations
-[REST APIs]: http://guides.dataverse.org/en/latest/api/index.html
+[REST APIs]: https://guides.dataverse.org/en/latest/api/index.html
 [Contributing Guide]: CONTRIBUTING.md
 [mailing list]: https://groups.google.com/group/dataverse-community
 [community call]: https://dataverse.org/community-calls
-[chat.dataverse.org]: http://chat.dataverse.org
+[chat.dataverse.org]: https://chat.dataverse.org
 [Dataverse Community Meeting]: https://dataverse.org/events
 [open source]: LICENSE.md

doc/release-notes/10001-datasets-files-api-user-permissions.md

@@ -0,0 +1,13 @@
+- New query parameter `includeDeaccessioned` added to the getVersion endpoint (/api/datasets/{id}/versions/{versionId}) to consider deaccessioned versions when searching for versions.
+
+
+- New endpoint to get user permissions on a dataset (/api/datasets/{id}/userPermissions). In particular, the user permissions that this API call checks, returned as booleans, are the following:
+
+  - Can view the unpublished dataset
+  - Can edit the dataset
+  - Can publish the dataset
+  - Can manage the dataset permissions
+  - Can delete the dataset draft
+
+
+- New permission check "canManageFilePermissions" added to the existing endpoint for getting user permissions on a file (/api/access/datafile/{id}/userPermissions).

doc/release-notes/10060-api-changelog.md

@@ -0,0 +1,3 @@
+We have started maintaining an API changelog: https://dataverse-guide--10127.org.readthedocs.build/en/10127/api/changelog.html
+
+See also #10060.

doc/release-notes/10093-signedUrl_improvements.md

@@ -0,0 +1,5 @@
+A new version of the standard Dataverse Previewers from https://github.com/gdcc/dataverse-previewers is available. The new version supports the use of signedUrls rather than API keys when previewing restricted files (including files in draft dataset versions). Upgrading is highly recommended.
+
+SignedUrls can now be used with PrivateUrl access tokens, which allows PrivateUrl users to view previewers that are configured to use SignedUrls. See #10093.
+
+Launching a dataset-level configuration tool will automatically generate an API token when needed. This is consistent with how other types of tools work. See #10045.

doc/release-notes/10104-dataset-citation-deaccessioned.md

@@ -0,0 +1 @@
+The getDatasetVersionCitation (/api/datasets/{id}/versions/{versionId}/citation) endpoint now accepts a new boolean optional query parameter "includeDeaccessioned", which, if enabled, causes the endpoint to consider deaccessioned versions when searching for versions to obtain the citation.

doc/release-notes/6.0-release-notes.md

@@ -166,6 +166,22 @@ If you are running Payara as a non-root user (and you should be!), **remember no
 
    Your `:MDCLogPath` database setting might be pointing to a Payara 5 directory such as `/usr/local/payara5/glassfish/domains/domain1/logs`. If so, edit this to be Payara 6. You'll probably want to copy your logs over as well.
 
+1. If you've enabled access logging or any other site-specific configuration, be sure to preserve them. For instance, the default domain.xml includes
+
+   ```
+        <http-service>
+        <access-log></access-log>
+   ```
+
+   but you may wish to include
+
+   ```
+        <http-service access-logging-enabled="true">
+        <access-log format="%client.name% %datetime% %request% %status% %response.length% %header.user-agent% %header.referer% %cookie.JSESSIONID% %header.x-forwarded-for%"></access-log>
+   ```
+
+   Be sure to keep a previous copy of your domain.xml for reference.
+
 1. Update systemd unit file (or other init system) from `/usr/local/payara5` to `/usr/local/payara6`, if applicable.
 
 1. Start Payara.

doc/release-notes/9412-markdown-previewer.md

@@ -0,0 +1 @@
+There is now a Markdown (.md) previewer: https://dataverse-guide--9986.org.readthedocs.build/en/9986/user/dataset-management.html#file-previews

doc/release-notes/9590-intellij-redeploy.md

@@ -0,0 +1,3 @@
+Developers can enjoy a dramatically faster feedback loop when iterating on code if they are using IntelliJ IDEA Ultimate (free educational licenses are available) and the Payara Platform Tools plugin.
+
+For details, see http://preview.guides.gdcc.io/en/develop/container/dev-usage.html#intellij-idea-ultimate-and-payara-platform-tools

doc/release-notes/9763-versions-api-improvements.md

@@ -0,0 +1,8 @@
+# Improvements in the /versions API
+
+- optional pagination has been added to `/api/datasets/{id}/versions` that may be useful in datasets with a large number of versions;
+- a new flag `includeFiles` is added to both `/api/datasets/{id}/versions` and `/api/datasets/{id}/versions/{vid}` (true by default), providing an option to drop the file information from the output;
+- when files are requested to be included, some database lookup optimizations have been added to improve the performance on datasets with large numbers of files. 
+
+This is reflected in the [Dataset Versions API](https://guides.dataverse.org/en/9763-lookup-optimizations/api/native-api.html#dataset-versions-api) section of the Guide.
+

doc/release-notes/9852-files-api-extension-deaccession.md

@@ -0,0 +1,12 @@
+Extended the existing endpoints:
+
+- getVersionFiles (/api/datasets/{id}/versions/{versionId}/files)
+- getVersionFileCounts (/api/datasets/{id}/versions/{versionId}/files/counts)
+
+The above endpoints now accept a new boolean optional query parameter "includeDeaccessioned", which, if enabled, causes the endpoint to consider deaccessioned versions when searching for versions to obtain files or file counts.
+
+Additionally, a new endpoint has been developed to support version deaccessioning through API (Given a dataset and a version).
+
+- deaccessionDataset (/api/datasets/{id}/versions/{versionId}/deaccession)
+
+Finally, the DataFile API payload has been extended to add the field "friendlyType"

doc/release-notes/9907-files-api-counts-with-criteria.md

@@ -0,0 +1,11 @@
+Extended the getVersionFileCounts endpoint (/api/datasets/{id}/versions/{versionId}/files/counts) to support filtering by criteria.
+
+In particular, the endpoint now accepts the following optional criteria query parameters:
+
+- contentType
+- accessStatus
+- categoryName
+- tabularTagName
+- searchText
+
+This filtering criteria is the same as the one for the getVersionFiles endpoint.

doc/release-notes/9955-Signposting-updates.md

@@ -0,0 +1,7 @@
+This release fixes several issues (#9952, #9953, #9957) where the Signposting output did not match the Signposting specification. These changes introduce backward-incompatibility, but since Signposting support was added recently (in Dataverse 5.14 in PR #8981), we feel it's best to do this clean up and not support the old implementation that was not fully compliant with the spec.
+
+To fix #9952, we surround the license info with `<` and `>`.
+
+To fix #9953, we no longer wrap the response in a `{"status":"OK","data":{` JSON object. This has also been noted in the guides at https://dataverse-guide--9955.org.readthedocs.build/en/9955/api/native-api.html#retrieve-signposting-information
+
+To fix #9957, we corrected the mime/content type, changing it from `json+ld` to `ld+json`. For backward compatibility, we are still supporting the old one, for now.

doc/release-notes/9958-dataset-api-downloadsize-ignore-tabular-size.md

@@ -0,0 +1,9 @@
+Added a new optional query parameter "mode" to the "getDownloadSize" API endpoint ("api/datasets/{identifier}/versions/{versionId}/downloadsize").
+
+This parameter applies a filter criteria to the operation and supports the following values:
+
+- All (Default): Includes both archival and original sizes for tabular files
+
+- Archival: Includes only the archival size for tabular files 
+
+- Original: Includes only the original size for tabular files

doc/release-notes/9995-files-api-downloadsize-with-criteria-and-deaccessioned-support.md

@@ -0,0 +1,12 @@
+Extended the getDownloadSize endpoint (/api/datasets/{id}/versions/{versionId}/downloadsize), including the following new features:
+
+- The endpoint now accepts a new boolean optional query parameter "includeDeaccessioned", which, if enabled, causes the endpoint to consider deaccessioned dataset versions when searching for versions to obtain the file total download size.
+
+
+- The endpoint now supports filtering by criteria. In particular, it accepts the following optional criteria query parameters:
+
+  - contentType
+  - accessStatus
+  - categoryName
+  - tabularTagName
+  - searchText

doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv

@@ -2,6 +2,6 @@ Tool	Type	Scope	Description
 Data Explorer	explore	file	"A GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis. See the README.md file at https://github.com/scholarsportal/dataverse-data-explorer-v2 for the instructions on adding Data Explorer to your Dataverse."
 Whole Tale	explore	dataset	"A platform for the creation of reproducible research packages that allows users to launch containerized interactive analysis environments based on popular tools such as Jupyter and RStudio. Using this integration, Dataverse users can launch Jupyter and RStudio environments to analyze published datasets. For more information, see the `Whole Tale User Guide <https://wholetale.readthedocs.io/en/stable/users_guide/integration.html>`_."
 Binder	explore	dataset	Binder allows you to spin up custom computing environments in the cloud (including Jupyter notebooks) with the files from your dataset. `Installation instructions <https://github.com/data-exp-lab/girder_ythub/issues/10>`_ are in the Data Exploration Lab girder_ythub project.
-File Previewers	explore	file	"A set of tools that display the content of files - including audio, html, `Hypothes.is <https://hypothes.is/>`_ annotations, images, PDF, text, video, tabular data, spreadsheets, GeoJSON, zip, and NcML files - allowing them to be viewed without downloading the file. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/gdcc/dataverse-previewers"
+File Previewers	explore	file	"A set of tools that display the content of files - including audio, html, `Hypothes.is <https://hypothes.is/>`_ annotations, images, PDF, Markdown, text, video, tabular data, spreadsheets, GeoJSON, zip, and NcML files - allowing them to be viewed without downloading the file. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/gdcc/dataverse-previewers"
 Data Curation Tool	configure	file	"A GUI for curating data by adding labels, groups, weights and other details to assist with informed reuse. See the README.md file at https://github.com/scholarsportal/Dataverse-Data-Curation-Tool for the installation instructions."
 Ask the Data	query	file	Ask the Data is an experimental tool that allows you ask natural language questions about the data contained in Dataverse tables (tabular data). See the README.md file at https://github.com/IQSS/askdataverse/tree/main/askthedata for the instructions on adding Ask the Data to your Dataverse installation. 

doc/sphinx-guides/source/_static/installation/files/etc/init.d/solr

@@ -7,7 +7,7 @@
 
 SOLR_DIR="/usr/local/solr/solr-9.3.0"
 SOLR_COMMAND="bin/solr"
-SOLR_ARGS="-m 1g -j jetty.host=127.0.0.1"
+SOLR_ARGS="-m 1g"
 SOLR_USER=solr
 
 case $1 in

doc/sphinx-guides/source/_static/installation/files/etc/systemd/solr.service

@@ -6,7 +6,7 @@ After = syslog.target network.target remote-fs.target nss-lookup.target
 User = solr
 Type = forking
 WorkingDirectory = /usr/local/solr/solr-9.3.0
-ExecStart = /usr/local/solr/solr-9.3.0/bin/solr start -m 1g -j "jetty.host=127.0.0.1"
+ExecStart = /usr/local/solr/solr-9.3.0/bin/solr start -m 1g
 ExecStop = /usr/local/solr/solr-9.3.0/bin/solr stop
 LimitNOFILE=65000
 LimitNPROC=65000

doc/sphinx-guides/source/_static/navbarscroll.js

@@ -1,6 +1,6 @@
 /*  
     Use to fix hidden section headers behind the navbar when using links with targets
-    See: http://stackoverflow.com/questions/10732690/offsetting-an-html-anchor-to-adjust-for-fixed-header    
+    See: https://stackoverflow.com/questions/10732690/offsetting-an-html-anchor-to-adjust-for-fixed-header    
 */
 $jqTheme(document).ready(function() {
   $jqTheme('a[href*="#"]:not([href="#"])').on('click', function() {

doc/sphinx-guides/source/_templates/navbar.html

@@ -15,7 +15,7 @@
                 <span class="icon-bar"></span>
                 <span class="icon-bar"></span>
             </button>
-            <a class="navbar-brand" href="http://dataverse.org" id="dataverse-org-homepage-url">Dataverse Project</a>
+            <a class="navbar-brand" href="https://dataverse.org" id="dataverse-org-homepage-url">Dataverse Project</a>
         </div>
         <!-- Collect the nav links, forms, and other content for toggling -->
 
@@ -24,15 +24,15 @@
                 <li class="dropdown">
                     <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">About <span class="caret"></span></a>
                     <ul class="dropdown-menu">
-                        <li><a target="_blank" href="http://dataverse.org/about">About the Project</a></li>
-                        <li><a target="_blank" href="http://dataverse.org/add-data">Add Data</a></li>
-                        <li><a target="_blank" href="http://dataverse.org/blog">Blog</a></li>
-                        <li><a target="_blank" href="http://dataverse.org/presentations">Presentations</a></li>
-                        <li><a target="_blank" href="http://dataverse.org/publications">Publications</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/about">About the Project</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/add-data">Add Data</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/blog">Blog</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/presentations">Presentations</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/publications">Publications</a></li>
                     </ul>
                 </li>
                 <li>
-                    <a target="_blank" href="http://dataverse.org/">
+                    <a target="_blank" href="https://dataverse.org/">
                         Community
                     </a>
                 </li>
@@ -49,18 +49,18 @@
                 <li>
                     <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-haspopup="true" aria-expanded="false">Software <span class="caret"></span></a>
                     <ul class="dropdown-menu">
-                        <li><a target="_blank" href="http://dataverse.org/software-features">Features</a></li>
-                        <li><a target="_blank" href="http://github.com/IQSS/dataverse">Source Code</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/user/index.html">User Guide</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/installation/index.html">Installation Guide</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/api/index.html">API Guide</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/developers/index.html">Developer Guide</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/style/index.html">Style Guide</a></li>
-                        <li><a target="_blank" href="http://guides.dataverse.org/en/latest/admin/index.html">Admin Guide</a></li>
+                        <li><a target="_blank" href="https://dataverse.org/software-features">Features</a></li>
+                        <li><a target="_blank" href="https://github.com/IQSS/dataverse">Source Code</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/user/index.html">User Guide</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/installation/index.html">Installation Guide</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/api/index.html">API Guide</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/developers/index.html">Developer Guide</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/style/index.html">Style Guide</a></li>
+                        <li><a target="_blank" href="https://guides.dataverse.org/en/latest/admin/index.html">Admin Guide</a></li>
                     </ul>
                 </li>
                 <li>
-                    <a target="_blank" href="http://dataverse.org/">
+                    <a target="_blank" href="https://dataverse.org/">
                         Contact
                     </a>
                 </li>