-
Notifications
You must be signed in to change notification settings - Fork 492
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #7579 from IQSS/7084-crawlable-file-access
7084 crawlable file access
- Loading branch information
Showing
14 changed files
with
566 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
## Release Highlights | ||
|
||
### A new file access API | ||
|
||
A new api offers *crawlable* access view of the folders and files within a datset: | ||
|
||
``` | ||
/api/datasets/<dataset id>/dirindex/ | ||
``` | ||
|
||
will output a simple html listing, based on the standard Apache | ||
directory index, with Access API download links for individual files, | ||
and recursive calls to the API above for sub-folders. (See the | ||
documentation entry in the guides for more information). | ||
|
||
Using this API, ``wget --recursive`` (or similar crawling client) can | ||
be used to download all the files in a dataset, preserving the file | ||
names and folder structure; without having to use the download-as-zip | ||
API. In addition to being faster (zipping is a relatively | ||
resource-intensive operation on the server side), this process can be | ||
restarted if interrupted (with ``wget --continue`` or equivalent) - | ||
unlike zipped multi-file downloads that always have to start from the | ||
beginning. | ||
|
||
On a system that uses S3 with download redirects, the individual file | ||
downloads will be handled by S3 directly, without having to be proxied | ||
through the Dataverse application. | ||
|
||
|
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.