Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

standardize image url #10855

Merged
merged 7 commits into from
Sep 24, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions doc/release-notes/10831-standardize-image-url-of-search-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
Search API (/api/search) response will now include new image_url format for the Datafile and Dataverse logo.
Note to release note writer: this supersedes the release note 10810-search-api-payload-extensions.md

For Dataverse:

- "image_url" (optional)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the image_url is always returned, even though the URL returns a 404 for files and a 204 for collections, then it is not an optional field. Anyway, check my final review comment, where I suggest not adding the fields to the payload in these cases.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@GPortas what if when no image is available the endpoint just returns image_url: null ?
In that way we will know that image_url could be either a string(image exists) or null.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed image_url if none exists


```javascript
"items": [
{
"name": "Darwin's Finches",
...
"image_url":"/api/access/dvCardImage/{identifier}"
(etc, etc)
```
For DataFile:
- "image_url" (optional)
```javascript
"items": [
{
"name": "test.txt",
...
"image_url":"/api/access/datafile/{identifier}?imageThumb=true"
(etc, etc)
```
12 changes: 5 additions & 7 deletions doc/sphinx-guides/source/api/search.rst
Original file line number Diff line number Diff line change
@@ -61,7 +61,7 @@ https://demo.dataverse.org/api/search?q=trees
"name":"Trees",
"type":"dataverse",
"url":"https://demo.dataverse.org/dataverse/trees",
"image_url":"data:image/png;base64,iVBORw0...",
"image_url":"https://demo.dataverse.org/api/access/dvCardImage/1",
"identifier":"trees",
"description":"A tree dataverse with some birds",
"published_at":"2016-05-10T12:53:38Z",
@@ -76,7 +76,7 @@ https://demo.dataverse.org/api/search?q=trees
"name":"Chestnut Trees",
"type":"dataverse",
"url":"https://demo.dataverse.org/dataverse/chestnuttrees",
"image_url":"data:image/png;base64,iVBORw0...",
"image_url":"https://demo.dataverse.org/api/access/dvCardImage/2",
"identifier":"chestnuttrees",
"description":"A dataverse with chestnut trees and an oriole",
"published_at":"2016-05-10T12:52:38Z",
@@ -91,7 +91,7 @@ https://demo.dataverse.org/api/search?q=trees
"name":"trees.png",
"type":"file",
"url":"https://demo.dataverse.org/api/access/datafile/12",
"image_url":"data:image/png;base64,iVBORw0...",
"image_url":"https://demo.dataverse.org/api/access/datafile/12?imageThumb=true",
"file_id":"12",
"description":"",
"published_at":"2016-05-10T12:53:39Z",
@@ -113,7 +113,7 @@ https://demo.dataverse.org/api/search?q=trees
"name":"Birds",
"type":"dataverse",
"url":"https://demo.dataverse.org/dataverse/birds",
"image_url":"data:image/png;base64,iVBORw0...",
"image_url":"https://demo.dataverse.org/api/access/dvCardImage/3",
"identifier":"birds",
"description":"A bird Dataverse collection with some trees",
"published_at":"2016-05-10T12:57:27Z",
@@ -173,8 +173,6 @@ https://demo.dataverse.org/api/search?q=trees
}
}
Note that the image_url field, if exists, will be returned as a regular URL for Datasets, while for Files and Dataverses, it will be returned as a Base64 URL. We plan to standardize this behavior so that the field always returns a regular URL. (See: https://github.com/IQSS/dataverse/issues/10831)

.. _advancedsearch-example:

Advanced Search Examples
@@ -202,7 +200,7 @@ In this example, ``show_relevance=true`` matches per field are shown. Available
"name":"Finches",
"type":"dataverse",
"url":"https://demo.dataverse.org/dataverse/finches",
"image_url":"data:image/png;base64,iVBORw0...",
"image_url":"https://demo.dataverse.org/api/access/dvCardImage/2",
"identifier":"finches",
"description":"A Dataverse collection with finches",
"published_at":"2016-05-10T12:57:38Z",
Original file line number Diff line number Diff line change
@@ -361,7 +361,15 @@ public String getDataverseLogoThumbnailAsBase64ById(Long dvId) {
}
return null;
}


public String getDataverseLogoThumbnailAsUrl(Long dvId) {
File dataverseLogoFile = getLogoById(dvId);
if (dataverseLogoFile != null && dataverseLogoFile.exists()) {
return SystemConfig.getDataverseSiteUrlStatic() + "/api/access/dvCardImage/" + dvId;
}
return null;
}

private File getLogo(Dataverse dataverse) {
if (dataverse.getId() == null) {
return null;
Original file line number Diff line number Diff line change
@@ -10,6 +10,7 @@
import edu.harvard.iq.dataverse.dataaccess.StorageIO;
import edu.harvard.iq.dataverse.dataset.DatasetUtil;
import edu.harvard.iq.dataverse.search.SolrSearchResult;
import edu.harvard.iq.dataverse.util.FileUtil;
import edu.harvard.iq.dataverse.util.SystemConfig;

import java.io.IOException;
@@ -48,6 +49,19 @@ public class ThumbnailServiceWrapper implements java.io.Serializable {
private Map<Long, DvObject> dvobjectViewMap = new HashMap<>();
private Map<Long, Boolean> hasThumbMap = new HashMap<>();

public String getFileCardImageAsUrl(SolrSearchResult result) {
DataFile dataFile = result != null && result.getEntity() != null ? ((DataFile) result.getEntity()) : null;
if (dataFile == null || result.isHarvested()
|| !isThumbnailAvailable(dataFile)
|| dataFile.isRestricted()
|| !dataFile.isReleased()
|| FileUtil.isActivelyEmbargoed(dataFile)
|| FileUtil.isRetentionExpired(dataFile)) {
return null;
}
return SystemConfig.getDataverseSiteUrlStatic() + "/api/access/datafile/" + dataFile.getId() + "?imageThumb=true";
}

// it's the responsibility of the user - to make sure the search result
// passed to this method is of the Datafile type!
public String getFileCardImageAsBase64Url(SolrSearchResult result) {
@@ -208,7 +222,13 @@ public String getDatasetCardImageAsUrl(Dataset dataset, Long versionId, boolean
public String getDataverseCardImageAsBase64Url(SolrSearchResult result) {
return dataverseService.getDataverseLogoThumbnailAsBase64ById(result.getEntityId());
}


// it's the responsibility of the user - to make sure the search result
// passed to this method is of the Dataverse type!
public String getDataverseCardImageAsUrl(SolrSearchResult result) {
return dataverseService.getDataverseLogoThumbnailAsUrl(result.getEntityId());
}

public void resetObjectMaps() {
dvobjectThumbnailsMap = new HashMap<>();
dvobjectViewMap = new HashMap<>();
Original file line number Diff line number Diff line change
@@ -593,7 +593,7 @@ public SolrQueryResponse search(
solrSearchResult.setDataverseAffiliation(dataverseAffiliation);
solrSearchResult.setDataverseParentAlias(dataverseParentAlias);
solrSearchResult.setDataverseParentName(dataverseParentName);
solrSearchResult.setImageUrl(thumbnailServiceWrapper.getDataverseCardImageAsBase64Url(solrSearchResult));
solrSearchResult.setImageUrl(thumbnailServiceWrapper.getDataverseCardImageAsUrl(solrSearchResult));
/**
* @todo Expose this API URL after "dvs" is changed to
* "dataverses". Also, is an API token required for published
@@ -652,7 +652,7 @@ public SolrQueryResponse search(
}
solrSearchResult.setHtmlUrl(baseUrl + "/dataset.xhtml?persistentId=" + parentGlobalId);
solrSearchResult.setDownloadUrl(baseUrl + "/api/access/datafile/" + entityid);
solrSearchResult.setImageUrl(thumbnailServiceWrapper.getFileCardImageAsBase64Url(solrSearchResult));
solrSearchResult.setImageUrl(thumbnailServiceWrapper.getFileCardImageAsUrl(solrSearchResult));
/**
* @todo We are not yet setting the API URL for files because
* not all files have metadata. Only subsettable files (those
64 changes: 64 additions & 0 deletions src/test/java/edu/harvard/iq/dataverse/api/SearchIT.java
Original file line number Diff line number Diff line change
@@ -1283,4 +1283,68 @@ public void tearDownDataverse() {
public static void cleanup() {
}

@Test
public void testSearchFilesAndUrlImages() {
Response createUser = UtilIT.createRandomUser();
createUser.prettyPrint();
String username = UtilIT.getUsernameFromResponse(createUser);
String apiToken = UtilIT.getApiTokenFromResponse(createUser);

Response createDataverseResponse = UtilIT.createRandomDataverse(apiToken);
createDataverseResponse.prettyPrint();
String dataverseAlias = UtilIT.getAliasFromResponse(createDataverseResponse);

Response createDatasetResponse = UtilIT.createRandomDatasetViaNativeApi(dataverseAlias, apiToken);
createDatasetResponse.prettyPrint();
Integer datasetId = UtilIT.getDatasetIdFromResponse(createDatasetResponse);
System.out.println("id: " + datasetId);
String datasetPid = JsonPath.from(createDatasetResponse.getBody().asString()).getString("data.persistentId");
System.out.println("datasetPid: " + datasetPid);

String pathToFile = "src/main/webapp/resources/images/dataverseproject.png";
Response uploadImage = UtilIT.uploadFileViaNative(datasetId.toString(), pathToFile, apiToken);
uploadImage.prettyPrint();
uploadImage.then().assertThat()
.statusCode(200);
pathToFile = "src/main/webapp/resources/js/mydata.js";
Response uploadFile = UtilIT.uploadFileViaNative(datasetId.toString(), pathToFile, apiToken);
uploadImage.prettyPrint();
uploadImage.then().assertThat()
.statusCode(200);

Response publishDataverse = UtilIT.publishDataverseViaSword(dataverseAlias, apiToken);
publishDataverse.prettyPrint();
publishDataverse.then().assertThat()
.statusCode(OK.getStatusCode());
Response publishDataset = UtilIT.publishDatasetViaNativeApi(datasetId, "major", apiToken);
publishDataset.prettyPrint();
publishDataset.then().assertThat()
.statusCode(OK.getStatusCode());

Response searchResp = UtilIT.search("dataverseproject", apiToken);
searchResp.prettyPrint();
searchResp.then().assertThat()
.statusCode(OK.getStatusCode())
.body("data.items[0].type", CoreMatchers.is("file"))
.body("data.items[0].file_content_type", CoreMatchers.is("image/png"))
.body("data.items[0].url", CoreMatchers.containsString("/api/access/datafile/"))
.body("data.items[0].image_url", CoreMatchers.containsString("/api/access/datafile/"))
.body("data.items[0].image_url", CoreMatchers.containsString("imageThumb=true"));

searchResp = UtilIT.search(dataverseAlias, apiToken);
searchResp.prettyPrint();
searchResp.then().assertThat()
.statusCode(OK.getStatusCode())
.body("data.items[0].type", CoreMatchers.is("dataverse"))
.body("data.items[0].url", CoreMatchers.containsString("/dataverse/"))
.body("data.items[0]", CoreMatchers.not(CoreMatchers.hasItem("url_image")));

searchResp = UtilIT.search("mydata", apiToken);
searchResp.prettyPrint();
searchResp.then().assertThat()
.statusCode(OK.getStatusCode())
.body("data.items[0].type", CoreMatchers.is("file"))
.body("data.items[0].url", CoreMatchers.containsString("/datafile/"))
.body("data.items[0]", CoreMatchers.not(CoreMatchers.hasItem("url_image")));
}
}