Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(operations): add explain endpoint for search #9832

Merged
merged 5 commits into from
Feb 14, 2024
Merged

feat(operations): add explain endpoint for search #9832

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c6eefc9
feat(operations): add search explain endpoint
RyanHolstien Feb 13, 2024
2f378f7
spotless
RyanHolstien Feb 13, 2024
0fcaaf7
review comments
RyanHolstien Feb 14, 2024
399191a
Merge branch 'master' into feat/searchExplain
RyanHolstien Feb 14, 2024
c494f7d
update other doc too
RyanHolstien Feb 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 20 additions & 19 deletions docs/authorization/policies.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,25 +94,26 @@ We currently support the following:

**Common metadata privileges** to view & modify any entity within DataHub.

| Common Privileges | Description |
|----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
| View Entity Page | Allow actor to access the entity page for the resource in the UI. If not granted, it will redirect them to an unauthorized page. |
| Edit Tags | Allow actor to add and remove tags to an asset. |
| Edit Glossary Terms | Allow actor to add and remove glossary terms to an asset. |
| Edit Owners | Allow actor to add and remove owners of an entity. |
| Edit Description | Allow actor to edit the description (documentation) of an entity. |
| Edit Links | Allow actor to edit links associated with an entity. |
| Edit Status | Allow actor to edit the status of an entity (soft deleted or not). |
| Edit Domain | Allow actor to edit the Domain of an entity. |
| Edit Deprecation | Allow actor to edit the Deprecation status of an entity. |
| Edit Assertions | Allow actor to add and remove assertions from an entity. |
| Edit All | Allow actor to edit any information about an entity. Super user privileges. Controls the ability to ingest using API when REST API Authorization is enabled. | |
| Get Timeline API[^1] | Allow actor to get the timeline of an entity via API. |
| Get Entity API[^1] | Allow actor to get an entity via API. |
| Get Timeseries Aspect API[^1] | Allow actor to get a timeseries aspect via API. |
| Get Aspect/Entity Count APIs[^1] | Allow actor to get aspect and entity counts via API. |
| Search API[^1] | Allow actor to search for entities via API. |
| Produce Platform Event API[^1] | Allow actor to ingest a platform event via API. |
| Common Privileges | Description |
|------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|
| View Entity Page | Allow actor to access the entity page for the resource in the UI. If not granted, it will redirect them to an unauthorized page. |
| Edit Tags | Allow actor to add and remove tags to an asset. |
| Edit Glossary Terms | Allow actor to add and remove glossary terms to an asset. |
| Edit Owners | Allow actor to add and remove owners of an entity. |
| Edit Description | Allow actor to edit the description (documentation) of an entity. |
| Edit Links | Allow actor to edit links associated with an entity. |
| Edit Status | Allow actor to edit the status of an entity (soft deleted or not). |
| Edit Domain | Allow actor to edit the Domain of an entity. |
| Edit Deprecation | Allow actor to edit the Deprecation status of an entity. |
| Edit Assertions | Allow actor to add and remove assertions from an entity. |
| Edit All | Allow actor to edit any information about an entity. Super user privileges. Controls the ability to ingest using API when REST API Authorization is enabled. | |
| Get Timeline API[^1] | Allow actor to get the timeline of an entity via API. |
| Get Entity API[^1] | Allow actor to get an entity via API. |
| Get Timeseries Aspect API[^1] | Allow actor to get a timeseries aspect via API. |
| Get Aspect/Entity Count APIs[^1] | Allow actor to get aspect and entity counts via API. |
| Search API[^1] | Allow actor to search for entities via API. |
| Produce Platform Event API[^1] | Allow actor to ingest a platform event via API. |
| Explain ElasticSearch Query API[^1] | Allow actor to explain an ElasticSearch query. |

[^1]: Only active if REST_API_AUTHORIZATION_ENABLED is true

Expand Down
16 changes: 16 additions & 0 deletions ...ata-io/src/main/java/com/linkedin/metadata/search/elasticsearch/ElasticSearchService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
import javax.annotation.Nullable;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.opensearch.action.explain.ExplainResponse;
import org.opensearch.action.search.SearchResponse;

@Slf4j
Expand Down Expand Up @@ -291,4 +292,19 @@ public Optional<SearchResponse> raw(@Nonnull String indexName, @Nullable String
public int maxResultSize() {
return ESUtils.MAX_RESULT_SIZE;
}

@Override
public ExplainResponse explain(
@Nonnull String query,
@Nonnull String documentId,
@Nonnull String entityName,
@Nullable Filter postFilters,
@Nullable SortCriterion sortCriterion,
@Nullable SearchFlags searchFlags,
int from,
int size,
@Nullable List<String> facets) {
return esSearchDAO.explain(
query, documentId, entityName, postFilters, sortCriterion, searchFlags, from, size, facets);
}
}
33 changes: 33 additions & 0 deletions metadata-io/src/main/java/com/linkedin/metadata/search/elasticsearch/query/ESSearchDAO.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@
import javax.annotation.Nullable;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.opensearch.action.explain.ExplainRequest;
import org.opensearch.action.explain.ExplainResponse;
import org.opensearch.action.search.SearchRequest;
import org.opensearch.action.search.SearchResponse;
import org.opensearch.client.Request;
Expand Down Expand Up @@ -463,4 +465,35 @@ private String createPointInTime(String[] indexArray, String keepAlive) {
throw new IllegalStateException("Failed to generate PointInTime Identifier.:", e);
}
}

public ExplainResponse explain(
@Nonnull String query,
@Nonnull String documentId,
@Nonnull String entityName,
@Nullable Filter postFilters,
@Nullable SortCriterion sortCriterion,
@Nullable SearchFlags searchFlags,
int from,
RyanHolstien marked this conversation as resolved.
Show resolved Hide resolved
int size,
@Nullable List<String> facets) {
EntitySpec entitySpec = entityRegistry.getEntitySpec(entityName);
Filter transformedFilters = transformFilterForEntities(postFilters, indexConvention);
final String finalQuery = query.isEmpty() ? "*" : query;
final SearchRequest searchRequest =
SearchRequestHandler.getBuilder(entitySpec, searchConfiguration, customSearchConfiguration)
.getSearchRequest(
finalQuery, transformedFilters, sortCriterion, from, size, searchFlags, facets);

ExplainRequest explainRequest = new ExplainRequest();
explainRequest
.query(searchRequest.source().query())
.id(documentId)
.index(indexConvention.getEntityIndexName(entityName));
try {
return client.explain(explainRequest, RequestOptions.DEFAULT);
} catch (IOException e) {
log.error("Failed to explain query.", e);
throw new IllegalStateException("Failed to explain query:", e);
}
}
}
7 changes: 7 additions & 0 deletions .../src/test/java/com/linkedin/metadata/search/elasticsearch/SearchDAOElasticSearchTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import static org.testng.AssertJUnit.assertNotNull;

import com.linkedin.metadata.config.search.SearchConfiguration;
import com.linkedin.metadata.models.registry.EntityRegistry;
import com.linkedin.metadata.search.query.SearchDAOTestBase;
import com.linkedin.metadata.utils.elasticsearch.IndexConvention;
import io.datahubproject.test.fixtures.search.SampleDataFixtureConfiguration;
Expand All @@ -23,11 +24,17 @@
public class SearchDAOElasticSearchTest extends SearchDAOTestBase {
@Autowired private RestHighLevelClient searchClient;
@Autowired private SearchConfiguration searchConfiguration;
@Autowired private EntityRegistry entityRegistry;

@Autowired
@Qualifier("sampleDataIndexConvention")
IndexConvention indexConvention;

@Override
protected EntityRegistry getInjectedRegistry() {
return entityRegistry;
}

@Test
public void initTest() {
assertNotNull(searchClient);
Expand Down
7 changes: 7 additions & 0 deletions ...ata-io/src/test/java/com/linkedin/metadata/search/opensearch/SearchDAOOpenSearchTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import static org.testng.AssertJUnit.assertNotNull;

import com.linkedin.metadata.config.search.SearchConfiguration;
import com.linkedin.metadata.models.registry.EntityRegistry;
import com.linkedin.metadata.search.query.SearchDAOTestBase;
import com.linkedin.metadata.utils.elasticsearch.IndexConvention;
import io.datahubproject.test.fixtures.search.SampleDataFixtureConfiguration;
Expand All @@ -23,11 +24,17 @@
public class SearchDAOOpenSearchTest extends SearchDAOTestBase {
@Autowired private RestHighLevelClient searchClient;
@Autowired private SearchConfiguration searchConfiguration;
@Autowired private EntityRegistry entityRegistry;

@Autowired
@Qualifier("sampleDataIndexConvention")
IndexConvention indexConvention;

@Override
protected EntityRegistry getInjectedRegistry() {
return entityRegistry;
}

@Test
public void initTest() {
assertNotNull(searchClient);
Expand Down
45 changes: 41 additions & 4 deletions metadata-io/src/test/java/com/linkedin/metadata/search/query/SearchDAOTestBase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
package com.linkedin.metadata.search.query;

import static com.linkedin.metadata.Constants.ELASTICSEARCH_IMPLEMENTATION_ELASTICSEARCH;
import static com.linkedin.metadata.Constants.*;
import static com.linkedin.metadata.utils.SearchUtil.AGGREGATION_SEPARATOR_CHAR;
import static org.junit.Assert.*;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertNotEquals;
import static org.testng.Assert.assertNotNull;
Expand Down Expand Up @@ -33,6 +34,7 @@
import java.util.Collections;
import java.util.List;
import java.util.Map;
import org.opensearch.action.explain.ExplainResponse;
import org.opensearch.client.RestHighLevelClient;
import org.springframework.test.context.testng.AbstractTestNGSpringContextTests;
import org.testng.annotations.Test;
Expand All @@ -45,7 +47,9 @@ public abstract class SearchDAOTestBase extends AbstractTestNGSpringContextTests

protected abstract IndexConvention getIndexConvention();

EntityRegistry _entityRegistry = new SnapshotEntityRegistry(new Snapshot());
protected abstract EntityRegistry getInjectedRegistry();

EntityRegistry entityRegistry = new SnapshotEntityRegistry(new Snapshot());

@Test
public void testTransformFilterForEntitiesNoChange() {
Expand Down Expand Up @@ -219,7 +223,7 @@ public void testTransformFilterForEntitiesWithSomeChanges() {
public void testTransformIndexIntoEntityNameSingle() {
ESSearchDAO searchDAO =
new ESSearchDAO(
_entityRegistry,
entityRegistry,
getSearchClient(),
getIndexConvention(),
false,
Expand Down Expand Up @@ -302,7 +306,7 @@ public void testTransformIndexIntoEntityNameSingle() {
public void testTransformIndexIntoEntityNameNested() {
ESSearchDAO searchDAO =
new ESSearchDAO(
_entityRegistry,
entityRegistry,
getSearchClient(),
getIndexConvention(),
false,
Expand Down Expand Up @@ -429,4 +433,37 @@ public void testTransformIndexIntoEntityNameNested() {
.setNumEntities(50);
assertEquals(searchDAO.transformIndexIntoEntityName(result), expectedResult);
}

@Test
public void testExplain() {
ESSearchDAO searchDAO =
new ESSearchDAO(
getInjectedRegistry(),
getSearchClient(),
getIndexConvention(),
false,
ELASTICSEARCH_IMPLEMENTATION_ELASTICSEARCH,
getSearchConfiguration(),
null);
ExplainResponse explainResponse =
searchDAO.explain(
"*",
"urn:li:dataset:(urn:li:dataPlatform:bigquery,bigquery-public-data.covid19_geotab_mobility_impact."
+ "ca_border_wait_times,PROD)",
DATASET_ENTITY_NAME,
null,
null,
null,
0,
10,
null);

assertNotNull(explainResponse);
assertEquals(explainResponse.getIndex(), "smpldat_datasetindex_v2");
assertEquals(
explainResponse.getId(),
"urn:li:dataset:(urn:li:dataPlatform:bigquery,bigquery-public-data.covid19_geotab_mobility_impact.ca_border_wait_times,PROD)");
assertTrue(explainResponse.isExists());
assertEquals(explainResponse.getExplanation().getValue(), 18.0f);
}
}
Loading
Loading