Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Raw Json support #257

Open
owaiskazi19 opened this issue Nov 4, 2022 · 9 comments
Open

[FEATURE] Raw Json support #257

owaiskazi19 opened this issue Nov 4, 2022 · 9 comments
Labels
enhancement New feature or request

Comments

@owaiskazi19
Copy link
Member

Is your feature request related to a problem?

Current JavaClient requires TypeMapping type which requires deserializing the Json Parser and Mapper to read raw json of mapping file using TypeMapping._DESERIALIZER. This process is not documented right now.

private String getAnomalyDetectorMappings() throws IOException {
        URL url = AnomalyDetectionIndices.class.getClassLoader().getResource(ANOMALY_DETECTORS_INDEX_MAPPING_FILE);
        return Resources.toString(url, Charsets.UTF_8);
    }

JsonpMapper mapper = sdkClient._transport().jsonpMapper();
        JsonParser parser = null;
        try {
            parser = mapper
                .jsonProvider()
                .createParser(new ByteArrayInputStream(getAnomalyDetectorMappings().getBytes(StandardCharsets.UTF_8)));
        } catch (Exception e) {
            e.printStackTrace();
        }

        CreateIndexRequest request = null;
        try {
            request = new CreateIndexRequest.Builder()
                .index(ANOMALY_DETECTORS_INDEX)
                .mappings(TypeMapping._DESERIALIZER.deserialize(parser, mapper))
                .build();
        } catch (Exception e) {
            e.printStackTrace();
        }

The mapper and parser should be present in java client and thus avoiding the user to create it on their end everytime.

What solution would you like?

When creating an index, provide a type to pass the raw json file directly which can be later parse on java client side to TyepMapping

request = new CreateIndexRequest.Builder()
                .index(ANOMALY_DETECTORS_INDEX)
                .mappingFile(<json_file_here>)
                .build();

What alternatives have you considered?

A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

@owaiskazi19 owaiskazi19 added enhancement New feature or request untriaged labels Nov 4, 2022
@saratvemulapalli
Copy link
Member

Thanks for opening this up @owaiskazi19.
I do see another issue[1] asking for a similar feature.
Is this something similar?

[1] #206

@owaiskazi19
Copy link
Member Author

Thanks for opening this up @owaiskazi19. I do see another issue[1] asking for a similar feature. Is this something similar?

[1] #206

Hey @saratvemulapalli! The issue is related to bug in deserialization for CreateIndexRequest. This issue focuses on adding an enhancement for supporting raw json file.

@dblock
Copy link
Member

dblock commented Nov 9, 2022

I like this. I think we want mapping that supports files, raw JSON, streams, etc.

@penghuo
Copy link

penghuo commented Apr 28, 2023

+1. add getSourceAsString() as RestHighLevelClient.

  1. In OpenSearch SQL, the return object is consturcted by type system dynamaically, there is no predefined TDcoument type. More info. https://github.com/opensearch-project/sql/blob/main/opensearch/src/main/java/org/opensearch/sql/opensearch/response/OpenSearchResponse.java#L125
    Similar in Spark OpenSearch datasource implementation.
  2. it's possible to use Map.class to handle the dynamic type, this approach requires unnecessary deserialization.

@rursprung
Copy link
Contributor

coming from #377 (comment) (thanks @dblock for pointing me this way!): i think this would be great to use in liquibase-opensearch, but not just for mappings but for everything (index settings, search templates, etc.)

@Jai2305
Copy link
Contributor

Jai2305 commented Mar 7, 2024

I wonder if we can do something like here for reference elastic/elasticsearch-java#200

@dblock
Copy link
Member

dblock commented Mar 7, 2024

@Jai2305 Yes, do double check that there's no non-APLv2-compatible code anywhere.

@dblock
Copy link
Member

dblock commented May 3, 2024

@reta should we close this with #910 or are there things here you'd want to keep open?

@reta
Copy link
Collaborator

reta commented May 3, 2024

@reta should we close this with #910 or are there things here you'd want to keep open?

Thanks @dblock , do you mind keeping it for a bit? For the context, I've been working on opensearch-project/spring-data-opensearch#227 which directly uses this kind of feature, just because Spring Data Elasticsearch has been relying on it. Once done, we could think about having such support in opensearch-java, and if it makes sense in general. Sounds good to you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

8 participants