[v3.0.0] Deprecate OpenSearch DSL format by seankao-az · Pull Request #3367 · opensearch-project/sql

seankao-az · 2025-02-28T16:27:34Z

Description

Removes the OpenSearch DSL response format.

This PR focus only on deprecating the format, and will silent the affected legacy test cases. #3372 is created to track the task to decide which of the legacy tests are worth keeping and what to do with them.

Documentation website changes required:

response_formats.md
limitation.md
sql-ppl-api.md

possibly more

Related Issues

Resolves #3280

Check List

~~- [ ] New functionality includes testing.~~

New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.

~~- [ ] API changes companion pull request created.~~

Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sean Kao <seankao@amazon.com>

This reverts commit e3157aa. Signed-off-by: Sean Kao <seankao@amazon.com>

Signed-off-by: Sean Kao <seankao@amazon.com>

seankao-az · 2025-03-03T18:59:34Z

legacy/src/main/java/org/opensearch/sql/legacy/query/OpenSearchActionFactory.java

          sqlExpr.accept(new TermFieldRewriter());
          // migrate aggregation to query planner framework.
-          if (shouldMigrateToQueryPlan(sqlExpr, request.getFormat())) {
+          if (!bypassMigrateToQueryPlan && shouldMigrateToQueryPlan(sqlExpr)) {


Previously some test cases use Format.JSON to bypass this check and get QueryAction for SELECT (either AggregationQueryAction or DefaultQueryAction). After removing Format.JSON, those test case wouldn't work without this flag.

Signed-off-by: Sean Kao <seankao@amazon.com>

dai-chen · 2025-03-04T17:26:38Z

legacy/src/main/java/org/opensearch/sql/legacy/query/OpenSearchActionFactory.java

-    if (format == Format.JSON) {
-      return false;
-    }


Now there is no JSON format, I'm thinking should we ignore/fix the failed IT instead of adding a flag bypassMigrateToQueryPlan?

main problem trying to solve with this flag is CheckScriptContents.getScriptFieldFromQuery needs access to a SearchRequestBuilder in QueryAction, which I think QueryPlanQueryAction doesn't expose. Will double check if there's another way

I think for the tests that used this method, it's a simple fix to change the query to not have any aggregation. Doing so will avoid migrating to QueryPlanner as well.

Those are UTs testing the string operators, and the GROUP BY and ORDER BY aren't used anyway, so it should be safe to remove them.

Signed-off-by: Sean Kao <seankao@amazon.com>

penghuo · 2025-03-05T16:15:56Z

legacy/src/main/java/org/opensearch/sql/legacy/query/OpenSearchActionFactory.java

+    return create(client, sql, false);
+  }
+
+  public static QueryAction create(Client client, String sql, boolean bypassMigrateToQueryPlan)


This function only used in UT? if so, add a tag/java-doc on this function.

i removed this in latest commit. It was an attempt to fix some test cases, but it turns out changing the test cases is easier.

dai-chen

Thanks for the changes!

penghuo

Thx!

seankao31 added 4 commits February 28, 2025 08:26

mark places to change

bd434ec

Signed-off-by: Sean Kao <seankao@amazon.com>

remove dsl format from doc

613da7c

Signed-off-by: Sean Kao <seankao@amazon.com>

remove Format.JSON

43fe0fa

Signed-off-by: Sean Kao <seankao@amazon.com>

fix test case wip

e3157aa

Signed-off-by: Sean Kao <seankao@amazon.com>

seankao-az changed the title ~~Deprecate OpenSearch DSL format~~ [v3.0.0] Deprecate OpenSearch DSL format Feb 28, 2025

seankao-az added breaking v3.0.0 labels Feb 28, 2025

seankao-az force-pushed the deprecate-dsl branch from d96e854 to 7de6452 Compare February 28, 2025 23:45

seankao31 added 5 commits February 28, 2025 15:51

Revert "fix test case wip"

85e2566

This reverts commit e3157aa. Signed-off-by: Sean Kao <seankao@amazon.com>

ignore legacy test using json response format

5f0ac18

Signed-off-by: Sean Kao <seankao@amazon.com>

spotlessApply

3cbaf82

Signed-off-by: Sean Kao <seankao@amazon.com>

remove json param check in executeRequest

c74b96b

Signed-off-by: Sean Kao <seankao@amazon.com>

fix getScriptFieldFromQuery can't get requestBuilder from QueryPlan

48d512a

Signed-off-by: Sean Kao <seankao@amazon.com>

seankao-az force-pushed the deprecate-dsl branch from 7de6452 to 48d512a Compare February 28, 2025 23:51

seankao31 added 7 commits February 28, 2025 15:52

spotlessApply

d0c1ef1

Signed-off-by: Sean Kao <seankao@amazon.com>

ignore test using json

82cfce0

Signed-off-by: Sean Kao <seankao@amazon.com>

fix test case

5f38dd5

Signed-off-by: Sean Kao <seankao@amazon.com>

ignore legacy tests affected

1cdeccd

Signed-off-by: Sean Kao <seankao@amazon.com>

fix json call

3b50834

Signed-off-by: Sean Kao <seankao@amazon.com>

remove patch file

bf12b0f

Signed-off-by: Sean Kao <seankao@amazon.com>

comment

681554f

Signed-off-by: Sean Kao <seankao@amazon.com>

seankao-az mentioned this pull request Mar 3, 2025

[v3.0.0] Handle legacy test cases relying on OpenSearch DSL format #3372

Open

remove unneeded file

e449c44

Signed-off-by: Sean Kao <seankao@amazon.com>

seankao-az commented Mar 3, 2025

View reviewed changes

seankao-az marked this pull request as ready for review March 3, 2025 19:41

seankao-az requested review from dai-chen, derek-ho, joshuali925, kavithacm and ps48 as code owners March 3, 2025 19:41

seankao-az requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, forestmvey, mengweieric, noCharger, penghuo and ykmr1224 as code owners March 3, 2025 19:41

delete unused class

b10d148

Signed-off-by: Sean Kao <seankao@amazon.com>

seankao-az self-assigned this Mar 3, 2025

dai-chen reviewed Mar 4, 2025

View reviewed changes

seankao31 added 3 commits March 4, 2025 11:45

revert change in prod code for test purpose; fix test

d43a5c0

Signed-off-by: Sean Kao <seankao@amazon.com>

re-evaluate affected tests

06bea4f

Signed-off-by: Sean Kao <seankao@amazon.com>

spotlessApply

085647f

Signed-off-by: Sean Kao <seankao@amazon.com>

seankao-az requested a review from dai-chen March 4, 2025 21:43

penghuo reviewed Mar 5, 2025

View reviewed changes

seankao-az requested a review from penghuo March 6, 2025 17:31

dai-chen approved these changes Mar 6, 2025

View reviewed changes

penghuo approved these changes Mar 7, 2025

View reviewed changes

noCharger approved these changes Mar 10, 2025

View reviewed changes

noCharger merged commit f750bc6 into opensearch-project:main Mar 10, 2025
22 checks passed

xinyual mentioned this pull request Jun 11, 2025

[backport to 2.19-dev] Backport calcite prs #3752

Merged

7 tasks

LantaoJin mentioned this pull request Jan 14, 2026

[BUG] SQL API returns "unknown response format: json" when using format=json in OpenSearch 3.4.0 #5038

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v3.0.0] Deprecate OpenSearch DSL format#3367

[v3.0.0] Deprecate OpenSearch DSL format#3367
noCharger merged 21 commits intoopensearch-project:mainfrom
seankao31:deprecate-dsl

seankao-az commented Feb 28, 2025 •

edited

Loading

Uh oh!

seankao-az Mar 3, 2025 •

edited

Loading

Uh oh!

dai-chen Mar 4, 2025

Uh oh!

seankao-az Mar 4, 2025

Uh oh!

seankao-az Mar 4, 2025

Uh oh!

penghuo Mar 5, 2025

Uh oh!

seankao-az Mar 5, 2025

Uh oh!

dai-chen left a comment

Uh oh!

penghuo left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Comments

Conversation

seankao-az commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

seankao-az Mar 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dai-chen Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

seankao-az Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

seankao-az Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

penghuo Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

seankao-az Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

dai-chen left a comment

Choose a reason for hiding this comment

Uh oh!

penghuo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Comments

seankao-az commented Feb 28, 2025 •

edited

Loading

seankao-az Mar 3, 2025 •

edited

Loading