Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Block execution engine settings in sql query settings API and add more unit tests #2407

Merged
merged 1 commit into from
Nov 1, 2023

Conversation

vmmusings
Copy link
Member

@vmmusings vmmusings commented Nov 1, 2023

Description

Currently SQL Query Settings API acts as a backdoor for changing cluster settings specific to sql engine. We no longer require this for the new settings introduced for spark execution engine.

New Settings Introduced.

plugins.query.datasources.encryption.masterkey
plugins.query.datasources.uri.hosts.denylist
plugins.query.executionengine.spark.config
plugins.query.executionengine.spark.session.limit
plugins.query.executionengine.spark.refresh_job.limit
plugins.query.executionengine.spark.session.index.ttl
plugins.query.executionengine.spark.result.index.ttl
plugins.query.executionengine.spark.auto_index_management.enabled

We also need to rethink the use cases of this API and deprecate if not required.[https://github.com//issues/2409]

This PR covers blocking execution engine settings.

Query Settings API Result before and after:

PUT {{baseUrl}}/_plugins/_query/settings
content-type: application/json

{
  "transient" : {
    "plugins.sql.enabled" : "false",
    "plugins.query.executionengine.spark.config" : "--conf spark.driverEnv.executors=2"
  }
}
{
  "acknowledged": true,
  "persistent": {},
  "transient": {
    "plugins": {
      "query": {
        "executionengine": {
          "spark": {
            "config": "--conf spark.driverEnv.executors=2"
          }
        }
      },
      "sql": {
        "enabled": "false"
      }
    }
  }
}

After the change.

PUT {{baseUrl}}/_plugins/_query/settings
content-type: application/json

{
  "transient" : {
    "plugins.sql.enabled" : "false",
    "plugins.query.executionengine.spark.config" : "--conf spark.driverEnv.executors=2",
    "plugins.query.datasources.uri.hosts.denylist": "127.001.1, 123.100.1"
  }
}

Response:

{
  "acknowledged": true,
  "persistent": {
    "plugins": {
      "sql": {
        "enabled": "false"
      }
    }
  },
  "transient": {}
}

Issues Resolved

#2382

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@vmmusings vmmusings changed the title Block settings in sql query settings API and add more unit tests Block execution engine settings in sql query settings API and add more unit tests Nov 1, 2023
Copy link

codecov bot commented Nov 1, 2023

Codecov Report

Merging #2407 (c161e77) into main (6cd2db0) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #2407   +/-   ##
=========================================
  Coverage     95.54%   95.54%           
  Complexity     4985     4985           
=========================================
  Files           478      478           
  Lines         13883    13883           
  Branches        931      931           
=========================================
  Hits          13265    13265           
  Misses          598      598           
  Partials         20       20           
Flag Coverage Δ
sql-engine 95.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Assertions.assertEquals(EMRS_APPLICATION_ID, startJobRunRequest.getApplicationId());
Assertions.assertEquals(EMRS_EXECUTION_ROLE, startJobRunRequest.getExecutionRoleArn());
Assertions.assertEquals(
ENTRY_POINT_START_JAR, startJobRunRequest.getJobDriver().getSparkSubmit().getEntryPoint());
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is specifically to assert the entry point class jar.

kavithacm
kavithacm previously approved these changes Nov 1, 2023
Copy link
Collaborator

@kavithacm kavithacm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
@vmmusings vmmusings merged commit 2f2ecd2 into opensearch-project:main Nov 1, 2023
22 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Nov 1, 2023
Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
(cherry picked from commit 2f2ecd2)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot pushed a commit that referenced this pull request Nov 1, 2023
Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
(cherry picked from commit 2f2ecd2)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
vmmusings pushed a commit that referenced this pull request Nov 2, 2023
…) (#2412)

(cherry picked from commit 2f2ecd2)

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
vmmusings pushed a commit that referenced this pull request Nov 2, 2023
…) (#2411)

(cherry picked from commit 2f2ecd2)

Signed-off-by: Vamsi Manohar <reddyvam@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023
mengweieric added a commit to mengweieric/sql that referenced this pull request Nov 8, 2023
vmmusings added a commit to mengweieric/sql that referenced this pull request Nov 13, 2023
vmmusings added a commit that referenced this pull request Nov 13, 2023
* Revert "Add more metrics and handle emr exception message (#2422) (#2426)"

This reverts commit b57f7cc.

* Revert "Block settings in sql query settings API and add more unit tests (#2407) (#2412)"

This reverts commit 3024737.

* Revert "Added session, statement, emrjob metrics to sql stats api (#2398) (#2400)"

This reverts commit 6e17ae6.

* Revert "Redefine Drop Index as logical delete (#2386) (#2397)"

This reverts commit e939bb6.

* Revert "add concurrent limit on datasource and sessions (#2390) (#2395)"

This reverts commit deb3ccf.

* Revert "Add Flint Index Purging Logic (#2372) (#2389)"

This reverts commit dd48b9b.

* Revert "Refactoring for tags usage in test files and also added explicit denly list setting. (#2383) (#2385)"

This reverts commit 37e010f.

* Revert "Enable session by default (#2373) (#2375)"

This reverts commit 7d95e4c.

* Revert "Create new session if client provided session is invalid (#2368) (#2371)"

This reverts commit 5ab7858.

* Revert "Add where clause support in create statement (#2366) (#2370)"

This reverts commit b620a56.

* Revert "create new session if current session not ready (#2363) (#2365)"

This reverts commit 5d07281.

* Revert "Handle Describe,Refresh and Show Queries Properly (#2357) (#2362)"

This reverts commit 16e2f30.

* Revert "Add Session limitation (#2354) (#2359)"

This reverts commit 0f334f8.

* Revert "Bug Fix, support cancel query in running state (#2351) (#2353)"

This reverts commit 9a40591.

* Revert "Fix bug, using basic instead of basicauth (#2342) (#2355)"

This reverts commit e4827a5.

* Revert "Add missing tags and MV support (#2336) (#2346)"

This reverts commit 8791bb0.

* Revert "[Backport 2.x] deprecated job-metadata-index (#2340) (#2343)"

This reverts commit bea432c.

* Revert "Integration with REPL Spark job (#2327) (#2338)"

This reverts commit 58a5ae5.

* Revert "Implement patch API for datasources (#2273) (#2329)"

This reverts commit 4c151fe.

* Revert "Add sessionId parameters for create async query API (#2312) (#2324)"

This reverts commit 3d1a376.

* Revert "Add Statement (#2294) (#2318) (#2319)"

This reverts commit b3c2e94.

* Revert "Upgrade json (#2307) (#2314)"

This reverts commit 6c65bb4.

* Revert "Minor Refactoring (#2308) (#2317)"

This reverts commit 051cc4f.

* Revert "add InteractiveSession and SessionManager (#2290) (#2293) (#2315)"

This reverts commit 6ac197b.

---------

Co-authored-by: Vamsi Manohar <reddyvam@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants