Security improvements #126

thalurur · 2021-08-24T03:43:39Z

Issue #, if available:
#75

Description of changes:

Filter data in the Explain and Add policy APIs when there are wild cards in input
- Instead of returning everything or nothing when requested for "*" in explain, return only indices that the user has permission to manage
Prevent chained transport action calls of IM
- This is to prevent requiring chained permissions for each API
Fixing bug - not deleting rollup/policies that are not rollup/policies

CheckList:
[ X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ng chained permission requirement

codecov-commenter · 2021-08-24T04:14:20Z

Codecov Report

Merging #126 (5fd97a2) into main (c081669) will decrease coverage by 0.26%.
The diff coverage is 46.90%.

@@             Coverage Diff              @@
##               main     #126      +/-   ##
============================================
- Coverage     75.46%   75.19%   -0.27%     
- Complexity     1898     1902       +4     
============================================
  Files           262      262              
  Lines         10934    11042     +108     
  Branches       1734     1737       +3     
============================================
+ Hits           8251     8303      +52     
- Misses         1727     1788      +61     
+ Partials        956      951       -5

Impacted Files	Coverage Δ
...transport/action/explain/TransportExplainAction.kt	`73.83% <24.13%> (-4.37%)`	⬇️
...form/action/index/TransportIndexTransformAction.kt	`59.57% <37.50%> (-0.90%)`	⬇️
...action/changepolicy/TransportChangePolicyAction.kt	`60.12% <40.00%> (-0.89%)`	⬇️
.../rollup/action/index/TransportIndexRollupAction.kt	`67.14% <40.00%> (-2.22%)`	⬇️
...sport/action/addpolicy/TransportAddPolicyAction.kt	`58.27% <41.66%> (-1.10%)`	⬇️
...sport/action/getpolicy/TransportGetPolicyAction.kt	`70.27% <50.00%> (-4.73%)`	⬇️
...ollup/action/delete/TransportDeleteRollupAction.kt	`70.27% <50.00%> (+29.09%)`	⬆️
...ment/rollup/action/get/TransportGetRollupAction.kt	`62.50% <50.00%> (-4.17%)`	⬇️
...ransform/action/get/TransportGetTransformAction.kt	`55.00% <50.00%> (-2.15%)`	⬇️
.../rollup/action/start/TransportStartRollupAction.kt	`68.67% <53.33%> (+1.56%)`	⬆️
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c081669...5fd97a2. Read the comment docs.

…ain APIs

dbbaughe · 2021-08-25T17:56:29Z

...in/kotlin/org/opensearch/indexmanagement/rollup/action/delete/TransportDeleteRollupAction.kt

+                        try {
+                            rollup = parseRollup(response, xContentRegistry)
+                        } catch (e: IllegalArgumentException) {
+                            actionListener.onFailure(OpenSearchStatusException("Rollup not found", RestStatus.NOT_FOUND))


Does it always throw IAE? Can it throw a parsing exception instead?
Is there any way to be more explicit? What if it is a rollup document and there was just a bug w/ the parsing and some bwc issue?

I am considering cases where its not a valid rollup, not other parsing issues. We throw IAE on all our parse method if there are fields present in the document that are not expected

Understood, I guess I am referring to -> what if there are job documents w/ similar field keys and so it does in fact go and try to parse the xcontent but then fails there instead of in the else statement.

in that it will be an uncaught exception and throw 500 status error with "parsing_exception" or any other exception that can be thrown during parsing.

I can catch all Exceptions and not delete if we are unsure if that makes more sense

dbbaughe · 2021-08-25T17:58:13Z

...in/kotlin/org/opensearch/indexmanagement/rollup/action/delete/TransportDeleteRollupAction.kt

@@ -100,12 +101,18 @@ class TransportDeleteRollupAction @Inject constructor(
                object : ActionListener<GetResponse> {
                    override fun onResponse(response: GetResponse) {
                        if (!response.isExists) {
-                            actionListener.onFailure(OpenSearchStatusException("Rollup ${request.id()} is not found", RestStatus.NOT_FOUND))


Any reason we want to remove this?

no specific reason, just didn't see a need to return the id again in response

Got it, technically people shouldn't really rely on the text of the response.. but just in case someone does, if we don't have a real reason for removing it then it is probably better to not change it.

got it sure, I can keep continue to keep it

Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

dbbaughe · 2021-08-25T21:02:28Z

...in/kotlin/org/opensearch/indexmanagement/rollup/action/delete/TransportDeleteRollupAction.kt

@@ -84,7 +85,7 @@ class TransportDeleteRollupAction @Inject constructor(

        fun start() {
            client.threadPool().threadContext.stashContext().use {
-                if (!filterByEnabled || user == null) {
+                if (user == null) {


I saw we completely removed the if checks in the policy delete one, but in here we still keep the user check. Why the difference?

sorry, my bad let me correct

Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

* Add integTest script to the repo (#94) Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Removing Usages of Action Get Call and using listeners (#100) Signed-off-by: Aditya Jindal <aditjind@amazon.com> * Enhance ISM template (#105) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Explain response still use old opendistro policy id (#109) * Explain response still use old opendistro policy id * Use hardcoded policyid setting in tests for explain response * Trying to fix flaky tests * Storing user information as part of the job when security plugin is installed (#113) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * ISM/Notification channel support (#117) * Updates NAME of transport actions * Upgrades Kotlin version, updates dependencies on other OS plugins, adds notification plugin as a test resource and includes it in test clusters * Adds support for Channels in error notifications and notification actions * Adds support for sending notifications to channels * Adds support for publishing notifications to the legacy destinations through the Notification plugin and some cleanup * Removes notification alerting jar dependency * Adds compile only dep on commons codec for digest utils sha1 method in ism rollup * Updates Error Notification to make channel/destination nullable, and adds helper methods for publish calls * Constructs URL for legacy custom webhook Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Fixes Feature enum and dep Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Trying something else Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Addresses comments Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Storing user object in all APIs and enabling filter of response based on user (#115) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Upgrade dependencies to 1.1 and build snapshot by default. (#121) Signed-off-by: dblock <dblock@amazon.com> * Security improvements (#126) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Removes support for notification plugin (#136) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Updating security filtering logic (#137) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Release note for 1.1.0.0 release. (#139) * Release note for 1.1.0.0 release. Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Correct copyright notices * Uses published daily snapshot dependencies (#141) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * License header check (#142) * Provide default copyright header using IDE feature Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Address #103 history write index is rolled over even if the history indices are disabled * Removed integtest.sh. (#148) Signed-off-by: dblock <dblock@dblock.org> * Adds mavenLocal back to repositories (#158) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Making snapshot name to scripted input in template (#77) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Fix issues with security changes in rollup runnner (#161) * Updates index management version to 1.2 (#157) * Updates index management version to 1.2 * Updates job scheduler snapshot to 1.2 in test resources Signed-off-by: Robert Downs <downsrob@amazon.com> * Adds setting to search all rollup jobs on a target index (#165) * Adds cluster setting to search all rollup jobs Signed-off-by: Clay Downs <downsrob@amazon.com> * Adds implementation for the delay feature in rollup jobs (#147) * Adds delay implementation for rollup jobs * Removes non-continuous delay implementation * Adds additional rollup delay tests Signed-off-by: Clay Downs <downsrob@amazon.com> * Updates testCompile mockito version, adds AwaitsFix annotation to MetadataRegressionIT tests (#168) * Updates testCompile mockito version to match OpenSearch changes * AwaitsFix on the failing MetadataRegressionIT tests Signed-off-by: Robert Downs <downsrob@amazon.com> * Adds cluster setting to configure index state management jitter (#153) * Adds jitter cluster setting, sets jitter to 0 for ISM tests Signed-off-by: Clay Downs <downsrob@amazon.com> * Allows out of band rollovers on an index without causing ISM to fail (#180) * Allows out of band rollovers on an index without causing ISM to fail Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Fixes detekt issue Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Remove policy API on read only indices (#182) * In explain API not showing the total count to all users (#185) Co-authored-by: Peter Zhu <zhujiaxi@amazon.com> Co-authored-by: Aditya Jindal <13850971+aditjind@users.noreply.github.com> Co-authored-by: Bowen Lan <62091230+bowenlan-amzn@users.noreply.github.com> Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com> Co-authored-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>

Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

Preventing transport action calling another transport action - creati…

ad68f9f

…ng chained permission requirement

thalurur requested review from dbbaughe and bowenlan-amzn August 24, 2021 04:27

Filtering the indices based on user permission in add policy and expl…

e7a35bf

…ain APIs

thalurur force-pushed the main branch from d346845 to e7a35bf Compare August 25, 2021 16:40

dbbaughe reviewed Aug 25, 2021

View reviewed changes

PR comments

95bb078

Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

thalurur force-pushed the main branch from 2998260 to 95bb078 Compare August 25, 2021 20:56

dbbaughe reviewed Aug 25, 2021

View reviewed changes

thalurur added 2 commits August 25, 2021 14:06

PR comments

137c832

Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

Correct RollupDelete test

5fd97a2

thalurur force-pushed the main branch from d401f96 to 5fd97a2 Compare August 25, 2021 21:50

dbbaughe approved these changes Aug 26, 2021

View reviewed changes

thalurur merged commit 5c41b04 into opensearch-project:main Aug 26, 2021

bowenlan-amzn added enhancement New request feature and removed enhancement New request labels Sep 3, 2021

downsrob mentioned this pull request Mar 8, 2022

Fixes test failures with security enabled #292

Merged

1 task

wuychn pushed a commit to ochprince/index-management that referenced this pull request Mar 16, 2023

Security improvements (opensearch-project#126)

11a1f73

Signed-off-by: Ravi Thaluru <ravi1092@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Security improvements #126

Security improvements #126

thalurur commented Aug 24, 2021 •

edited

Loading

codecov-commenter commented Aug 24, 2021 •

edited

Loading

dbbaughe Aug 25, 2021

thalurur Aug 25, 2021

dbbaughe Aug 25, 2021

thalurur Aug 25, 2021

dbbaughe Aug 25, 2021

thalurur Aug 25, 2021

dbbaughe Aug 25, 2021

thalurur Aug 25, 2021

dbbaughe Aug 25, 2021

thalurur Aug 25, 2021

Security improvements #126

Security improvements #126

Conversation

thalurur commented Aug 24, 2021 • edited Loading

codecov-commenter commented Aug 24, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thalurur commented Aug 24, 2021 •

edited

Loading

codecov-commenter commented Aug 24, 2021 •

edited

Loading