Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated changes to security #11

Open
wants to merge 32 commits into
base: pitsegmentsrest
Choose a base branch
from
Open

Updated changes to security #11

wants to merge 32 commits into from

Conversation

bharath-techie
Copy link
Owner

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dreamer-89 and others added 20 commits August 9, 2022 09:58
@dreamer-89
Add IndexShard#getLatestReplicationCheckpoint behind segrep enable fe…
a120dd2 
…ature flag (opensearch-project#4163)

* Add IndexShard#getLatestReplicationCheckpoint behind segrep enable feature flag

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comment. Move tests to SegmentReplicationIndexShardTests

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Add segrep enbaled index settings in TargetServiceTests, SourceHandlerTests

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@navneet1v
Refactor the src and test of GeoBoundsAggregation on GeoPoint from se…
4751f3b 
…rver folder to geo module.

Refactors the GeoBoundsAggregation for geo_point types from the core server 
to the geo module.

Signed-off-by: Navneet Verma <navneev@amazon.com>
@sachinpkale
[Remote Store] Add validator that forces segment replication type bef…
1c3f034 
…ore enabling remote store (opensearch-project#4175)

* Add validator that forces segment replication type before enabling remote store

Signed-off-by: Sachin Kale <kalsac@amazon.com>
@mch2
[Segment Replication] Fix OngoingSegmentReplications to key by alloca…
7dad063 
…tion ID instead of DiscoveryNode. (opensearch-project#4182)

* Fix OngoingSegmentReplications to key by allocation ID instead of DiscoveryNode.

This change fixes segrep to work with multiple shards per node by keying ongoing replications on
allocation ID.  This also updates cancel methods to ensure state is properly cleared on shard cancel.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Clean up cancel methods.

Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>
@sachinpkale
[Remote Store] Change remote_store setting validation message to make…
8ccedd8 
… it more clear (opensearch-project#4199)

* Change remote_store setting validation message to make it more clear

Signed-off-by: Sachin Kale <kalsac@amazon.com>
@dreamer-89 @Poojita-Raj
[Bug] [Segment Replication] Update store metadata recovery diff logic…
b9a64cb 
… to ignore missing files causing exception (opensearch-project#4185)

* Update Store for segment replication dif

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* Update recoveryDiff logic to ingore missing files causing exception on replica during copy

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Signed-off-by: Poojita Raj <poojiraj@amazon.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Co-authored-by: Poojita Raj <poojiraj@amazon.com>
@sachinpkale
[Remote Store] Add RemoteSegmentStoreDirectory to interact with remot…
5f2e66b 
…e segment store (opensearch-project#4020)

* Add RemoteSegmentStoreDirectory to interact with remote segment store

Signed-off-by: Sachin Kale <kalsac@amazon.com>
@reta
Update Gradle to 7.5.1 (opensearch-project#4211)
0dbbd72 
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@dependabot
Bump com.diffplug.spotless from 6.9.0 to 6.9.1 (opensearch-project#4210)
46d7a47 
Bumps com.diffplug.spotless from 6.9.0 to 6.9.1.

---
updated-dependencies:
- dependency-name: com.diffplug.spotless
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@Rishikesh1159
[Segment Replication] Adding PrimaryMode check before publishing chec…
96bfd1f 
…kpoint and processing a received checkpoint. (opensearch-project#4157)

* Adding PrimaryMode check before publishing checkpoint.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Applying spotless check

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Moving segrep specific tests to SegmentReplicationIndexShardTests.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Adding logic and tests for rejecting checkpoints if shard is in PrimaryMode.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Applying ./gradlew :server:spotlessApply.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Applying ./gradlew :server:spotlessApply

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Changing log level to warn in shouldProcessCheckpoint() of IndexShard.java class.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

* Removing unnecessary lazy logging in shouldProcessCheckpoint().

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
@yyyogev
Set analyzer to regex query string search (opensearch-project#3967)
ea4cfcc 
Sets analyzer to regex query string search

Signed-off-by: yyyogev <yogev.metzuyanim@logz.io>
@nknize
[Upgrade] to lucene-9.4.0-snapshot-ddf0d0a (opensearch-project#4183)
411321e 
Upgrades to lucene-9.4.0-snapshot-ddf0d0a by refactoring to TermOrdValComparator 
as a top level class.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@pgtgrly
Adding documentation about creation of uber-JARs (opensearch-project#…
a081f2f 
…3785)

* Adding documentation about creation of uber-JARs

Signed-off-by: Pranav Garg <pranavgarg@gmail.com>

* Fixing linelint error

Signed-off-by: Pranav Garg <pranavgarg@gmail.com>

* Comprehensive changes

Signed-off-by: Pranav Garg <pranavgarg@gmail.com>

* Adding PR changes

Signed-off-by: Pranav Garg <pranavgarg@gmail.com>

* PR changes

Signed-off-by: Pranav Garg <pranavgarg@gmail.com>

Signed-off-by: Pranav Garg <pranavgarg@gmail.com>
Co-authored-by: Pranav Garg <garprana@amazon.com>
@mch2
Segment Replication - Remove unnecessary call to markAllocationIdAsIn…
280b938 
…Sync. (opensearch-project#4224)

This PR Removes an unnecessary call to markAllocationIdAsInSync on the primary shard when replication events complete.
Recovery will manage this initial call.

Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>
@nknize
[MUTE] search with uppercase regex in bwc testing (opensearch-project…
1e47b95 
…#4226)

Prevents "search with uppercase regex" in 190_index_prefix_search.yml from
running in bwc testing until feature is backported.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
@Bukhtawar
Introduce TranslogFactory for Local/Remote Translog support (opensear…
551f7c3 
…ch-project#4172)

* Introduce TranslogFactory for Local/Remote Translog support

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
@bharath-techie
Updated changes to security
a625607
@opensearch-trigger-bot @opensearch-ci-bot
Added bwc version 2.2.1 (opensearch-project#4194)
e17af2a 
Co-authored-by: opensearch-ci-bot <opensearch-ci-bot@users.noreply.github.com>
@dreamer-89
[Segment Replication] Wait for documents to replicate to replica shar…
237f1a5 
…ds (opensearch-project#4236)

* [Segment Replication] Add thread sleep to account for replica lag in delete operations test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments, assertBusy on doc count rather than sleep

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@mch2
Segment Replication - Add additional unit tests for update & delete o…
3a97d4c 
…ps. (opensearch-project#4237)

* Segment Replication - Add additional unit tests for update & delete operations.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix spotless.

Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>
@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: a625607

@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: 78984d1

@mch2
Support shard promotion with Segment Replication. (opensearch-project…
f65e02d 
…#4135)

* Support shard promotion with Segment Replication.

This change adds basic failover support with segment replication.  Once selected, a replica will commit and reopen a writeable engine.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Add check to ensure a closed shard does not publish checkpoints.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Clean up in SegmentReplicationIT.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* PR feedback.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix merge conflict.

Signed-off-by: Marc Handalian <handalm@amazon.com>

Signed-off-by: Marc Handalian <handalm@amazon.com>
@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: f10730dbcf969ad1b7ff89285882bec7d736113c

kartg and others added 4 commits August 17, 2022 14:27
@kartg
Added timing data and more granular stages to SegmentReplicationState (…
a2ba3a8 
…opensearch-project#4222)

* Added timing data and more granular stages to SegmentReplicationState

This change introduces instrumentation logging that measures the latency of the various stages of segment replication as seen by each replica. Logs have also been added to the source node for checkpoint publishing and checkpoint metadata responses. All logging is currently at the TRACE level.

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Fixing SegmentReplicationTarget tests

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Incorporated PR feedback

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Fixing SegmentReplicationTargetService tests

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
@owaiskazi19
Handles the status code for . properties (opensearch-project#4246)
36f1d77 
* Return 400 status code for array out of bound

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Spotless apply

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* PR comments

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
@dreamer-89
[Segment Replication] Update PrimaryShardAllocator to prefer replicas…
d308a29 
… with higher replication checkpoint (opensearch-project#4041)

* [Segment Replication] Update PrimaryShardAllocator to prefer replicas having higher replication checkpoint

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Use empty replication checkpoint to avoid NPE

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Update NodeGatewayStartedShards to optionally wire in/out ReplicationCheckpoint field

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Use default replication checkpoint causing EOF errors on empty checkpoint

* Add indexSettings to GatewayAllocator to allow ReplicationCheckpoint comparator only for segrep enabled indices

* Add unit tests for primary term first replica promotion & comparator fix

* Fix NPE on empty IndexMetadata

* Remove settings from AllocationService and directly inject in GatewayAllocator

* Add more unit tests and minor code clean up

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Address review comments & integration test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix comparator on null ReplicationCheckpoint

Signed-off-by: Suraj Singh <surajrider@gmail.com>

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@bharath-techie
addressing comments
96c8b3e 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: 96c8b3e

reta and others added 5 commits August 18, 2022 15:51
@reta
Fixing NodeGatewayStartedShards bwc (de)serialization issues (opensea…
ee26e01 
…rch-project#4258)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
@dbwiddis
Fix AbstractStringFieldDataTestCase tests to account for TotalHits lo…
4643620 
…wer bound (opensearch-project#4270)

Fixes tests to account for TotalHits uncertainty as of Lucene 9.

Signed-off-by: Daniel Widdis <widdis@gmail.com>
@dependabot
Bump com.gradle.enterprise from 3.10.3 to 3.11.1 (opensearch-project#…
6629b09 
…4273)

Bumps com.gradle.enterprise from 3.10.3 to 3.11.1.

---
updated-dependencies:
- dependency-name: com.gradle.enterprise
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@navneet1v
Refactored the src and test of GeoHashGrid and GeoTileGrid Aggregatio…
5433056 
…ns on GeoPoint from server folder to geo module.(opensearch-project#4071) (opensearch-project#4072) (opensearch-project#4180)

The changes also includes:
   * Updated Search plugin to provide the interface so that plugins can also register the composite aggregations
   * Added YAML test for the geo_grid, geo_tile and composite aggregation

Signed-off-by: Navneet Verma <navneev@amazon.com>
@bharath-techie
Merge branch 'main' of https://github.com/opensearch-project/OpenSearch…
3c8116c 
… into pitsegmentsrest1
@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: 3c8116c

@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: 00daa5f66b5b9cea70f8652bdb39f88379b9884f

@bharath-techie
security changes
fa5940f 
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>
@github-actions
Copy link

Gradle Check (Jenkins) Run Completed with:

  • RESULT: null ❌
  • URL:
  • CommitID: fa5940f

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.