Skip to content

Commit e564a38

Browse files
vibrantvarunvibrantvarun
authored
Merge branch 'main' into POC_on_collector_context
Signed-off-by: Varun Jain <varunudr@amazon.com>
2 parents 8499eb6 + 8fd9aeb commit e564a38

File tree

43 files changed

+924
-230
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+924
-230
lines changed

‎.github/workflows/trigger-manifest-generation.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
name: Trigger manifest generation workflow
2+
3+
on:
4+
workflow_dispatch:
5+
push:
6+
paths:
7+
- buildSrc/version.properties
8+
9+
jobs:
10+
trigger-manifest-workflow:
11+
if: github.repository == 'opensearch-project/OpenSearch'
12+
runs-on: ubuntu-latest
13+
steps:
14+
- name: Trigger manifest-update workflow
15+
run: |
16+
echo "Triggering manifest-update workflow at https://build.ci.opensearch.org/job/manifest-update/"
17+
curl -f -X POST https://build.ci.opensearch.org/job/manifest-update/build --user ${{ secrets.JENKINS_GITHUB_USER}}:${{ secrets.JENKINS_GITHUB_USER_TOKEN}}

‎CHANGELOG.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,17 +15,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
1515
- Introduce a new pull-based ingestion plugin for file-based indexing (for local testing) ([#18591](https://github.com/opensearch-project/OpenSearch/pull/18591))
1616
- Add support for search pipeline in search and msearch template ([#18564](https://github.com/opensearch-project/OpenSearch/pull/18564))
1717
- Add BooleanQuery rewrite moving constant-scoring must clauses to filter clauses ([#18510](https://github.com/opensearch-project/OpenSearch/issues/18510))
18+
- Add support for non-timing info in profiler ([#18460](https://github.com/opensearch-project/OpenSearch/issues/18460))
1819
- Add functionality for plugins to inject QueryCollectorContext during QueryPhase ([#18637](https://github.com/opensearch-project/OpenSearch/pull/18637))
1920

2021
### Changed
2122
- Update Subject interface to use CheckedRunnable ([#18570](https://github.com/opensearch-project/OpenSearch/issues/18570))
23+
- Update SecureAuxTransportSettingsProvider to distinguish between aux transport types ([#18616](https://github.com/opensearch-project/OpenSearch/pull/18616))
24+
- Making multi rate limiters in repository dynamic [#18069](https://github.com/opensearch-project/OpenSearch/pull/18069)
2225

2326
### Dependencies
2427
- Bump `stefanzweifel/git-auto-commit-action` from 5 to 6 ([#18524](https://github.com/opensearch-project/OpenSearch/pull/18524))
2528
- Bump Apache Lucene to 10.2.2 ([#18573](https://github.com/opensearch-project/OpenSearch/pull/18573))
2629
- Bump `org.apache.logging.log4j:log4j-core` from 2.24.3 to 2.25.0 ([#18589](https://github.com/opensearch-project/OpenSearch/pull/18589))
2730
- Bump `com.google.code.gson:gson` from 2.13.0 to 2.13.1 ([#18585](https://github.com/opensearch-project/OpenSearch/pull/18585))
2831
- Bump `com.azure:azure-core-http-netty` from 1.15.11 to 1.15.12 ([#18586](https://github.com/opensearch-project/OpenSearch/pull/18586))
32+
- Bump `com.squareup.okio:okio` from 3.13.0 to 3.14.0 ([#18645](https://github.com/opensearch-project/OpenSearch/pull/18645))
33+
- Bump `com.netflix.nebula.ospackage-base` from 11.11.2 to 12.0.0 ([#18646](https://github.com/opensearch-project/OpenSearch/pull/18646))
2934

3035
### Deprecated
3136

@@ -34,11 +39,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
3439
### Fixed
3540
- Add task cancellation checks in aggregators ([#18426](https://github.com/opensearch-project/OpenSearch/pull/18426))
3641
- Fix concurrent timings in profiler ([#18540](https://github.com/opensearch-project/OpenSearch/pull/18540))
42+
- Fix regex query from query string query to work with field alias ([#18215](https://github.com/opensearch-project/OpenSearch/issues/18215))
3743
- [Autotagging] Fix delete rule event consumption in InMemoryRuleProcessingService ([#18628](https://github.com/opensearch-project/OpenSearch/pull/18628))
3844
- Cannot communicate with HTTP/2 when reactor-netty is enabled ([#18599](https://github.com/opensearch-project/OpenSearch/pull/18599))
3945
- Fix the visit of sub queries for HasParentQuery and HasChildQuery ([#18621](https://github.com/opensearch-project/OpenSearch/pull/18621))
4046

41-
4247
### Security
4348

4449
[Unreleased 3.x]: https://github.com/opensearch-project/OpenSearch/compare/3.1...main

‎distribution/packages/build.gradle‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ import java.util.regex.Pattern
6363
*/
6464

6565
plugins {
66-
id "com.netflix.nebula.ospackage-base" version "11.11.2"
66+
id "com.netflix.nebula.ospackage-base" version "12.0.0"
6767
}
6868

6969
void addProcessFilesTask(String type, boolean jdk) {

‎plugins/repository-hdfs/src/test/java/org/opensearch/repositories/hdfs/HdfsTests.java‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,7 @@ public void testSimpleWorkflow() {
160160

161161
public void testMissingUri() {
162162
try {
163-
OpenSearchIntegTestCase.putRepository(client().admin().cluster(), "test-repo", "hdfs", Settings.builder());
163+
OpenSearchIntegTestCase.putRepository(client().admin().cluster(), "test-repo1", "hdfs", Settings.builder());
164164
fail();
165165
} catch (RepositoryException e) {
166166
assertTrue(e.getCause() instanceof IllegalArgumentException);
@@ -193,7 +193,7 @@ public void testNonHdfsUri() {
193193
public void testPathSpecifiedInHdfs() {
194194
try {
195195
Settings.Builder settings = Settings.builder().put("uri", "hdfs:///some/path");
196-
OpenSearchIntegTestCase.putRepository(client().admin().cluster(), "test-repo", "hdfs", settings);
196+
OpenSearchIntegTestCase.putRepository(client().admin().cluster(), "test-repo2", "hdfs", settings);
197197
fail();
198198
} catch (RepositoryException e) {
199199
assertTrue(e.getCause() instanceof IllegalArgumentException);
@@ -204,7 +204,7 @@ public void testPathSpecifiedInHdfs() {
204204
public void testMissingPath() {
205205
try {
206206
Settings.Builder settings = Settings.builder().put("uri", "hdfs:///");
207-
OpenSearchIntegTestCase.putRepository(client().admin().cluster(), "test-repo", "hdfs", settings);
207+
OpenSearchIntegTestCase.putRepository(client().admin().cluster(), "test-repo3", "hdfs", settings);
208208
fail();
209209
} catch (RepositoryException e) {
210210
assertTrue(e.getCause() instanceof IllegalArgumentException);

‎plugins/transport-grpc/src/main/java/org/opensearch/plugin/transport/grpc/ssl/SecureNetty4GrpcServerTransport.java‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -106,15 +106,16 @@ public String settingKey() {
106106
* @param provider for SSLContext and SecureAuxTransportParameters (ClientAuth and enabled ciphers).
107107
*/
108108
private JdkSslContext getSslContext(Settings settings, SecureAuxTransportSettingsProvider provider) throws SSLException {
109-
Optional<SSLContext> sslContext = provider.buildSecureAuxServerTransportContext(settings, this);
109+
Optional<SSLContext> sslContext = provider.buildSecureAuxServerTransportContext(settings, this.settingKey());
110110
if (sslContext.isEmpty()) {
111111
try {
112112
sslContext = Optional.of(SSLContext.getDefault());
113113
} catch (NoSuchAlgorithmException e) {
114114
throw new SSLException("Failed to build default SSLContext for " + SecureNetty4GrpcServerTransport.class.getName(), e);
115115
}
116116
}
117-
SecureAuxTransportSettingsProvider.SecureAuxTransportParameters params = provider.parameters().orElseGet(DefaultParameters::new);
117+
SecureAuxTransportSettingsProvider.SecureAuxTransportParameters params = provider.parameters(settings, this.settingKey())
118+
.orElseGet(DefaultParameters::new);
118119
ClientAuth clientAuth = ClientAuth.valueOf(params.clientAuth().orElseThrow().toUpperCase(Locale.ROOT));
119120
return new JdkSslContext(
120121
sslContext.get(),

‎plugins/transport-grpc/src/test/java/org/opensearch/plugin/transport/grpc/ssl/SecureSettingsHelpers.java‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,6 @@
1010

1111
import org.opensearch.common.settings.Settings;
1212
import org.opensearch.plugins.SecureAuxTransportSettingsProvider;
13-
import org.opensearch.transport.AuxTransport;
1413

1514
import javax.net.ssl.KeyManagerFactory;
1615
import javax.net.ssl.SSLContext;
@@ -110,7 +109,7 @@ static SecureAuxTransportSettingsProvider getSecureSettingsProvider(
110109
) {
111110
return new SecureAuxTransportSettingsProvider() {
112111
@Override
113-
public Optional<SSLContext> buildSecureAuxServerTransportContext(Settings settings, AuxTransport transport)
112+
public Optional<SSLContext> buildSecureAuxServerTransportContext(Settings settings, String auxTransportType)
114113
throws SSLException {
115114
// Choose a random protocol from among supported test defaults
116115
String protocol = randomFrom(DEFAULT_SSL_PROTOCOLS);
@@ -126,7 +125,7 @@ public Optional<SSLContext> buildSecureAuxServerTransportContext(Settings settin
126125
}
127126

128127
@Override
129-
public Optional<SecureAuxTransportParameters> parameters() {
128+
public Optional<SecureAuxTransportParameters> parameters(Settings settings, String auxTransportType) {
130129
return Optional.of(new SecureAuxTransportParameters() {
131130
@Override
132131
public Optional<String> clientAuth() {

‎rest-api-spec/src/main/resources/rest-api-spec/test/search/61_query_string_field_alias.yml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
setup:
2+
- skip:
3+
version: " - 3.1.99"
4+
reason: "regex query over field alias support starts 3.2"
5+
6+
- do:
7+
indices.create:
8+
index: test_index
9+
body:
10+
settings:
11+
number_of_shards: 1
12+
number_of_replicas: 0
13+
mappings:
14+
properties:
15+
test:
16+
type: text
17+
test_alias:
18+
type: alias
19+
path: test
20+
21+
- do:
22+
bulk:
23+
refresh: true
24+
body: |
25+
{"index":{"_index":"test_index","_id":"1"}}
26+
{"test":"hello"}
27+
{"index":{"_index":"test_index","_id":"2"}}
28+
{"test":"world"}
29+
30+
---
31+
"regex search on normal field":
32+
- do:
33+
search:
34+
rest_total_hits_as_int: true
35+
index: test_index
36+
body:
37+
query:
38+
query_string:
39+
query: "test: /h[a-z].*/"
40+
41+
- match: {hits.total: 1}
42+
- match: {hits.hits.0._id: "1"}
43+
44+
---
45+
"regex search on alias field":
46+
- do:
47+
search:
48+
rest_total_hits_as_int: true
49+
index: test_index
50+
body:
51+
query:
52+
query_string:
53+
query: "test_alias: /h[a-z].*/"
54+
55+
- match: {hits.total: 1}
56+
- match: {hits.hits.0._id: "1"}

‎server/src/internalClusterTest/java/org/opensearch/repositories/RepositoriesServiceIT.java‎

Lines changed: 129 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232

3333
package org.opensearch.repositories;
3434

35+
import org.apache.lucene.store.RateLimiter;
3536
import org.opensearch.action.admin.cluster.repositories.get.GetRepositoriesResponse;
3637
import org.opensearch.cluster.metadata.RepositoryMetadata;
3738
import org.opensearch.common.settings.Settings;
@@ -42,12 +43,18 @@
4243
import org.opensearch.test.OpenSearchIntegTestCase;
4344
import org.opensearch.transport.client.Client;
4445

46+
import java.nio.file.Path;
4547
import java.util.Collection;
4648
import java.util.Collections;
4749
import java.util.concurrent.atomic.AtomicInteger;
4850

4951
import static org.opensearch.cluster.metadata.IndexMetadata.SETTING_NUMBER_OF_REPLICAS;
5052
import static org.opensearch.cluster.metadata.IndexMetadata.SETTING_NUMBER_OF_SHARDS;
53+
import static org.opensearch.repositories.blobstore.BlobStoreRepository.MAX_REMOTE_DOWNLOAD_BYTES_PER_SEC;
54+
import static org.opensearch.repositories.blobstore.BlobStoreRepository.MAX_REMOTE_LOW_PRIORITY_UPLOAD_BYTES_PER_SEC;
55+
import static org.opensearch.repositories.blobstore.BlobStoreRepository.MAX_REMOTE_UPLOAD_BYTES_PER_SEC;
56+
import static org.opensearch.repositories.blobstore.BlobStoreRepository.MAX_RESTORE_BYTES_PER_SEC;
57+
import static org.opensearch.repositories.blobstore.BlobStoreRepository.MAX_SNAPSHOT_BYTES_PER_SEC;
5158
import static org.hamcrest.Matchers.containsString;
5259
import static org.hamcrest.Matchers.equalTo;
5360
import static org.hamcrest.Matchers.hasSize;
@@ -138,10 +145,11 @@ public void testCreatSnapAndUpdateReposityCauseInfiniteLoop() throws Interrupted
138145

139146
// create repository
140147
final String repositoryName = "test-repo";
148+
Path path = randomRepoPath();
141149
Settings.Builder repoSettings = Settings.builder()
142-
.put("location", randomRepoPath())
143-
.put("max_snapshot_bytes_per_sec", "10mb")
144-
.put("max_restore_bytes_per_sec", "10mb");
150+
.put("location", path)
151+
.put(MAX_SNAPSHOT_BYTES_PER_SEC, "10mb")
152+
.put(MAX_RESTORE_BYTES_PER_SEC, "10mb");
145153
OpenSearchIntegTestCase.putRepositoryWithNoSettingOverrides(
146154
client().admin().cluster(),
147155
repositoryName,
@@ -176,7 +184,7 @@ public void testCreatSnapAndUpdateReposityCauseInfiniteLoop() throws Interrupted
176184

177185
try {
178186
logger.info("--> begin to reset repository");
179-
repoSettings = Settings.builder().put("location", randomRepoPath()).put("max_snapshot_bytes_per_sec", "300mb");
187+
repoSettings = Settings.builder().put("location", randomRepoPath()).put(MAX_SNAPSHOT_BYTES_PER_SEC, "300mb");
180188
OpenSearchIntegTestCase.putRepositoryWithNoSettingOverrides(
181189
client().admin().cluster(),
182190
repositoryName,
@@ -194,4 +202,121 @@ public void testCreatSnapAndUpdateReposityCauseInfiniteLoop() throws Interrupted
194202

195203
thread.join();
196204
}
205+
206+
public void testAdjustBytesPerSecSettingForSnapAndRestore() {
207+
final InternalTestCluster cluster = internalCluster();
208+
final RepositoriesService repositoriesService = cluster.getDataOrClusterManagerNodeInstances(RepositoriesService.class)
209+
.iterator()
210+
.next();
211+
212+
// create repository
213+
final String repositoryName = "test-repo1";
214+
long rateBytes = 200000;
215+
Path path = randomRepoPath();
216+
Settings.Builder repoSettings = Settings.builder()
217+
.put("location", path)
218+
.put(MAX_SNAPSHOT_BYTES_PER_SEC, (rateBytes + "b"))
219+
.put(MAX_RESTORE_BYTES_PER_SEC, (rateBytes + "b"))
220+
.put(MAX_REMOTE_UPLOAD_BYTES_PER_SEC, (rateBytes + "b"))
221+
.put(MAX_REMOTE_LOW_PRIORITY_UPLOAD_BYTES_PER_SEC, (rateBytes + "b"))
222+
.put(MAX_REMOTE_DOWNLOAD_BYTES_PER_SEC, (rateBytes + "b"));
223+
OpenSearchIntegTestCase.putRepositoryWithNoSettingOverrides(
224+
client().admin().cluster(),
225+
repositoryName,
226+
FsRepository.TYPE,
227+
true,
228+
repoSettings
229+
);
230+
231+
FsRepository repository = (FsRepository) repositoriesService.repository(repositoryName);
232+
RateLimiter snapshotRateLimiter = repository.snapshotRateLimiter();
233+
assertThat(snapshotRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
234+
RateLimiter restoreRateLimiter = repository.restoreRateLimiter();
235+
assertThat(restoreRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
236+
RateLimiter remoteUploadRateLimiter = repository.remoteUploadRateLimiter();
237+
assertThat(remoteUploadRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
238+
RateLimiter remoteUploadLowPriorityRateLimiter = repository.remoteUploadLowPriorityRateLimiter();
239+
assertThat(remoteUploadLowPriorityRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
240+
RateLimiter remoteDownloadRateLimiter = repository.remoteDownloadRateLimiter();
241+
assertThat(remoteDownloadRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
242+
243+
// adjust all the reloadable settings
244+
{
245+
rateBytes = rateBytes / 2;
246+
repoSettings = Settings.builder()
247+
.put(MAX_SNAPSHOT_BYTES_PER_SEC, (rateBytes + "b"))
248+
.put(MAX_RESTORE_BYTES_PER_SEC, (rateBytes + "b"))
249+
.put(MAX_REMOTE_UPLOAD_BYTES_PER_SEC, (rateBytes + "b"))
250+
.put(MAX_REMOTE_LOW_PRIORITY_UPLOAD_BYTES_PER_SEC, (rateBytes + "b"))
251+
.put(MAX_REMOTE_DOWNLOAD_BYTES_PER_SEC, (rateBytes + "b"));
252+
OpenSearchIntegTestCase.putRepositoryWithNoSettingOverrides(
253+
client().admin().cluster(),
254+
repositoryName,
255+
FsRepository.TYPE,
256+
true,
257+
repoSettings
258+
);
259+
FsRepository newRepository = (FsRepository) repositoriesService.repository(repositoryName);
260+
assertThat(newRepository, sameInstance(repository));
261+
snapshotRateLimiter = newRepository.snapshotRateLimiter();
262+
assertThat(snapshotRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
263+
restoreRateLimiter = newRepository.restoreRateLimiter();
264+
assertThat(restoreRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
265+
remoteUploadRateLimiter = newRepository.remoteUploadRateLimiter();
266+
assertThat(remoteUploadRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
267+
remoteUploadLowPriorityRateLimiter = newRepository.remoteUploadLowPriorityRateLimiter();
268+
assertThat(remoteUploadLowPriorityRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
269+
remoteDownloadRateLimiter = newRepository.remoteDownloadRateLimiter();
270+
assertThat(remoteDownloadRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
271+
}
272+
273+
// In addition to the settings in RELOADABLE_SETTINGS, all the new settings should be equal to current settings
274+
{
275+
long newRateBytes = rateBytes / 2;
276+
repoSettings = Settings.builder()
277+
.put("location", path)
278+
.put(MAX_SNAPSHOT_BYTES_PER_SEC, (newRateBytes + "b"))
279+
.put(MAX_RESTORE_BYTES_PER_SEC, (newRateBytes + "b"));
280+
OpenSearchIntegTestCase.putRepositoryWithNoSettingOverrides(
281+
client().admin().cluster(),
282+
repositoryName,
283+
FsRepository.TYPE,
284+
true,
285+
repoSettings
286+
);
287+
FsRepository newRepository = (FsRepository) repositoriesService.repository(repositoryName);
288+
assertThat(newRepository, sameInstance(repository));
289+
snapshotRateLimiter = newRepository.snapshotRateLimiter();
290+
assertThat(snapshotRateLimiter.getMBPerSec(), equalTo((double) newRateBytes / (1024 * 1024)));
291+
restoreRateLimiter = newRepository.restoreRateLimiter();
292+
assertThat(restoreRateLimiter.getMBPerSec(), equalTo((double) newRateBytes / (1024 * 1024)));
293+
remoteUploadRateLimiter = newRepository.remoteUploadRateLimiter();
294+
assertThat(remoteUploadRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
295+
remoteUploadLowPriorityRateLimiter = newRepository.remoteUploadLowPriorityRateLimiter();
296+
assertThat(remoteUploadLowPriorityRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
297+
remoteDownloadRateLimiter = newRepository.remoteDownloadRateLimiter();
298+
assertThat(remoteDownloadRateLimiter.getMBPerSec(), equalTo((double) rateBytes / (1024 * 1024)));
299+
}
300+
301+
// the new settings are not all equal to the old settings, so the repository will be not reloaded
302+
{
303+
rateBytes = rateBytes / 2;
304+
repoSettings = Settings.builder()
305+
.put("location", path)
306+
.put("io_buffer_size", "8mb")
307+
.put(MAX_RESTORE_BYTES_PER_SEC, (rateBytes + "b"))
308+
.put(MAX_REMOTE_UPLOAD_BYTES_PER_SEC, (rateBytes + "b"))
309+
.put(MAX_REMOTE_LOW_PRIORITY_UPLOAD_BYTES_PER_SEC, (rateBytes + "b"))
310+
.put(MAX_REMOTE_DOWNLOAD_BYTES_PER_SEC, (rateBytes + "b"));
311+
OpenSearchIntegTestCase.putRepositoryWithNoSettingOverrides(
312+
client().admin().cluster(),
313+
repositoryName,
314+
FsRepository.TYPE,
315+
true,
316+
repoSettings
317+
);
318+
FsRepository newRepository = (FsRepository) repositoriesService.repository(repositoryName);
319+
assertNotEquals(newRepository, repository);
320+
}
321+
}
197322
}

0 commit comments

Comments
 (0)