Skip to content

Commit

Permalink
Update BanyanDB Java Client to 0.5.0 (#11370)
Browse files Browse the repository at this point in the history
  • Loading branch information
@lujiajing1126
lujiajing1126 authored Oct 4, 2023
1 parent d6187a1 commit 338ebca
Show file tree
Hide file tree
Showing 11 changed files with 49 additions and 31 deletions.
2 changes: 2 additions & 0 deletions docs/en/changes/changes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
including `ui_template，ui_menu，continuous_profiling_policy`.
* Add a release mechanism for alarm windows when it is expired in case of OOM.
* Fix Zipkin trace receiver response: make the HTTP status code from `200` to `202`.
* Update BanyanDB Java Client to 0.5.0.
* Fix getInstances query in the BanyanDB Metadata DAO.

#### UI

Expand Down
3 changes: 1 addition & 2 deletions docs/en/setup/backend/configuration-vocabulary.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,7 @@ The Configuration Vocabulary lists all available configurations provided by `app
| - | - | maxSizeOfBatchSql | The maximum size of batch size of SQL execution | SW_STORAGE_MAX_SIZE_OF_BATCH_SQL | 2000 |
| - | - | asyncBatchPersistentPoolSize | async flush data into database thread size | SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE | 4 |
| - | banyandb | - | BanyanDB storage. | - | - |
| - | - | host | Host of the BanyanDB. | SW_STORAGE_BANYANDB_HOST | 127.0.0.1 |
| - | - | port | Port of the BanyanDB. | SW_STORAGE_BANYANDB_PORT | 17912 |
| - | - | targets | Hosts with ports of the BanyanDB. | SW_STORAGE_BANYANDB_TARGETS | 127.0.0.1:17912 |
| - | - | maxBulkSize | The maximum size of write entities in a single batch write call. | SW_STORAGE_BANYANDB_MAX_BULK_SIZE | 5000 |
| - | - | flushInterval | Period of flush interval. In the timeunit of seconds. | SW_STORAGE_BANYANDB_FLUSH_INTERVAL | 15 |
| - | - | metricsShardsNumber | Shards Number for measure/metrics. | SW_STORAGE_BANYANDB_METRICS_SHARDS_NUMBER | 1 |
Expand Down
7 changes: 4 additions & 3 deletions docs/en/setup/backend/storages/banyandb.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,12 @@
[BanyanDB](https://github.com/apache/skywalking-banyandb) is a dedicated storage implementation developed by the SkyWalking Team and the community.
Activate BanyanDB as the storage, and set storage provider to **banyandb**.

The OAP requires BanyanDB 0.5 server. As BanyanDB is still in the beta phase, we don't provide any compatibility besides the required version.

```yaml
storage:
banyandb:
host: ${SW_STORAGE_BANYANDB_HOST:127.0.0.1}
port: ${SW_STORAGE_BANYANDB_PORT:17912}
targets: ${SW_STORAGE_BANYANDB_TARGETS:127.0.0.1:17912}
maxBulkSize: ${SW_STORAGE_BANYANDB_MAX_BULK_SIZE:5000}
flushInterval: ${SW_STORAGE_BANYANDB_FLUSH_INTERVAL:15}
metricsShardsNumber: ${SW_STORAGE_BANYANDB_METRICS_SHARDS_NUMBER:1}
Expand All @@ -22,4 +23,4 @@ storage:
```
For more details, please refer to the documents of [BanyanDB](https://skywalking.apache.org/docs/skywalking-banyandb/next/readme/)
and [BanyanDB Java Client](https://github.com/apache/skywalking-banyandb-java-client) subprojects.
and [BanyanDB Java Client](https://github.com/apache/skywalking-banyandb-java-client) subprojects.
2 changes: 1 addition & 1 deletion oap-server-bom/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@
<awaitility.version>3.0.0</awaitility.version>
<httpcore.version>4.4.13</httpcore.version>
<commons-compress.version>1.21</commons-compress.version>
<banyandb-java-client.version>0.4.0</banyandb-java-client.version>
<banyandb-java-client.version>0.5.0</banyandb-java-client.version>
<kafka-clients.version>2.8.1</kafka-clients.version>
<spring-kafka-test.version>2.4.6.RELEASE</spring-kafka-test.version>
<consul.client.version>1.5.3</consul.client.version>
Expand Down
3 changes: 1 addition & 2 deletions oap-server/server-starter/src/main/resources/application.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -213,8 +213,7 @@ storage:
maxSizeOfBatchSql: ${SW_STORAGE_MAX_SIZE_OF_BATCH_SQL:2000}
asyncBatchPersistentPoolSize: ${SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE:4}
banyandb:
host: ${SW_STORAGE_BANYANDB_HOST:127.0.0.1}
port: ${SW_STORAGE_BANYANDB_PORT:17912}
targets: ${SW_STORAGE_BANYANDB_TARGETS:127.0.0.1:17912}
maxBulkSize: ${SW_STORAGE_BANYANDB_MAX_BULK_SIZE:5000}
flushInterval: ${SW_STORAGE_BANYANDB_FLUSH_INTERVAL:15}
metricsShardsNumber: ${SW_STORAGE_BANYANDB_METRICS_SHARDS_NUMBER:1}
Expand Down
4 changes: 2 additions & 2 deletions .../java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageClient.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@ public class BanyanDBStorageClient implements Client, HealthCheckable {
final BanyanDBClient client;
private final DelegatedHealthChecker healthChecker = new DelegatedHealthChecker();

public BanyanDBStorageClient(String host, int port) {
this.client = new BanyanDBClient(host, port);
public BanyanDBStorageClient(String... targets) {
this.client = new BanyanDBClient(targets);
}

@Override
Expand Down
14 changes: 12 additions & 2 deletions .../java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageConfig.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,21 @@

package org.apache.skywalking.oap.server.storage.plugin.banyandb;

import com.google.common.base.Splitter;
import com.google.common.collect.Iterables;
import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oap.server.library.module.ModuleConfig;

@Getter
@Setter
public class BanyanDBStorageConfig extends ModuleConfig {
private String host = "127.0.0.1";
private int port = 17912;
/**
* A comma-separated list of BanyanDB targets.
*
* @since 9.7.0
*/
private String targets = "127.0.0.1:17912";
/**
* The maximum size of write entities in a single batch write call.
*/
Expand Down Expand Up @@ -96,4 +102,8 @@ public class BanyanDBStorageConfig extends ModuleConfig {
* @since 9.4.0
*/
private String specificGroupSettings;

public String[] getTargetArray() {
return Iterables.toArray(Splitter.on(",").omitEmptyStrings().trimResults().split(this.targets), String.class);
}
}
2 changes: 1 addition & 1 deletion ...ava/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageProvider.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ public void onInitialized(final BanyanDBStorageConfig initialized) {
public void prepare() throws ServiceNotProvidedException, ModuleStartException {
this.registerServiceImplementation(StorageBuilderFactory.class, new StorageBuilderFactory.Default());

this.client = new BanyanDBStorageClient(config.getHost(), config.getPort());
this.client = new BanyanDBStorageClient(config.getTargetArray());
this.modelInstaller = new BanyanDBIndexInstaller(client, getManager(), this.config);

// Stream
Expand Down
39 changes: 23 additions & 16 deletions ...pache/skywalking/oap/server/storage/plugin/banyandb/measure/BanyanDBMetadataQueryDAO.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,12 @@
package org.apache.skywalking.oap.server.storage.plugin.banyandb.measure;

import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
import com.google.gson.Gson;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import org.apache.commons.lang3.StringUtils;
import org.apache.skywalking.banyandb.v1.client.AbstractCriteria;
import org.apache.skywalking.banyandb.v1.client.DataPoint;
import org.apache.skywalking.banyandb.v1.client.MeasureQuery;
import org.apache.skywalking.banyandb.v1.client.MeasureQueryResponse;
Expand Down Expand Up @@ -92,12 +94,12 @@ public BanyanDBMetadataQueryDAO(BanyanDBStorageClient client) {
@Override
public List<Service> listServices() throws IOException {
MeasureQueryResponse resp = query(ServiceTraffic.INDEX_NAME,
SERVICE_TRAFFIC_TAGS,
Collections.emptySet(), new QueryBuilder<MeasureQuery>() {
@Override
protected void apply(MeasureQuery query) {
}
});
SERVICE_TRAFFIC_TAGS,
Collections.emptySet(), new QueryBuilder<MeasureQuery>() {
@Override
protected void apply(MeasureQuery query) {
}
});

final List<Service> services = new ArrayList<>();
MetadataRegistry.Schema schema = MetadataRegistry.INSTANCE.findMetadata(ServiceTraffic.INDEX_NAME, DownSampling.Minute);
Expand Down Expand Up @@ -144,8 +146,8 @@ public ServiceInstance getInstance(String instanceId) throws IOException {
new QueryBuilder<MeasureQuery>() {
@Override
protected void apply(MeasureQuery query) {
query.and(eq(InstanceTraffic.SERVICE_ID, id.getServiceId()))
.and(eq(InstanceTraffic.NAME, id.getName()));
query.and(eq(InstanceTraffic.SERVICE_ID, id.getServiceId()))
.and(eq(InstanceTraffic.NAME, id.getName()));
}
});
MetadataRegistry.Schema schema = MetadataRegistry.INSTANCE.findMetadata(InstanceTraffic.INDEX_NAME, DownSampling.Minute);
Expand All @@ -155,14 +157,19 @@ protected void apply(MeasureQuery query) {
@Override
public List<ServiceInstance> getInstances(List<String> instanceIds) throws IOException {
MeasureQueryResponse resp = query(InstanceTraffic.INDEX_NAME,
INSTANCE_TRAFFIC_TAGS,
Collections.emptySet(),
new QueryBuilder<MeasureQuery>() {
@Override
protected void apply(MeasureQuery query) {
query.and(in(InstanceTraffic.ID, instanceIds));
}
});
INSTANCE_TRAFFIC_TAGS,
Collections.emptySet(),
new QueryBuilder<MeasureQuery>() {
@Override
protected void apply(MeasureQuery query) {
List<AbstractCriteria> instanceRelationsQueryConditions = new ArrayList<>(instanceIds.size());
for (final String instanceId : instanceIds) {
final IDManager.ServiceInstanceID.InstanceIDDefinition def = IDManager.ServiceInstanceID.analysisId(instanceId);
and(Lists.newArrayList(eq(InstanceTraffic.SERVICE_ID, def.getServiceId()), eq(InstanceTraffic.NAME, def.getName())));
}
query.criteria(or(instanceRelationsQueryConditions));
}
});
MetadataRegistry.Schema schema = MetadataRegistry.INSTANCE.findMetadata(InstanceTraffic.INDEX_NAME, DownSampling.Minute);
return resp.getDataPoints().stream().map(e -> buildInstance(e, schema)).collect(Collectors.toList());
}
Expand Down
2 changes: 1 addition & 1 deletion test/e2e-v2/script/docker-compose/base-compose.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ services:
SW_STORAGE_ES_CLUSTER_NODES: es:9200
SW_JDBC_URL: jdbc:mysql://mysql:3306/swtest?allowMultiQueries=true
SW_STORAGE_INFLUXDB_URL: http://influxdb:8086
SW_STORAGE_BANYANDB_HOST: "banyandb"
SW_STORAGE_BANYANDB_TARGETS: banyandb:17912
SW_CONFIG_ETCD_PERIOD: 1
SW_CONFIG_ETCD_ENDPOINTS: http://etcd:2379
SW_CLUSTER_ETCD_ENDPOINTS: http://etcd:2379
Expand Down
2 changes: 1 addition & 1 deletion test/e2e-v2/script/env
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,6 @@ SW_AGENT_CLIENT_JS_COMMIT=af0565a67d382b683c1dbd94c379b7080db61449
SW_AGENT_CLIENT_JS_TEST_COMMIT=4f1eb1dcdbde3ec4a38534bf01dded4ab5d2f016
SW_KUBERNETES_COMMIT_SHA=e2c61c6774cf377b23516fca6f8a1e119d3191c5
SW_ROVER_COMMIT=fc8d074c6d34ecfee585a7097cbd5aef1ca680a5
SW_BANYANDB_COMMIT=df34a83663d66af9687ff484884ceae669c83980
SW_BANYANDB_COMMIT=63be6e80bef82ea274d7352cc7a461b757d28355

SW_CTL_COMMIT=9d2d0edccda2afe5cf24f1e632142f40e80e8fa4

0 comments on commit 338ebca

Please sign in to comment.