Skip to content

Commit

Permalink
Support Scratch The OAP Config Dump. (#11579)
Browse files Browse the repository at this point in the history
  • Loading branch information
@wu-sheng
wu-sheng authored Nov 22, 2023
1 parent 0fdac95 commit dd09d08
Show file tree
Hide file tree
Showing 20 changed files with 462 additions and 19 deletions.
29 changes: 20 additions & 9 deletions dist-material/release-docs/LICENSE
View file
Original file line number Diff line number Diff line change
Expand Up @@ -461,9 +461,11 @@ The following components are provided under the MIT License. See project link fo
The text of each license is also included in licenses/LICENSE-[project].txt.

https://npmjs.com/package/@babel/parser/v/7.23.0 7.23.0 MIT
https://npmjs.com/package/@ctrl/tinycolor/v/3.5.0 3.5.0 MIT
https://npmjs.com/package/@ctrl/tinycolor/v/3.6.1 3.6.1 MIT
https://npmjs.com/package/@egjs/hammerjs/v/2.0.17 2.0.17 MIT
https://npmjs.com/package/@element-plus/icons-vue/v/1.1.4 1.1.4 MIT
https://npmjs.com/package/@element-plus/icons-vue/v/2.1.0 2.1.0 MIT
https://npmjs.com/package/@floating-ui/core/v/0.7.3 0.7.3 MIT
https://npmjs.com/package/@floating-ui/dom/v/0.5.4 0.5.4 MIT
https://npmjs.com/package/@interactjs/actions/v/1.10.17 1.10.17 MIT
https://npmjs.com/package/@interactjs/auto-scroll/v/1.10.17 1.10.17 MIT
https://npmjs.com/package/@interactjs/auto-start/v/1.10.17 1.10.17 MIT
Expand All @@ -484,10 +486,10 @@ The text of each license is also included in licenses/LICENSE-[project].txt.
https://npmjs.com/package/@intlify/message-compiler/v/9.2.2 9.2.2 MIT
https://npmjs.com/package/@intlify/shared/v/9.2.2 9.2.2 MIT
https://npmjs.com/package/@intlify/vue-devtools/v/9.2.2 9.2.2 MIT
https://npmjs.com/package/@popperjs/core/v/2.11.6 2.11.6 MIT
https://npmjs.com/package/@popperjs/core/v/2.11.7 2.11.7 MIT
https://npmjs.com/package/@types/hammerjs/v/2.0.41 2.0.41 MIT
https://npmjs.com/package/@types/lodash/v/4.14.191 4.14.191 MIT
https://npmjs.com/package/@types/lodash-es/v/4.17.6 4.17.6 MIT
https://npmjs.com/package/@types/lodash-es/v/4.17.11 4.17.11 MIT
https://npmjs.com/package/@vue/compiler-core/v/3.2.45 3.2.45 MIT
https://npmjs.com/package/@vue/compiler-dom/v/3.2.45 3.2.45 MIT
https://npmjs.com/package/@vue/compiler-sfc/v/3.2.45 3.2.45 MIT
Expand All @@ -502,21 +504,28 @@ The text of each license is also included in licenses/LICENSE-[project].txt.
https://npmjs.com/package/@vue/server-renderer/v/3.2.45 3.2.45 MIT
https://npmjs.com/package/@vue/shared/v/3.2.45 3.2.45 MIT
https://npmjs.com/package/async-validator/v/4.2.5 4.2.5 MIT
https://npmjs.com/package/axios/v/0.24.0 0.24.0 MIT
https://npmjs.com/package/asynckit/v/0.4.0 0.4.0 MIT
https://npmjs.com/package/axios/v/1.6.0 1.6.0 MIT
https://npmjs.com/package/axios/node_modules/form-data/v/4.0.0 4.0.0 MIT
https://npmjs.com/package/axios/node_modules/proxy-from-env/v/1.1.0 1.1.0 MIT
https://npmjs.com/package/batch-processor/v/1.0.0 1.0.0 MIT
https://mvnrepository.com/artifact/com.graphql-java-kickstart/graphql-java-tools/13.0.1 MIT
https://mvnrepository.com/artifact/com.graphql-java/graphql-java/21.0 MIT
https://mvnrepository.com/artifact/com.graphql-java/graphql-java-extended-scalars/18.1 MIT
https://npmjs.com/package/combined-stream/v/1.0.8 1.0.8 MIT
https://npmjs.com/package/component-emitter/v/1.3.0 1.3.0 MIT
https://npmjs.com/package/cssfilter/v/0.0.10 0.0.10 MIT
https://npmjs.com/package/csstype/v/2.6.21 2.6.21 MIT
https://npmjs.com/package/d3-dsv/node_modules/commander/v/7.2.0 7.2.0 MIT
https://npmjs.com/package/d3-tip/v/0.9.1 0.9.1 MIT
https://npmjs.com/package/dayjs/v/1.11.7 1.11.7 MIT
https://npmjs.com/package/element-plus/v/2.1.0 2.1.0 MIT
https://npmjs.com/package/element-plus/node_modules/@vueuse/core/v/7.7.1 7.7.1 MIT
https://npmjs.com/package/element-plus/node_modules/@vueuse/core/node_modules/@vueuse/shared/v/7.7.1 7.7.1 MIT
https://npmjs.com/package/element-plus/node_modules/@vueuse/core/node_modules/vue-demi/v/0.13.11 0.13.11 MIT
https://npmjs.com/package/delayed-stream/v/1.0.0 1.0.0 MIT
https://npmjs.com/package/element-plus/v/2.2.5 2.2.5 MIT
https://npmjs.com/package/element-plus/node_modules/@types/web-bluetooth/v/0.0.14 0.0.14 MIT
https://npmjs.com/package/element-plus/node_modules/@vueuse/core/v/8.9.4 8.9.4 MIT
https://npmjs.com/package/element-plus/node_modules/@vueuse/core/node_modules/@vueuse/shared/v/8.9.4 8.9.4 MIT
https://npmjs.com/package/element-plus/node_modules/@vueuse/core/node_modules/vue-demi/v/0.14.6 0.14.6 MIT
https://npmjs.com/package/element-plus/node_modules/@vueuse/metadata/v/8.9.4 8.9.4 MIT
https://npmjs.com/package/element-resize-detector/v/1.2.4 1.2.4 MIT
https://npmjs.com/package/escape-html/v/1.0.3 1.0.3 MIT
https://npmjs.com/package/estree-walker/v/2.0.2 2.0.2 MIT
Expand All @@ -527,6 +536,8 @@ The text of each license is also included in licenses/LICENSE-[project].txt.
https://npmjs.com/package/lodash-es/v/4.17.21 4.17.21 MIT
https://npmjs.com/package/lodash-unified/v/1.0.3 1.0.3 MIT
https://npmjs.com/package/memoize-one/v/6.0.0 6.0.0 MIT
https://npmjs.com/package/mime-db/v/1.52.0 1.52.0 MIT
https://npmjs.com/package/mime-types/v/2.1.35 2.1.35 MIT
https://npmjs.com/package/mitt/v/2.1.0 2.1.0 MIT
https://npmjs.com/package/moment/v/2.29.4 2.29.4 MIT
https://npmjs.com/package/monaco-editor/v/0.34.1 0.34.1 MIT
Expand Down
1 change: 1 addition & 0 deletions docs/en/changes/changes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
* Repleace go2sky E2E to GO agent.
* Replace Metrics v2 protocol with MQE in UI templates and E2E Test.
* Fix incorrect apisix metrics otel rules.
* Support `Scratch The OAP Config Dump`.

#### UI

Expand Down
74 changes: 74 additions & 0 deletions docs/en/debugging/config_dump.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Scratch The OAP Config Dump

SkyWalking OAP behaviors could be controlled through hundreds of configurations. It is hard to know what is the final
configuration as all the configurations could be overrided by system environments.

The core config file [application.yml](../../../oap-server/server-starter/src/main/resources/application.yml) lists all
the configurations
and their default values. However, it is still hard to know the runtime value.

Scratch is a tool to dump the final configuration. It is provided within OAP rest server, which could be accessed
through HTTP GET `http://{core restHost}:{core restPort}/debugging/config/dump`.

```shell
> curl http://127.0.0.1:12800/debugging/config/dump
cluster.provider=standalone
core.provider=default
core.default.prepareThreads=2
core.default.restHost=0.0.0.0
core.default.searchableLogsTags=level,http.status_code
core.default.role=Mixed
core.default.persistentPeriod=25
core.default.syncPeriodHttpUriRecognitionPattern=10
core.default.restIdleTimeOut=30000
core.default.dataKeeperExecutePeriod=5
core.default.topNReportPeriod=10
core.default.gRPCSslTrustedCAPath=
core.default.downsampling=[Hour, Day]
core.default.serviceNameMaxLength=70
core.default.gRPCSslEnabled=false
core.default.restPort=12800
core.default.serviceCacheRefreshInterval=10
...
```

All booting configurations with their runtime values are listed, including the selected provider for each module.

## Protect The Secrets

Some of the configurations contain sensitive values, such as username, password, token, etc. These values would be
masked
in the dump result. For example, the `storage.elasticsearch.password` in the following configurations,

```yaml
storage:
selector: ${SW_STORAGE:h2}
elasticsearch:
password: ${SW_ES_PASSWORD:""}
```
It would be masked and shown as `********` in the dump result.

```shell
> curl http://127.0.0.1:12800/debugging/config/dump
...
storage.elasticsearch.password=********
...
```

By default, we mask the config keys through the following configurations.

```yaml
# Include the list of keywords to filter configurations including secrets. Separate keywords by a comma.
keywords4MaskingSecretsOfConfig: ${SW_CORE_KEYWORDS_FOR_MASKING_SECRETS:user,password,token,accessKey,secretKey}
```

## Disable The Config Dump Service

By default, this service is open for helping users to debug and diagnose. If you want to disable it, you need to diable the whole
`debugging-query` module through setting `selector=-`.

```yaml
debugging-query:
selector: ${SW_DEBUGGING_QUERY:-}
```
2 changes: 2 additions & 0 deletions docs/en/setup/backend/configuration-vocabulary.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,8 @@ The Configuration Vocabulary lists all available configurations provided by `app
| - | - | kafkaTopicLog | Kafka topic name for log. | SW_EXPORTER_KAFKA_TOPIC_LOG | skywalking-export-log |
| - | - | exportErrorStatusTraceOnly | Export error status trace segments through the Kafka channel. | SW_EXPORTER_KAFKA_TRACE_FILTER_ERROR | false |
| health-checker | default | checkIntervalSeconds | The period of checking OAP internal health status (in seconds). | SW_HEALTH_CHECKER_INTERVAL_SECONDS | 5 |
| debugging-query | default | | | | |
| - | - | keywords4MaskingSecretsOfConfig | Include the list of keywords to filter configurations including secrets. Separate keywords by a comma. | SW_DEBUGGING_QUERY_KEYWORDS_FOR_MASKING_SECRETS | user,password,token,accessKey,secretKey,authentication |
| configuration-discovery | default | disableMessageDigest | If true, agent receives the latest configuration every time, even without making any changes. By default, OAP uses the SHA512 message digest mechanism to detect changes in configuration. | SW_DISABLE_MESSAGE_DIGEST | false |
| receiver-event | default | gRPC services that handle events data. | - | - | |
| aws-firehose-receiver | default | host | Binding IP of HTTP server | SW_RECEIVER_AWS_FIREHOSE_HTTP_HOST | 0.0.0.0 |
Expand Down
4 changes: 4 additions & 0 deletions docs/menu.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,10 @@ catalog:
path: "/en/api/logql-service"
- name: "Security Notice"
path: "/en/security/readme"
- name: "Debugging"
catalog:
- name: "Scratch The OAP Config Dump"
path: "en/debugging/config_dump"
- name: "Academy"
catalog:
- name: "Scaling SkyWalking server automatically in kubernetes"
Expand Down
2 changes: 1 addition & 1 deletion ...r/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ public void prepare() throws ServiceNotProvidedException, ModuleStartException {
httpServer.initialize();

this.registerServiceImplementation(ConfigService.class, new ConfigService(moduleConfig, this));
this.registerServiceImplementation(ServerStatusService.class, new ServerStatusService(getManager()));
this.registerServiceImplementation(ServerStatusService.class, new ServerStatusService(getManager(), moduleConfig));
this.registerServiceImplementation(
DownSamplingConfigService.class, new DownSamplingConfigService(moduleConfig.getDownsampling()));

Expand Down
56 changes: 54 additions & 2 deletions ...-core/src/main/java/org/apache/skywalking/oap/server/core/status/ServerStatusService.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
import java.util.List;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import org.apache.skywalking.oap.server.core.CoreModuleConfig;
import org.apache.skywalking.oap.server.library.module.ApplicationConfiguration;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.apache.skywalking.oap.server.library.module.Service;
import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
Expand All @@ -38,14 +40,17 @@
@RequiredArgsConstructor
public class ServerStatusService implements Service {
private final ModuleManager manager;
private final CoreModuleConfig moduleConfig;
@Getter
private BootingStatus bootingStatus = new BootingStatus();
@Getter
private ClusterStatus clusterStatus = new ClusterStatus();

private List<ServerStatusWatcher> statusWatchers = new ArrayList<>();

public void bootedNow(long uptime) {
private List<ApplicationConfiguration.ModuleConfiguration> configurations;

public void bootedNow(List<ApplicationConfiguration.ModuleConfiguration> configurations, long uptime) {
bootingStatus.setBooted(true);
bootingStatus.setUptime(uptime);
manager.find(TelemetryModule.NAME)
Expand All @@ -55,14 +60,18 @@ public void bootedNow(long uptime) {
// Set uptime to second
.setValue(uptime / 1000d);
this.statusWatchers.forEach(watcher -> watcher.onServerBooted(bootingStatus));
this.configurations = configurations;
}

public void rebalancedCluster(long rebalancedTime) {
clusterStatus.setRebalancedTime(rebalancedTime);
manager.find(TelemetryModule.NAME)
.provider()
.getService(MetricsCreator.class)
.createGauge("cluster_rebalanced_time", "oap cluster rebalanced time after scale", MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE)
.createGauge(
"cluster_rebalanced_time", "oap cluster rebalanced time after scale", MetricsTag.EMPTY_KEY,
MetricsTag.EMPTY_VALUE
)
.setValue(rebalancedTime / 1000d);

this.statusWatchers.forEach(watcher -> watcher.onClusterRebalanced(clusterStatus));
Expand All @@ -71,4 +80,47 @@ public void rebalancedCluster(long rebalancedTime) {
public void registerWatcher(ServerStatusWatcher watcher) {
this.statusWatchers.add(watcher);
}

/**
* @return a complete list of booting configurations with effected values.
* @since 9.7.0
*/
public String dumpBootingConfigurations(String keywords4MaskingSecretsOfConfig) {
if (configurations == null || configurations.isEmpty()) {
return "No available booting configurations.";
}
final String[] keywords = keywords4MaskingSecretsOfConfig.split(",");
StringBuilder configList = new StringBuilder();
for (ApplicationConfiguration.ModuleConfiguration configuration : configurations) {
final String moduleName = configuration.getModuleName();
if (configuration.getProviders().size() == 1) {
configList.append(moduleName)
.append(".provider=")
.append(configuration.getProviders().keySet().iterator().next())
.append("\n");
}
configuration.getProviders().forEach(
(providerName, providerConfiguration) ->
providerConfiguration.getProperties().forEach(
(key, value) -> {
for (final String keyword : keywords) {
if (key.toString().toLowerCase().contains(keyword.toLowerCase())) {
value = "******";
}
}

configList.append(moduleName)
.append(".")
.append(providerName)
.append(".")
.append(key)
.append("=")
.append(value)
.append("\n");
}
)
);
}
return configList.toString();
}
}
Loading

0 comments on commit dd09d08

Please sign in to comment.