How to improve the rate at which the OAP Server consumes Kafka?

[linshenfanying]

[linshenfanying]

We have utilized SkyWalking for application tracing, and the corresponding architecture is shown in the diagram. A total of 220 services and approximately 1000 instances have been integrated. We have noticed that the consumption rate of Kafka by SkyWalking is very slow, with a persistent backlog, and the consumption capacity often drops suddenly. We have also implemented cluster separation.

1. The configuration of our OAP Server is 2 CPU cores and 4 GB of memory. We would like to ask for optimization suggestions.

2. How can we improve the rate at which SkyWalking OAP Server consumes Kafka?
   What are the recommended resource configurations for the OAP Server? Is the OAP Server’s performance more related to CPU or memory?

[wu-sheng]

OAP is very CPU and memory required applications. You should grant all resources.

[wu-sheng]

We exposed Kafka client configurations, if you prefer Kafka, you could setup accordingly.

skywalking/oap-server/server-starter/src/main/resources/application.yml

Lines 385 to 396 in 5c5274b

	kafka-fetcher:
	selector: ${SW_KAFKA_FETCHER:-}
	default:
	bootstrapServers: ${SW_KAFKA_FETCHER_SERVERS:localhost:9092}
	namespace: ${SW_NAMESPACE:""}
	partitions: ${SW_KAFKA_FETCHER_PARTITIONS:3}
	replicationFactor: ${SW_KAFKA_FETCHER_PARTITIONS_FACTOR:2}
	enableNativeProtoLog: ${SW_KAFKA_FETCHER_ENABLE_NATIVE_PROTO_LOG:true}
	enableNativeJsonLog: ${SW_KAFKA_FETCHER_ENABLE_NATIVE_JSON_LOG:true}
	consumers: ${SW_KAFKA_FETCHER_CONSUMERS:1}
	kafkaHandlerThreadPoolSize: ${SW_KAFKA_HANDLER_THREAD_POOL_SIZE:-1}
	kafkaHandlerThreadPoolQueueSize: ${SW_KAFKA_HANDLER_THREAD_POOL_QUEUE_SIZE:-1}