Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: use new protocol for remote engine service write #1146

Merged
merged 3 commits into from
Aug 14, 2023

Conversation

ShiKaiWi
Copy link
Member

@ShiKaiWi ShiKaiWi commented Aug 10, 2023

Rationale

According to the cpu profile results, it costs too much cpu resource when encode/decode the payloads for the remote engine write service.

Detailed Changes

  • Impose a new protocol for communication between remote engine client and service
  • Avoid attaching the schema information in the payload
  • Do the encoding work outside the io thread pool

Test Plan

Proved 60% traffic drop and 25% cpu performance improvement in one of our prod environment.

@ShiKaiWi ShiKaiWi force-pushed the improve-remote-engine-service branch from 83a891c to 35d1d81 Compare August 14, 2023 01:54
server/src/grpc/remote_engine_service/mod.rs Show resolved Hide resolved
table_engine/src/remote/model.rs Outdated Show resolved Hide resolved
server/src/grpc/remote_engine_service/mod.rs Outdated Show resolved Hide resolved
table_engine/src/remote/model.rs Show resolved Hide resolved
Copy link
Contributor

@jiacai2050 jiacai2050 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShiKaiWi ShiKaiWi merged commit 4185b66 into apache:main Aug 14, 2023
5 checks passed
@ShiKaiWi ShiKaiWi deleted the improve-remote-engine-service branch August 14, 2023 04:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants