-
Notifications
You must be signed in to change notification settings - Fork 509
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HDDS-10890. [hsync] Increase default value for hdds.container.ratis.log.appender.queue.num-elements. #6711
Conversation
…og.appender.queue.num-elements. Change-Id: Ie3d85e7cd89fae7fae1fea19c6d770d53d5bfb85
With this configuration, 1 client thread: sudo -u hdfs ozone freon dne --clients=32 --container-id=4 -t 1 -n 1000000 --ratis --sleep-time-ms=0
32 clients: sudo -u hdfs ozone freon dne --clients=32 --container-id=4 -t 32 -n 1000000 --ratis --sleep-time-ms=0
16k per second is about the OM echo throughput on the same cluster. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 the change looks good.
The change looks good to me. It's quite interesting why it's 1 previously. |
Thanks @jojochuang for the patch, @ChenSammi, @szetszwo for the review. |
…der.queue.num-elements (apache#6711)
…concile-cli * HDDS-10239-container-reconciliation: (296 commits) HDDS-10897. Refactor OzoneQuota (apache#6714) HDDS-10422. Fix some warnings about exposing internal representation in hdds-common (apache#6351) HDDS-10899. Refactor Lease callbacks (apache#6715) HDDS-10890. Increase default value for hdds.container.ratis.log.appender.queue.num-elements (apache#6711) HDDS-10832. Client should switch to streaming based on OpenKeySession replication (apache#6683) HDDS-10435. Support S3 object tags for existing requests (apache#6607) HDDS-10883. Improve logging in Recon for finalising DN logic. (apache#6704) HDDS-8752. Enable TestOzoneRpcClientAbstract#testOverWriteKeyWithAndWithOutVersioning (apache#6702) HDDS-10875. XceiverRatisServer#getRaftPeersInPipeline should be called before XceiverRatisServer#removeGroup (apache#6696) HDDS-10514. Recon - Provide DN decommissioning detailed status and info inline with current CLI command output. (apache#6376) HDDS-10878. Bump zstd-jni to 1.5.6-3 (apache#6701) HDDS-10877. Bump Dropwizard metrics to 3.2.6 (apache#6699) HDDS-10876. Bump jackson to 2.16.2 (apache#6697) HDDS-6116. Remove flaky tag from TestSCMInstallSnapshot (apache#6695) HDDS-2643. TestOzoneDelegationTokenSecretManager#testRenewTokenFailureRenewalTime fails intermittently. HDDS-10699. Refactor ContainerBalancerTask and TestContainerBalancerTask (apache#6537) HDDS-10861. Ozone cli supports default ozone.om.service.id (apache#6680) HDDS-10859. Improve error messages when decommission and maintenance fail-early (apache#6678) HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667) HDDS-10559. Add a warning or a check to run repair tool as System user (apache#6574) ... Conflicts: hadoop-ozone/dist/src/main/smoketest/admincli/container.robot
What changes were proposed in this pull request?
HDDS-10890. [hsync] Increase default value for hdds.container.ratis.log.appender.queue.num-elements.
Please describe your PR in detail:
Using Freon DN Echo tool, I found that increasing hdds.container.ratis.log.appender.queue.num-elements value drastically improve DN Echo throughput and latency.
Set it to 1024 to be consistent with OM and SCM.
ozone.om.ratis.log.appender.queue.num-elements
ozone.scm.ha.ratis.log.appender.queue.num-elements
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10890
How was this patch tested?
Benchmarked on a real cluster