Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcdctl check: add load test #9121

Closed
xiang90 opened this issue Jan 9, 2018 · 12 comments
Closed

etcdctl check: add load test #9121

xiang90 opened this issue Jan 9, 2018 · 12 comments
Assignees
Milestone

Comments

@xiang90
Copy link
Contributor

xiang90 commented Jan 9, 2018

we have performance test to ensure the cluster's performance is good enough.

on the other dimension, we probably want to ensure the cluster can hold enough data without OOM.

I would suggest to start with:

  1. small with 10,000 1kb kv pairs
  2. medium with 100,000 1kb kv pairs
  3. large with 1,000,000 1kb kv pairs
  4. xlarge with 3,000,000 1kb kv pairs

/cc @spzala maybe you are interested in this one as you worked on the doc for perf check.

@xiang90
Copy link
Contributor Author

xiang90 commented Jan 9, 2018

this can help us to have a unified load test, and handle issues like #9112 much easier.

@xiang90 xiang90 added this to the v3.4.0 milestone Jan 9, 2018
@CaoShuFeng
Copy link

CaoShuFeng commented Jan 9, 2018

It will be very useful for us to know much performance degradation will occur when kv paris become larger and larger.
So I'd suggest to test the read/write speed here.

@xiang90
Copy link
Contributor Author

xiang90 commented Jan 9, 2018

It will be very useful for us to know much performance degradation will occur when kv paris become larger and larger.

For that, you should try out benchmark tool in this repo. If you have a specific use case, you should do it yourself.

@CaoShuFeng
Copy link

For that, you should try out benchmark tool in this repo. If you have a specific use case, you should do it yourself.

Oh, there is already a benchmark tool!
Thanks

@spzala
Copy link
Member

spzala commented Jan 11, 2018

@xiang90 sure, sorry missed it earlier. I will be working on it. Thanks!

@spzala spzala self-assigned this Jan 11, 2018
@spzala spzala added the WIP label Jan 12, 2018
@spzala
Copy link
Member

spzala commented Jan 14, 2018

@xiang90 hi, to play with benchmark I was trying to install/run it as go get github.com/coreos/etcd/tools/benchmark and go run tools/benchmark/main.go but both gives me following errros

go get github.com/coreos/etcd/tools/benchmark
# gopkg.in/cheggaaa/pb.v1
../go/src/gopkg.in/cheggaaa/pb.v1/pb_x.go:43:13: undefined: unix.IoctlGetWinsize
../go/src/gopkg.in/cheggaaa/pb.v1/pb_x.go:60:26: undefined: unix.IoctlGetTermios
../go/src/gopkg.in/cheggaaa/pb.v1/pb_x.go:70:12: undefined: unix.IoctlSetTermios
../go/src/gopkg.in/cheggaaa/pb.v1/pb_x.go:88:12: undefined: unix.IoctlSetTermios

Am I doing something wrong? Is this something am missing in my environment? Googling on the error gives me very little information related to golang. Thanks for any help!

@spzala
Copy link
Member

spzala commented Jan 14, 2018

I am running go version go1.9.2 linux/amd64

@gyuho
Copy link
Contributor

gyuho commented Jan 14, 2018

@spzala Try go install -v ./cmd/tools/benchmark

@spzala
Copy link
Member

spzala commented Jan 14, 2018

Aha, that's it!! Worked like a champ. Thanks @gyuho

@spzala
Copy link
Member

spzala commented Jan 17, 2018

@xiang90 hello, if I understood the issue correctly, we need to test the cases you have suggested manually and make sure how much physical memory is used. And then create a test similar to one in benchmark put.go but with memory usage? or are you referring a test related to check perf? Thanks! /cc @gyuho

@xiang90
Copy link
Contributor Author

xiang90 commented Jan 17, 2018

@spzala

Right now, in etcdctl there is a perf check command: https://github.com/coreos/etcd/tree/master/etcdctl#check-perf-options.

etcdctl usually gets installed by users, and the perf check command is very simple and intuitive

etcdctl check perf --load=small

the benchmark tool can also measure throughput/latency like perf check. but it is designed for flexibility not for ease of use. users have to provide all sorts of parameters instead of just small or large.

similarly, we want to add a easy to use command etcdctl check datascale to ensure the deployment can hold the given set of data. the command should measure the memory usage, and report success if the memory usage is <= 60% (or some other reasonable value) of the memory on the machine.

@spzala
Copy link
Member

spzala commented Jan 17, 2018

@xiang90 perfect, thank you so much!!

spzala added a commit to spzala/etcd that referenced this issue Jan 22, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads. Return user with a message
that whether there are enough memory for a given workload with pass
or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Jan 22, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads. Return user with a message
that whether there are enough memory for a given workload with pass
or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Jan 22, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads. Return user with a message
that whether there are enough memory for a given workload with pass
or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Jan 24, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads. Return user with a message
that whether there are enough memory for a given workload with pass
or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Jan 26, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads. Return user with a message
that whether there are enough memory for a given workload with pass
or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 8, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads. Return user with a message
that whether there are enough memory for a given workload with pass
or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 14, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads on a given endpoint. If no endpoint
is provided, localhost will be used. Return user with a message that whether
there are enough memory for a given workload with pass or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 14, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads on a given endpoint. If no endpoint
is provided, localhost will be used. Return user with a message that whether
there are enough memory for a given workload with pass or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 15, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads on a given endpoint. If no endpoint
is provided, localhost will be used. Return user with a message that whether
there are enough memory for a given workload with pass or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 15, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads on a given endpoint. If no endpoint
is provided, localhost will be used. Return user with a message that whether
there are enough memory for a given workload with pass or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 16, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads. Return user with a message
that whether there are enough memory for a given workload with pass
or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 16, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads. Return user with a message
that whether there are enough memory for a given workload with pass
or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 16, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads. Return user with a message
that whether there are enough memory for a given workload with pass
or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 16, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads. Return user with a message
that whether there are enough memory for a given workload with pass
or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 16, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads on a given endpoint. If no endpoint
is provided, localhost will be used. Return user with a message that whether
there are enough memory for a given workload with pass or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 16, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads on a given endpoint. If no endpoint
is provided, localhost will be used. Return user with a message that whether
there are enough memory for a given workload with pass or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 16, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads on a given endpoint. If no endpoint
is provided, localhost will be used. Return user with a message that whether
there are enough memory for a given workload with pass or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 16, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads on a given endpoint. If no endpoint
is provided, localhost will be used. Return user with a message that whether
there are enough memory for a given workload with pass or fail.

Fixed etcd-io#9121
spzala added a commit to spzala/etcd that referenced this issue Feb 17, 2018
Create a new command similar to check perf that can check the memory
consumption for putting different workloads on a given endpoint. If no endpoint
is provided, localhost will be used. Return user with a message that whether
there are enough memory for a given workload with pass or fail.

Fixed etcd-io#9121
@spzala spzala removed the WIP label Aug 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

4 participants