Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

integration/clientv3/examples test flakes frequently #12983

Closed
ptabor opened this issue May 17, 2021 · 6 comments
Closed

integration/clientv3/examples test flakes frequently #12983

ptabor opened this issue May 17, 2021 · 6 comments
Labels
backport/v3.5 priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. stale type/flake
Milestone
etcd-v3.5

Comments

@ptabor
Copy link
Contributor

ptabor commented May 17, 2021

ExampleCluster_memberAddAsLearner

I managed to repro this with:

for i in `seq 1 100`; do (cd tests && 'env' 'go' 'test' '-timeout=15m' '--race=false' '--cpu=4' './integration/clientv3/examples' --count=1 -v -run ExampleCluster_memberAddAsLearner| tee log.log); done

but it does not always flake.

Flake on Actions:
https://github.com/etcd-io/etcd/pull/12981/checks?check_run_id=2595306923

Uploaded log: logs_1206.zip

2021/05/16 18:49:34 2021-05-16T18:49:34.400Z	INFO	m2	started stream writer with remote peer	{"member": "m2", "local-member-id": "685c5c71ce9a9328", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.400Z	INFO	m2	started stream writer with remote peer	{"member": "m2", "local-member-id": "685c5c71ce9a9328", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.400Z	INFO	m2	started stream reader with remote peer	{"member": "m2", "stream-reader-type": "stream MsgApp v2", "local-member-id": "685c5c71ce9a9328", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.400Z	INFO	m0	started remote peer	{"member": "m0", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.400Z	INFO	m0	added remote peer	{"member": "m0", "local-member-id": "c5c5a20ca4073d6a", "remote-peer-id": "3b13bbeaeef551eb", "remote-peer-urls": ["http://localhost:32381"]}
2021/05/16 18:49:34 2021-05-16T18:49:34.400Z	INFO	m0	started stream writer with remote peer	{"member": "m0", "local-member-id": "c5c5a20ca4073d6a", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.400Z	INFO	m0	started stream writer with remote peer	{"member": "m0", "local-member-id": "c5c5a20ca4073d6a", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.400Z	INFO	m0	started stream reader with remote peer	{"member": "m0", "stream-reader-type": "stream MsgApp v2", "local-member-id": "c5c5a20ca4073d6a", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.400Z	INFO	m0	started stream reader with remote peer	{"member": "m0", "stream-reader-type": "stream Message", "local-member-id": "c5c5a20ca4073d6a", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.400Z	INFO	m2	started stream reader with remote peer	{"member": "m2", "stream-reader-type": "stream Message", "local-member-id": "685c5c71ce9a9328", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.401Z	INFO	m0	applied a configuration change through raft	{"member": "m0", "local-member-id": "c5c5a20ca4073d6a", "raft-conf-change": "ConfChangeAddLearnerNode", "raft-conf-change-node-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.401Z	INFO	m2.raft	685c5c71ce9a9328 ignoring conf change {ConfChangeRemoveNode 4256952690501046763 [] 4425483133757641240} at config voters=(7519987121669182248 13966091899056808041 14250974771059047786) learners=(4256952690501046763): possible unapplied conf change at index 25 (applied to 24)	{"member": "m2"}
2021/05/16 18:49:34 2021-05-16T18:49:34.401Z	INFO	m1.raft	c1d186962fbde069 switched to configuration voters=(7519987121669182248 13966091899056808041 14250974771059047786) learners=(4256952690501046763)	{"member": "m1"}
2021/05/16 18:49:34 2021-05-16T18:49:34.402Z	INFO	m1	added member	{"member": "m1", "cluster-id": "1cfa2497ce0563d4", "local-member-id": "c1d186962fbde069", "added-peer-id": "3b13bbeaeef551eb", "added-peer-peer-urls": ["http://localhost:32381"]}
2021/05/16 18:49:34 2021-05-16T18:49:34.402Z	INFO	m1	starting remote peer	{"member": "m1", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.402Z	INFO	m1	started HTTP pipelining with remote peer	{"member": "m1", "local-member-id": "c1d186962fbde069", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.402Z	WARN	m2	failed to reach the peer URL	{"member": "m2", "address": "http://localhost:32381/version", "remote-member-id": "3b13bbeaeef551eb", "error": "Get \"http://localhost:32381/version\": dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 18:49:34 2021-05-16T18:49:34.402Z	WARN	m2	failed to get version	{"member": "m2", "remote-member-id": "3b13bbeaeef551eb", "error": "Get \"http://localhost:32381/version\": dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 18:49:34 2021-05-16T18:49:34.402Z	INFO	m1	started stream writer with remote peer	{"member": "m1", "local-member-id": "c1d186962fbde069", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.402Z	INFO	m1	started stream writer with remote peer	{"member": "m1", "local-member-id": "c1d186962fbde069", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.402Z	INFO	m1	started remote peer	{"member": "m1", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.402Z	INFO	m1	added remote peer	{"member": "m1", "local-member-id": "c1d186962fbde069", "remote-peer-id": "3b13bbeaeef551eb", "remote-peer-urls": ["http://localhost:32381"]}
2021/05/16 18:49:34 2021-05-16T18:49:34.402Z	INFO	m1	started stream reader with remote peer	{"member": "m1", "stream-reader-type": "stream MsgApp v2", "local-member-id": "c1d186962fbde069", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:34 2021-05-16T18:49:34.403Z	INFO	m1	started stream reader with remote peer	{"member": "m1", "stream-reader-type": "stream Message", "local-member-id": "c1d186962fbde069", "remote-peer-id": "3b13bbeaeef551eb"}
2021/05/16 18:49:38 2021-05-16T18:49:38.404Z	WARN	m2	failed to reach the peer URL	{"member": "m2", "address": "http://localhost:32381/version", "remote-member-id": "3b13bbeaeef551eb", "error": "Get \"http://localhost:32381/version\": dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 18:49:38 2021-05-16T18:49:38.404Z	WARN	m2	failed to get version	{"member": "m2", "remote-member-id": "3b13bbeaeef551eb", "error": "Get \"http://localhost:32381/version\": dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 18:49:39 2021-05-16T18:49:39.400Z	WARN	m2	prober detected unhealthy status	{"member": "m2", "round-tripper-name": "ROUND_TRIPPER_RAFT_MESSAGE", "remote-peer-id": "3b13bbeaeef551eb", "rtt": "0s", "error": "dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 18:49:39 2021-05-16T18:49:39.401Z	WARN	m0	prober detected unhealthy status	{"member": "m0", "round-tripper-name": "ROUND_TRIPPER_RAFT_MESSAGE", "remote-peer-id": "3b13bbeaeef551eb", "rtt": "0s", "error": "dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 18:49:39 2021-05-16T18:49:39.401Z	WARN	m0	prober detected unhealthy status	{"member": "m0", "round-tripper-name": "ROUND_TRIPPER_SNAPSHOT", "remote-peer-id": "3b13bbeaeef551eb", "rtt": "0s", "error": "dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 18:49:39 2021-05-16T18:49:39.401Z	WARN	m2	prober detected unhealthy status	{"member": "m2", "round-tripper-name": "ROUND_TRIPPER_SNAPSHOT", "remote-peer-id": "3b13bbeaeef551eb", "rtt": "0s", "error": "dial tcp [::1]:32381: connect: connection refused"}

This line looks suspecious

2021/05/16 18:49:34 2021-05-16T18:49:34.401Z	INFO	m2.raft	685c5c71ce9a9328 ignoring conf change {ConfChangeRemoveNode 4256952690501046763 [] 4425483133757641240} at config voters=(7519987121669182248 13966091899056808041 14250974771059047786) learners=(4256952690501046763): possible unapplied conf change at index 25 (applied to 24)	{"member": "m2"}

goroutine 1 [select, 14 minutes]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0003d70e0)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.37.0/internal/transport/transport.go:322 +0x99
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.37.0/internal/transport/transport.go:337
google.golang.org/grpc.(*csAttempt).recvMsg(0xc003a3ef00, 0x1025000, 0xc000284440, 0x0, 0x0, 0x0)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.37.0/stream.go:937 +0x731
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc003a3ef00, 0xc0037fcbe0, 0xa)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.37.0/stream.go:802 +0x46
google.golang.org/grpc.(*clientStream).withRetry(0xc0003d6ea0, 0xc000379298, 0xc000379268, 0xc0037fcbea, 0x9c7a49)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.37.0/stream.go:660 +0x9f
google.golang.org/grpc.(*clientStream).RecvMsg(0xc0003d6ea0, 0x1025000, 0xc000284440, 0x0, 0x0)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.37.0/stream.go:801 +0x105
google.golang.org/grpc.invoke(0x11d7738, 0xc0038c1cb0, 0x109b415, 0x22, 0x101aea0, 0xc0038c1c50, 0x1025000, 0xc000284440, 0xc002d5e000, 0xc0002844c0, ...)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.37.0/call.go:73 +0x142
go.etcd.io/etcd/client/v3.(*Client).unaryClientInterceptor.func1(0x11d76c8, 0xc0038c1cb0, 0x109b415, 0x22, 0x101aea0, 0xc0038c1c50, 0x1025000, 0xc000284440, 0xc002d5e000, 0x10c6d98, ...)
	/home/runner/work/etcd/etcd/client/v3/retry_interceptor.go:58 +0x46a
google.golang.org/grpc.(*ClientConn).Invoke(0xc002d5e000, 0x11d76c8, 0xc00011e010, 0x109b415, 0x22, 0x101aea0, 0xc0038c1c50, 0x1025000, 0xc000284440, 0x1823240, ...)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.37.0/call.go:35 +0x109
go.etcd.io/etcd/api/v3/etcdserverpb.(*clusterClient).MemberRemove(0xc00018e108, 0x11d76c8, 0xc00011e010, 0xc0038c1c50, 0x1823240, 0x3, 0x3, 0xfc4be0, 0x1, 0xc0038c1c50)
	/home/runner/work/etcd/etcd/api/etcdserverpb/rpc.pb.go:7083 +0xcf
go.etcd.io/etcd/client/v3.(*retryClusterClient).MemberRemove(0xc0039fd310, 0x11d76c8, 0xc00011e010, 0xc0038c1c50, 0x1823240, 0x3, 0x3, 0xc0032fc4b0, 0x0, 0x0)
	/home/runner/work/etcd/etcd/client/v3/retry.go:175 +0x7c
go.etcd.io/etcd/client/v3.(*cluster).MemberRemove(0xc003a42db0, 0x11d76c8, 0xc00011e010, 0x3b13bbeaeef551eb, 0x1, 0x1, 0xc0032fc4b0)
	/home/runner/work/etcd/etcd/client/v3/cluster.go:103 +0x88
go.etcd.io/etcd/tests/v3/integration/clientv3/examples_test.ExampleCluster_memberAddAsLearner.func1()
	/home/runner/work/etcd/etcd/tests/integration/clientv3/examples/example_cluster_test.go:106 +0x247
go.etcd.io/etcd/tests/v3/integration/clientv3/examples_test.forUnitTestsRunInMockedContext(...)
	/home/runner/work/etcd/etcd/tests/integration/clientv3/examples/main_test.go:40
go.etcd.io/etcd/tests/v3/integration/clientv3/examples_test.ExampleCluster_memberAddAsLearner()
	/home/runner/work/etcd/etcd/tests/integration/clientv3/examples/example_cluster_test.go:90 +0x2b
testing.runExample(0x1099ea7, 0x21, 0x10c6868, 0x10a96f3, 0x2e, 0x0, 0x0)
	/opt/hostedtoolcache/go/1.16.4/x64/src/testing/run_example.go:63 +0x222
testing.runExamples(0xc000379e58, 0x1816a80, 0x18, 0x18, 0xc02079108fcf076d)
	/opt/hostedtoolcache/go/1.16.4/x64/src/testing/example.go:44 +0x17a
testing.(*M).Run(0xc0001f0380, 0x0)
	/opt/hostedtoolcache/go/1.16.4/x64/src/testing/testing.go:1418 +0x273
go.etcd.io/etcd/tests/v3/integration/clientv3/examples_test.TestMain(0xc0001f0380)
	/home/runner/work/etcd/etcd/tests/integration/clientv3/examples/main_test.go:46 +0x48
main.main()
	_testmain.go:91 +0x165

ptab@ptab ~/corp/etcd% (cd tests && 'env' 'go' 'test' '-timeout=15m' '--race=false' '--cpu=4' './integration/clientv3/examples' --count=1 -v -run ExampleCluster_memberAddAsLearner| tee log.log)
=== RUN   ExampleCluster_memberAddAsLearner
2021/05/16 22:58:56 Working directory '/home/ptab/corp/etcd/tests/integration/clientv3/examples' expected to be in temp-dir ('/tmp').Have you executed integration.BeforeTest(t) ?
2021/05/16 22:58:56 2021-05-16T22:58:56.587+0200	INFO	m0	LISTEN GRPC	{"member": "m0", "m.grpcAddr": "localhost:m0", "m.Name": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.587+0200	INFO	m1	LISTEN GRPC	{"member": "m1", "m.grpcAddr": "localhost:m1", "m.Name": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.587+0200	INFO	m2	LISTEN GRPC	{"member": "m2", "m.grpcAddr": "localhost:m2", "m.Name": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.587+0200	INFO	m2	launching a member	{"member": "m2", "name": "m2", "advertise-peer-urls": ["unix://127.0.0.1:210053101996"], "listen-client-urls": ["unix://127.0.0.1:210063101996"], "grpc-address": "unix://localhost:m20"}
2021/05/16 22:58:56 2021-05-16T22:58:56.587+0200	INFO	m1	launching a member	{"member": "m1", "name": "m1", "advertise-peer-urls": ["unix://127.0.0.1:210033101996"], "listen-client-urls": ["unix://127.0.0.1:210043101996"], "grpc-address": "unix://localhost:m10"}
2021/05/16 22:58:56 2021-05-16T22:58:56.587+0200	INFO	m0	launching a member	{"member": "m0", "name": "m0", "advertise-peer-urls": ["unix://127.0.0.1:210013101996"], "listen-client-urls": ["unix://127.0.0.1:210023101996"], "grpc-address": "unix://localhost:m00"}
2021/05/16 22:58:56 2021-05-16T22:58:56.605+0200	INFO	m1	opened backend db	{"member": "m1", "path": "/tmp/lazy_cluster813018228/etcd819471171/member/snap/db", "took": "17.295908ms"}
2021/05/16 22:58:56 2021-05-16T22:58:56.605+0200	INFO	m0	opened backend db	{"member": "m0", "path": "/tmp/lazy_cluster668773490/etcd777733929/member/snap/db", "took": "17.274012ms"}
2021/05/16 22:58:56 2021-05-16T22:58:56.605+0200	INFO	m2	opened backend db	{"member": "m2", "path": "/tmp/lazy_cluster025711558/etcd404414061/member/snap/db", "took": "17.357534ms"}
2021/05/16 22:58:56 2021-05-16T22:58:56.637+0200	INFO	m2	starting local member	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "cluster-id": "3c5b384f03b0f420"}
2021/05/16 22:58:56 2021-05-16T22:58:56.637+0200	INFO	m2.raft	f22d0a0f9f1b431b switched to configuration voters=()	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.637+0200	INFO	m2.raft	f22d0a0f9f1b431b became follower at term 0	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.637+0200	INFO	m2.raft	newRaft f22d0a0f9f1b431b [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.637+0200	INFO	m2.raft	f22d0a0f9f1b431b became follower at term 1	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.637+0200	INFO	m2.raft	f22d0a0f9f1b431b switched to configuration voters=(11609101907503085289)	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.637+0200	INFO	m2.raft	f22d0a0f9f1b431b switched to configuration voters=(11609101907503085289 16575215055615236278)	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.637+0200	INFO	m2.raft	f22d0a0f9f1b431b switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.640+0200	INFO	m1	starting local member	{"member": "m1", "local-member-id": "e606fe4619a718b6", "cluster-id": "3c5b384f03b0f420"}
2021/05/16 22:58:56 2021-05-16T22:58:56.640+0200	INFO	m1.raft	e606fe4619a718b6 switched to configuration voters=()	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.640+0200	INFO	m1.raft	e606fe4619a718b6 became follower at term 0	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.640+0200	INFO	m1.raft	newRaft e606fe4619a718b6 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.640+0200	INFO	m1.raft	e606fe4619a718b6 became follower at term 1	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.640+0200	INFO	m1.raft	e606fe4619a718b6 switched to configuration voters=(11609101907503085289)	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.640+0200	INFO	m1.raft	e606fe4619a718b6 switched to configuration voters=(11609101907503085289 16575215055615236278)	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.640+0200	INFO	m1.raft	e606fe4619a718b6 switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.640+0200	INFO	m0	starting local member	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "cluster-id": "3c5b384f03b0f420"}
2021/05/16 22:58:56 2021-05-16T22:58:56.640+0200	INFO	m0.raft	a11bd0ad5e07f6e9 switched to configuration voters=()	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.640+0200	INFO	m0.raft	a11bd0ad5e07f6e9 became follower at term 0	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.641+0200	INFO	m0.raft	newRaft a11bd0ad5e07f6e9 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.641+0200	INFO	m0.raft	a11bd0ad5e07f6e9 became follower at term 1	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.641+0200	INFO	m0.raft	a11bd0ad5e07f6e9 switched to configuration voters=(11609101907503085289)	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.641+0200	INFO	m0.raft	a11bd0ad5e07f6e9 switched to configuration voters=(11609101907503085289 16575215055615236278)	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.641+0200	INFO	m0.raft	a11bd0ad5e07f6e9 switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.648+0200	WARN	m2	simple token is not cryptographically signed	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.651+0200	WARN	m1	simple token is not cryptographically signed	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.651+0200	WARN	m0	simple token is not cryptographically signed	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.652+0200	INFO	m2	kvstore restored	{"member": "m2", "current-rev": 1}
2021/05/16 22:58:56 2021-05-16T22:58:56.655+0200	INFO	m1	kvstore restored	{"member": "m1", "current-rev": 1}
2021/05/16 22:58:56 2021-05-16T22:58:56.658+0200	INFO	m0	kvstore restored	{"member": "m0", "current-rev": 1}
2021/05/16 22:58:56 2021-05-16T22:58:56.658+0200	INFO	m2	enabled backend quota with default value	{"member": "m2", "quota-name": "v3-applier", "quota-size-bytes": 2147483648, "quota-size": "2.1 GB"}
2021/05/16 22:58:56 2021-05-16T22:58:56.661+0200	INFO	m2	starting remote peer	{"member": "m2", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.661+0200	INFO	m2	started HTTP pipelining with remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.662+0200	INFO	m2	started stream writer with remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.663+0200	INFO	m2	started stream writer with remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.663+0200	INFO	m1	starting remote peer	{"member": "m1", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.663+0200	INFO	m1	started HTTP pipelining with remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.664+0200	INFO	m1	started stream writer with remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.665+0200	INFO	m2	started remote peer	{"member": "m2", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.665+0200	INFO	m2	added remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "a11bd0ad5e07f6e9", "remote-peer-urls": ["unix://127.0.0.1:210013101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.665+0200	INFO	m2	starting remote peer	{"member": "m2", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.665+0200	INFO	m2	started HTTP pipelining with remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.665+0200	INFO	m2	started stream reader with remote peer	{"member": "m2", "stream-reader-type": "stream Message", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.666+0200	INFO	m2	started stream reader with remote peer	{"member": "m2", "stream-reader-type": "stream MsgApp v2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.666+0200	INFO	m0	starting remote peer	{"member": "m0", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.666+0200	INFO	m0	started HTTP pipelining with remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.669+0200	INFO	m2	started stream writer with remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.670+0200	INFO	m1	started remote peer	{"member": "m1", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.670+0200	INFO	m1	added remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "a11bd0ad5e07f6e9", "remote-peer-urls": ["unix://127.0.0.1:210013101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.670+0200	INFO	m1	starting remote peer	{"member": "m1", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.670+0200	INFO	m1	started HTTP pipelining with remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.671+0200	INFO	m0	started remote peer	{"member": "m0", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.672+0200	INFO	m0	added remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "e606fe4619a718b6", "remote-peer-urls": ["unix://127.0.0.1:210033101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.672+0200	INFO	m0	starting remote peer	{"member": "m0", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.672+0200	INFO	m0	started HTTP pipelining with remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.673+0200	INFO	m1	started stream writer with remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.673+0200	INFO	m1	started stream writer with remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.673+0200	INFO	m0	started stream reader with remote peer	{"member": "m0", "stream-reader-type": "stream MsgApp v2", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.673+0200	INFO	m1	started stream reader with remote peer	{"member": "m1", "stream-reader-type": "stream Message", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.673+0200	INFO	m0	started stream reader with remote peer	{"member": "m0", "stream-reader-type": "stream Message", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.672+0200	INFO	m1	started stream reader with remote peer	{"member": "m1", "stream-reader-type": "stream MsgApp v2", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.673+0200	INFO	m2	started stream writer with remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.673+0200	INFO	m1	started stream writer with remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.673+0200	INFO	m0	started stream writer with remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m0	started stream writer with remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m2	started remote peer	{"member": "m2", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m2	added remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "e606fe4619a718b6", "remote-peer-urls": ["unix://127.0.0.1:210033101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m2	starting etcd server	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "local-server-version": "3.5.0-alpha.0", "cluster-version": "to_be_decided"}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m1	started remote peer	{"member": "m1", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m1	added remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "f22d0a0f9f1b431b", "remote-peer-urls": ["unix://127.0.0.1:210053101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m1	starting etcd server	{"member": "m1", "local-member-id": "e606fe4619a718b6", "local-server-version": "3.5.0-alpha.0", "cluster-version": "to_be_decided"}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m1	started stream reader with remote peer	{"member": "m1", "stream-reader-type": "stream MsgApp v2", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m1	started stream reader with remote peer	{"member": "m1", "stream-reader-type": "stream Message", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m0	started remote peer	{"member": "m0", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m0	added remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "f22d0a0f9f1b431b", "remote-peer-urls": ["unix://127.0.0.1:210053101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m0	starting etcd server	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "local-server-version": "3.5.0-alpha.0", "cluster-version": "to_be_decided"}
2021/05/16 22:58:56 2021-05-16T22:58:56.674+0200	INFO	m0	started stream writer with remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.675+0200	INFO	m0	started stream writer with remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.675+0200	INFO	m0	started stream reader with remote peer	{"member": "m0", "stream-reader-type": "stream MsgApp v2", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.675+0200	INFO	m0	started stream reader with remote peer	{"member": "m0", "stream-reader-type": "stream Message", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.675+0200	INFO	m1	starting initial election tick advance	{"member": "m1", "election-ticks": 10}
2021/05/16 22:58:56 2021-05-16T22:58:56.676+0200	INFO	m1.raft	e606fe4619a718b6 switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.676+0200	INFO	m1	added member	{"member": "m1", "cluster-id": "3c5b384f03b0f420", "local-member-id": "e606fe4619a718b6", "added-peer-id": "a11bd0ad5e07f6e9", "added-peer-peer-urls": ["unix://127.0.0.1:210013101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.676+0200	INFO	m0	starting initial election tick advance	{"member": "m0", "election-ticks": 10}
2021/05/16 22:58:56 2021-05-16T22:58:56.676+0200	INFO	m1.raft	e606fe4619a718b6 switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.677+0200	INFO	m1	added member	{"member": "m1", "cluster-id": "3c5b384f03b0f420", "local-member-id": "e606fe4619a718b6", "added-peer-id": "e606fe4619a718b6", "added-peer-peer-urls": ["unix://127.0.0.1:210033101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.677+0200	INFO	m1.raft	e606fe4619a718b6 switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.677+0200	INFO	m1	added member	{"member": "m1", "cluster-id": "3c5b384f03b0f420", "local-member-id": "e606fe4619a718b6", "added-peer-id": "f22d0a0f9f1b431b", "added-peer-peer-urls": ["unix://127.0.0.1:210053101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.677+0200	INFO	m0.raft	a11bd0ad5e07f6e9 switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.677+0200	INFO	m2	starting initial election tick advance	{"member": "m2", "election-ticks": 10}
2021/05/16 22:58:56 2021-05-16T22:58:56.677+0200	INFO	m2	started stream reader with remote peer	{"member": "m2", "stream-reader-type": "stream Message", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.677+0200	INFO	m2.raft	f22d0a0f9f1b431b switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.677+0200	INFO	m0	added member	{"member": "m0", "cluster-id": "3c5b384f03b0f420", "local-member-id": "a11bd0ad5e07f6e9", "added-peer-id": "a11bd0ad5e07f6e9", "added-peer-peer-urls": ["unix://127.0.0.1:210013101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.678+0200	INFO	m2	added member	{"member": "m2", "cluster-id": "3c5b384f03b0f420", "local-member-id": "f22d0a0f9f1b431b", "added-peer-id": "a11bd0ad5e07f6e9", "added-peer-peer-urls": ["unix://127.0.0.1:210013101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.678+0200	INFO	m0.raft	a11bd0ad5e07f6e9 switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.678+0200	INFO	m2.raft	f22d0a0f9f1b431b switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.678+0200	INFO	m0	added member	{"member": "m0", "cluster-id": "3c5b384f03b0f420", "local-member-id": "a11bd0ad5e07f6e9", "added-peer-id": "e606fe4619a718b6", "added-peer-peer-urls": ["unix://127.0.0.1:210033101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.678+0200	INFO	m2	added member	{"member": "m2", "cluster-id": "3c5b384f03b0f420", "local-member-id": "f22d0a0f9f1b431b", "added-peer-id": "e606fe4619a718b6", "added-peer-peer-urls": ["unix://127.0.0.1:210033101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.678+0200	INFO	m0.raft	a11bd0ad5e07f6e9 switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.678+0200	INFO	m2.raft	f22d0a0f9f1b431b switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227)	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.678+0200	INFO	m0	added member	{"member": "m0", "cluster-id": "3c5b384f03b0f420", "local-member-id": "a11bd0ad5e07f6e9", "added-peer-id": "f22d0a0f9f1b431b", "added-peer-peer-urls": ["unix://127.0.0.1:210053101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.678+0200	INFO	m2	added member	{"member": "m2", "cluster-id": "3c5b384f03b0f420", "local-member-id": "f22d0a0f9f1b431b", "added-peer-id": "f22d0a0f9f1b431b", "added-peer-peer-urls": ["unix://127.0.0.1:210053101996"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.677+0200	INFO	m2	started stream reader with remote peer	{"member": "m2", "stream-reader-type": "stream MsgApp v2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.681+0200	INFO	m2	set message encoder	{"member": "m2", "from": "f22d0a0f9f1b431b", "to": "a11bd0ad5e07f6e9", "stream-type": "stream Message"}
2021/05/16 22:58:56 2021-05-16T22:58:56.681+0200	INFO	m0	set message encoder	{"member": "m0", "from": "a11bd0ad5e07f6e9", "to": "f22d0a0f9f1b431b", "stream-type": "stream Message"}
2021/05/16 22:58:56 2021-05-16T22:58:56.681+0200	INFO	m0	peer became active	{"member": "m0", "peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.681+0200	INFO	m0	established TCP streaming connection with remote peer	{"member": "m0", "stream-writer-type": "stream Message", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.681+0200	INFO	m2	peer became active	{"member": "m2", "peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.681+0200	INFO	m2	established TCP streaming connection with remote peer	{"member": "m2", "stream-writer-type": "stream Message", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.682+0200	INFO	m2	launched a member	{"member": "m2", "name": "m2", "advertise-peer-urls": ["unix://127.0.0.1:210053101996"], "listen-client-urls": ["unix://127.0.0.1:210063101996"], "grpc-address": "unix://localhost:m20"}
2021/05/16 22:58:56 2021-05-16T22:58:56.682+0200	INFO	m1	launched a member	{"member": "m1", "name": "m1", "advertise-peer-urls": ["unix://127.0.0.1:210033101996"], "listen-client-urls": ["unix://127.0.0.1:210043101996"], "grpc-address": "unix://localhost:m10"}
2021/05/16 22:58:56 2021-05-16T22:58:56.682+0200	INFO	m0	established TCP streaming connection with remote peer	{"member": "m0", "stream-reader-type": "stream Message", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.682+0200	INFO	m2	established TCP streaming connection with remote peer	{"member": "m2", "stream-reader-type": "stream Message", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.682+0200	INFO	m2	set message encoder	{"member": "m2", "from": "f22d0a0f9f1b431b", "to": "e606fe4619a718b6", "stream-type": "stream MsgApp v2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.683+0200	INFO	m2	peer became active	{"member": "m2", "peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.683+0200	INFO	m2	established TCP streaming connection with remote peer	{"member": "m2", "stream-writer-type": "stream MsgApp v2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.683+0200	INFO	m2	set message encoder	{"member": "m2", "from": "f22d0a0f9f1b431b", "to": "e606fe4619a718b6", "stream-type": "stream Message"}
2021/05/16 22:58:56 2021-05-16T22:58:56.683+0200	INFO	m2	established TCP streaming connection with remote peer	{"member": "m2", "stream-writer-type": "stream Message", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.685+0200	INFO	m2	set message encoder	{"member": "m2", "from": "f22d0a0f9f1b431b", "to": "a11bd0ad5e07f6e9", "stream-type": "stream MsgApp v2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.685+0200	INFO	m2	established TCP streaming connection with remote peer	{"member": "m2", "stream-writer-type": "stream MsgApp v2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.685+0200	INFO	m0	launched a member	{"member": "m0", "name": "m0", "advertise-peer-urls": ["unix://127.0.0.1:210013101996"], "listen-client-urls": ["unix://127.0.0.1:210023101996"], "grpc-address": "unix://localhost:m00"}
2021/05/16 22:58:56 2021-05-16T22:58:56.686+0200	INFO	m2	established TCP streaming connection with remote peer	{"member": "m2", "stream-reader-type": "stream MsgApp v2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.686+0200	INFO	m1	peer became active	{"member": "m1", "peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.686+0200	INFO	m1	established TCP streaming connection with remote peer	{"member": "m1", "stream-reader-type": "stream MsgApp v2", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.686+0200	INFO	m2	established TCP streaming connection with remote peer	{"member": "m2", "stream-reader-type": "stream MsgApp v2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m1	set message encoder	{"member": "m1", "from": "e606fe4619a718b6", "to": "a11bd0ad5e07f6e9", "stream-type": "stream MsgApp v2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m1	peer became active	{"member": "m1", "peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m1	established TCP streaming connection with remote peer	{"member": "m1", "stream-writer-type": "stream MsgApp v2", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m0	set message encoder	{"member": "m0", "from": "a11bd0ad5e07f6e9", "to": "f22d0a0f9f1b431b", "stream-type": "stream MsgApp v2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m0	established TCP streaming connection with remote peer	{"member": "m0", "stream-writer-type": "stream MsgApp v2", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m1	set message encoder	{"member": "m1", "from": "e606fe4619a718b6", "to": "f22d0a0f9f1b431b", "stream-type": "stream MsgApp v2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m1	established TCP streaming connection with remote peer	{"member": "m1", "stream-writer-type": "stream MsgApp v2", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m1	set message encoder	{"member": "m1", "from": "e606fe4619a718b6", "to": "a11bd0ad5e07f6e9", "stream-type": "stream Message"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m1	established TCP streaming connection with remote peer	{"member": "m1", "stream-writer-type": "stream Message", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m1	set message encoder	{"member": "m1", "from": "e606fe4619a718b6", "to": "f22d0a0f9f1b431b", "stream-type": "stream Message"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m1	established TCP streaming connection with remote peer	{"member": "m1", "stream-writer-type": "stream Message", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m2	established TCP streaming connection with remote peer	{"member": "m2", "stream-reader-type": "stream Message", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.687+0200	INFO	m1	established TCP streaming connection with remote peer	{"member": "m1", "stream-reader-type": "stream MsgApp v2", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.688+0200	INFO	m0	set message encoder	{"member": "m0", "from": "a11bd0ad5e07f6e9", "to": "e606fe4619a718b6", "stream-type": "stream MsgApp v2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.688+0200	INFO	m0	peer became active	{"member": "m0", "peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.688+0200	INFO	m0	established TCP streaming connection with remote peer	{"member": "m0", "stream-writer-type": "stream MsgApp v2", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.688+0200	INFO	m0	established TCP streaming connection with remote peer	{"member": "m0", "stream-reader-type": "stream MsgApp v2", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.688+0200	INFO	m0	established TCP streaming connection with remote peer	{"member": "m0", "stream-reader-type": "stream Message", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.689+0200	INFO	m0	established TCP streaming connection with remote peer	{"member": "m0", "stream-reader-type": "stream MsgApp v2", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.690+0200	INFO	m0	set message encoder	{"member": "m0", "from": "a11bd0ad5e07f6e9", "to": "e606fe4619a718b6", "stream-type": "stream Message"}
2021/05/16 22:58:56 2021-05-16T22:58:56.690+0200	INFO	m0	established TCP streaming connection with remote peer	{"member": "m0", "stream-writer-type": "stream Message", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "e606fe4619a718b6"}
2021/05/16 22:58:56 2021-05-16T22:58:56.690+0200	INFO	m1	established TCP streaming connection with remote peer	{"member": "m1", "stream-reader-type": "stream Message", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "a11bd0ad5e07f6e9"}
2021/05/16 22:58:56 2021-05-16T22:58:56.690+0200	INFO	m1	established TCP streaming connection with remote peer	{"member": "m1", "stream-reader-type": "stream Message", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "f22d0a0f9f1b431b"}
2021/05/16 22:58:56 2021-05-16T22:58:56.726+0200	INFO	m1	initialized peer connections; fast-forwarding election ticks	{"member": "m1", "local-member-id": "e606fe4619a718b6", "forward-ticks": 8, "forward-duration": "80ms", "election-ticks": 10, "election-timeout": "100ms", "active-remote-members": 2}
2021/05/16 22:58:56 2021-05-16T22:58:56.726+0200	INFO	m1.raft	e606fe4619a718b6 is starting a new election at term 1	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.726+0200	INFO	m1.raft	e606fe4619a718b6 became candidate at term 2	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.726+0200	INFO	m1.raft	e606fe4619a718b6 received MsgVoteResp from e606fe4619a718b6 at term 2	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.726+0200	INFO	m1.raft	e606fe4619a718b6 [logterm: 1, index: 3] sent MsgVote request to a11bd0ad5e07f6e9 at term 2	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.726+0200	INFO	m1.raft	e606fe4619a718b6 [logterm: 1, index: 3] sent MsgVote request to f22d0a0f9f1b431b at term 2	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.726+0200	INFO	m0	initialized peer connections; fast-forwarding election ticks	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "forward-ticks": 8, "forward-duration": "80ms", "election-ticks": 10, "election-timeout": "100ms", "active-remote-members": 2}
2021/05/16 22:58:56 2021-05-16T22:58:56.728+0200	INFO	m2	initialized peer connections; fast-forwarding election ticks	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "forward-ticks": 8, "forward-duration": "80ms", "election-ticks": 10, "election-timeout": "100ms", "active-remote-members": 2}
2021/05/16 22:58:56 2021-05-16T22:58:56.728+0200	INFO	m2.raft	f22d0a0f9f1b431b [term: 1] received a MsgVote message with higher term from e606fe4619a718b6 [term: 2]	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.728+0200	INFO	m2.raft	f22d0a0f9f1b431b became follower at term 2	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.728+0200	INFO	m2.raft	f22d0a0f9f1b431b [logterm: 1, index: 3, vote: 0] cast MsgVote for e606fe4619a718b6 [logterm: 1, index: 3] at term 2	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.728+0200	INFO	m0.raft	a11bd0ad5e07f6e9 [term: 1] received a MsgVote message with higher term from e606fe4619a718b6 [term: 2]	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.728+0200	INFO	m0.raft	a11bd0ad5e07f6e9 became follower at term 2	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.728+0200	INFO	m0.raft	a11bd0ad5e07f6e9 [logterm: 1, index: 3, vote: 0] cast MsgVote for e606fe4619a718b6 [logterm: 1, index: 3] at term 2	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.730+0200	INFO	m1.raft	e606fe4619a718b6 received MsgVoteResp from a11bd0ad5e07f6e9 at term 2	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.730+0200	INFO	m1.raft	e606fe4619a718b6 has received 2 MsgVoteResp votes and 0 vote rejections	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.730+0200	INFO	m1.raft	e606fe4619a718b6 became leader at term 2	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.730+0200	INFO	m1.raft	raft.node: e606fe4619a718b6 elected leader e606fe4619a718b6 at term 2	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.730+0200	INFO	m2.raft	raft.node: f22d0a0f9f1b431b elected leader e606fe4619a718b6 at term 2	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.730+0200	INFO	m0.raft	raft.node: a11bd0ad5e07f6e9 elected leader e606fe4619a718b6 at term 2	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.735+0200	INFO	m1	setting up initial cluster version	{"member": "m1", "cluster-version": "3.5"}
2021/05/16 22:58:56 2021-05-16T22:58:56.738+0200	INFO	m1	set initial cluster version	{"member": "m1", "cluster-id": "3c5b384f03b0f420", "local-member-id": "e606fe4619a718b6", "cluster-version": "3.5"}
2021/05/16 22:58:56 2021-05-16T22:58:56.739+0200	INFO	m1	enabled capabilities for version	{"member": "m1", "cluster-version": "3.5"}
2021/05/16 22:58:56 2021-05-16T22:58:56.739+0200	INFO	m0	set initial cluster version	{"member": "m0", "cluster-id": "3c5b384f03b0f420", "local-member-id": "a11bd0ad5e07f6e9", "cluster-version": "3.5"}
2021/05/16 22:58:56 2021-05-16T22:58:56.739+0200	INFO	m2	set initial cluster version	{"member": "m2", "cluster-id": "3c5b384f03b0f420", "local-member-id": "f22d0a0f9f1b431b", "cluster-version": "3.5"}
2021/05/16 22:58:56 2021-05-16T22:58:56.739+0200	INFO	m0	published local member to cluster through raft	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "local-member-attributes": "{Name:m0 ClientURLs:[unix://127.0.0.1:210023101996]}", "request-path": "/0/members/a11bd0ad5e07f6e9/attributes", "cluster-id": "3c5b384f03b0f420", "publish-timeout": "5.2s"}
2021/05/16 22:58:56 2021-05-16T22:58:56.739+0200	INFO	m1	published local member to cluster through raft	{"member": "m1", "local-member-id": "e606fe4619a718b6", "local-member-attributes": "{Name:m1 ClientURLs:[unix://127.0.0.1:210043101996]}", "request-path": "/0/members/e606fe4619a718b6/attributes", "cluster-id": "3c5b384f03b0f420", "publish-timeout": "5.2s"}
2021/05/16 22:58:56 2021-05-16T22:58:56.739+0200	INFO	m2	published local member to cluster through raft	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "local-member-attributes": "{Name:m2 ClientURLs:[unix://127.0.0.1:210063101996]}", "request-path": "/0/members/f22d0a0f9f1b431b/attributes", "cluster-id": "3c5b384f03b0f420", "publish-timeout": "5.2s"}
2021/05/16 22:58:56  - m0 -> a11bd0ad5e07f6e9 (unix://localhost:m00)
2021/05/16 22:58:56  - m1 -> e606fe4619a718b6 (unix://localhost:m10)
2021/05/16 22:58:56  - m2 -> f22d0a0f9f1b431b (unix://localhost:m20)
2021/05/16 22:58:56 2021-05-16T22:58:56.782+0200	INFO	m1.raft	e606fe4619a718b6 switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227) learners=(9777699696455160727)	{"member": "m1"}
2021/05/16 22:58:56 2021-05-16T22:58:56.782+0200	INFO	m2.raft	f22d0a0f9f1b431b switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227) learners=(9777699696455160727)	{"member": "m2"}
2021/05/16 22:58:56 2021-05-16T22:58:56.782+0200	INFO	m1	added member	{"member": "m1", "cluster-id": "3c5b384f03b0f420", "local-member-id": "e606fe4619a718b6", "added-peer-id": "87b15e11c7a79397", "added-peer-peer-urls": ["http://localhost:32381"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.782+0200	INFO	m1	starting remote peer	{"member": "m1", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.782+0200	INFO	m1	started HTTP pipelining with remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.782+0200	INFO	m2	added member	{"member": "m2", "cluster-id": "3c5b384f03b0f420", "local-member-id": "f22d0a0f9f1b431b", "added-peer-id": "87b15e11c7a79397", "added-peer-peer-urls": ["http://localhost:32381"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.782+0200	INFO	m2	starting remote peer	{"member": "m2", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.782+0200	INFO	m2	started HTTP pipelining with remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.783+0200	INFO	m1	started remote peer	{"member": "m1", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.783+0200	INFO	m1	added remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "87b15e11c7a79397", "remote-peer-urls": ["http://localhost:32381"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.783+0200	INFO	m0.raft	a11bd0ad5e07f6e9 switched to configuration voters=(11609101907503085289 16575215055615236278 17450615193340691227) learners=(9777699696455160727)	{"member": "m0"}
2021/05/16 22:58:56 2021-05-16T22:58:56.784+0200	INFO	m1	started stream writer with remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.784+0200	INFO	m1	started stream writer with remote peer	{"member": "m1", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.784+0200	INFO	m1	started stream reader with remote peer	{"member": "m1", "stream-reader-type": "stream MsgApp v2", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.785+0200	INFO	m0	added member	{"member": "m0", "cluster-id": "3c5b384f03b0f420", "local-member-id": "a11bd0ad5e07f6e9", "added-peer-id": "87b15e11c7a79397", "added-peer-peer-urls": ["http://localhost:32381"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.785+0200	INFO	m0	starting remote peer	{"member": "m0", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.786+0200	INFO	m0	started HTTP pipelining with remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.786+0200	INFO	m2	started remote peer	{"member": "m2", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.786+0200	INFO	m2	added remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "87b15e11c7a79397", "remote-peer-urls": ["http://localhost:32381"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.786+0200	INFO	m2	started stream writer with remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.787+0200	INFO	m2	started stream writer with remote peer	{"member": "m2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.787+0200	INFO	m1	started stream reader with remote peer	{"member": "m1", "stream-reader-type": "stream Message", "local-member-id": "e606fe4619a718b6", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.787+0200	INFO	m0	started remote peer	{"member": "m0", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.787+0200	INFO	m2	started stream reader with remote peer	{"member": "m2", "stream-reader-type": "stream MsgApp v2", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.787+0200	INFO	m2	started stream reader with remote peer	{"member": "m2", "stream-reader-type": "stream Message", "local-member-id": "f22d0a0f9f1b431b", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.788+0200	INFO	m0	added remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "87b15e11c7a79397", "remote-peer-urls": ["http://localhost:32381"]}
2021/05/16 22:58:56 2021-05-16T22:58:56.788+0200	INFO	m0	started stream writer with remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.789+0200	INFO	m0	started stream writer with remote peer	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.789+0200	INFO	m0	started stream reader with remote peer	{"member": "m0", "stream-reader-type": "stream MsgApp v2", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.789+0200	INFO	m0	started stream reader with remote peer	{"member": "m0", "stream-reader-type": "stream Message", "local-member-id": "a11bd0ad5e07f6e9", "remote-peer-id": "87b15e11c7a79397"}
2021/05/16 22:58:56 2021-05-16T22:58:56.789+0200	INFO	m0	applied a configuration change through raft	{"member": "m0", "local-member-id": "a11bd0ad5e07f6e9", "raft-conf-change": "ConfChangeAddLearnerNode", "raft-conf-change-node-id": "87b15e11c7a79397"}

!!!  2021/05/16 22:58:56 2021-05-16T22:58:56.790+0200	INFO	m1.raft	e606fe4619a718b6 ignoring conf change {ConfChangeRemoveNode 9777699696455160727 [] 17791885354803724549} at config voters=(11609101907503085289 16575215055615236278 17450615193340691227) learners=(9777699696455160727): possible unapplied conf change at index 9 (applied to 8)	{"member": "m1"}

2021/05/16 22:58:56 2021-05-16T22:58:56.794+0200	WARN	m1	failed to reach the peer URL	{"member": "m1", "address": "http://localhost:32381/version", "remote-member-id": "87b15e11c7a79397", "error": "Get \"http://localhost:32381/version\": dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 22:58:56 2021-05-16T22:58:56.794+0200	WARN	m1	failed to get version	{"member": "m1", "remote-member-id": "87b15e11c7a79397", "error": "Get \"http://localhost:32381/version\": dial tcp [::1]:32381: connect: connection refused"}

2021/05/16 22:59:00 2021-05-16T22:59:00.797+0200	WARN	m1	failed to reach the peer URL	{"member": "m1", "address": "http://localhost:32381/version", "remote-member-id": "87b15e11c7a79397", "error": "Get \"http://localhost:32381/version\": dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 22:59:00 2021-05-16T22:59:00.798+0200	WARN	m1	failed to get version	{"member": "m1", "remote-member-id": "87b15e11c7a79397", "error": "Get \"http://localhost:32381/version\": dial tcp [::1]:32381: connect: connection refused"}


2021/05/16 22:59:01 2021-05-16T22:59:01.784+0200	WARN	m1	prober detected unhealthy status	{"member": "m1", "round-tripper-name": "ROUND_TRIPPER_RAFT_MESSAGE", "remote-peer-id": "87b15e11c7a79397", "rtt": "0s", "error": "dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 22:59:01 2021-05-16T22:59:01.787+0200	WARN	m1	prober detected unhealthy status	{"member": "m1", "round-tripper-name": "ROUND_TRIPPER_SNAPSHOT", "remote-peer-id": "87b15e11c7a79397", "rtt": "0s", "error": "dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 22:59:01 2021-05-16T22:59:01.788+0200	WARN	m2	prober detected unhealthy status	{"member": "m2", "round-tripper-name": "ROUND_TRIPPER_SNAPSHOT", "remote-peer-id": "87b15e11c7a79397", "rtt": "0s", "error": "dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 22:59:01 2021-05-16T22:59:01.788+0200	WARN	m2	prober detected unhealthy status	{"member": "m2", "round-tripper-name": "ROUND_TRIPPER_RAFT_MESSAGE", "remote-peer-id": "87b15e11c7a79397", "rtt": "0s", "error": "dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 22:59:01 2021-05-16T22:59:01.789+0200	WARN	m0	prober detected unhealthy status	{"member": "m0", "round-tripper-name": "ROUND_TRIPPER_SNAPSHOT", "remote-peer-id": "87b15e11c7a79397", "rtt": "0s", "error": "dial tcp [::1]:32381: connect: connection refused"}
2021/05/16 22:59:01 2021-05-16T22:59:01.790+0200	WARN	m0	prober detected unhealthy status	{"member": "m0", "round-tripper-name": "ROUND_TRIPPER_RAFT_MESSAGE", "remote-peer-id": "87b15e11c7a79397", "rtt": "0s", "error": "dial tcp [::1]:32381: connect: connection refused"}
@ptabor
Copy link
Contributor Author

ptabor commented May 18, 2021

Another case on arm64:
https://github.com/etcd-io/etcd/pull/12933/checks?check_run_id=2612954287

@ptabor ptabor mentioned this issue May 18, 2021
Merged
@ptabor ptabor added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label May 19, 2021
@ptabor
Copy link
Contributor Author

ptabor commented May 19, 2021

Marking as important as it's pretty likely to be server side problem and the RAFT log:

ignoring conf change {ConfChangeRemoveNode 4256952690501046763 [] 4425483133757641240} at config voters=(7519987121669182248 13966091899056808041 14250974771059047786) learners=(4256952690501046763): possible unapplied conf change at index 25 (applied to 24)	{"member": "m2"}

is pretty scary in this simple test scenario.
@tangcong @lilic @hexfusion @serathius -- pls keep it on the radar.

@lilic
Copy link
Contributor

lilic commented May 19, 2021

Thank you @ptabor, should we also mark it as important in regards to v3.5 going GA?

@ptabor ptabor added this to the etcd-v3.5 milestone May 19, 2021
@ptabor
Copy link
Contributor Author

ptabor commented May 19, 2021

@lilic - yes. I think we should. Added to markers that we already have.

@ptabor ptabor mentioned this issue May 19, 2021
Closed
4 tasks
@ptabor
Copy link
Contributor Author

ptabor commented Jun 2, 2021

https://github.com/etcd-io/etcd/pull/13009/checks?check_run_id=2621944462

@stale
Copy link

stale bot commented Aug 31, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Aug 31, 2021
@stale stale bot closed this as completed Sep 22, 2021
@chaochn47 chaochn47 mentioned this issue Apr 10, 2023
Closed
@chaochn47 chaochn47 mentioned this issue May 9, 2023
Merged
@chaochn47 chaochn47 mentioned this issue May 24, 2023
Closed
@chaochn47 chaochn47 mentioned this issue Jun 23, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
backport/v3.5 priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. stale type/flake
Milestone
etcd-v3.5
Development

No branches or pull requests
2 participants
@lilic @ptabor