etcdserver: make corruption check work under peer mTLS

[jingyih]

Currently, corruption check does not work when mTLS is enabled between peers. (See #10934)

With this PR, corruption check uses the existing peer-to-peer http communication framework to get peer hashKV.

Manually tested it works. Example server log:

09:18:14 etcd1 | {"level":"info","ts":"2020-02-13T09:18:14.234-0800","caller":"etcdserver/corrupt.go:130","msg":"enabled corruption checking","local-member-id":"d05e7521f6de6bab","interval":"10s"}
09:18:14 etcd3 | {"level":"info","ts":"2020-02-13T09:18:14.245-0800","caller":"etcdserver/corrupt.go:130","msg":"enabled corruption checking","local-member-id":"3fb0bb8f51408909","interval":"10s"}
09:18:14 etcd2 | {"level":"info","ts":"2020-02-13T09:18:14.257-0800","caller":"etcdserver/corrupt.go:130","msg":"enabled corruption checking","local-member-id":"f0c6bf0f0690ad50","interval":"10s"}
09:18:24 etcd2 | {"level":"info","ts":"2020-02-13T09:18:24.261-0800","caller":"etcdserver/corrupt.go:244","msg":"finished peer corruption check","number-of-peers-checked":2}
09:18:34 etcd2 | {"level":"info","ts":"2020-02-13T09:18:34.266-0800","caller":"etcdserver/corrupt.go:244","msg":"finished peer corruption check","number-of-peers-checked":2}
09:18:44 etcd2 | {"level":"info","ts":"2020-02-13T09:18:44.267-0800","caller":"etcdserver/corrupt.go:244","msg":"finished peer corruption check","number-of-peers-checked":2}
09:18:54 etcd2 | {"level":"info","ts":"2020-02-13T09:18:54.277-0800","caller":"etcdserver/corrupt.go:244","msg":"finished peer corruption check","number-of-peers-checked":2}

[codecov-io]

Codecov Report

Merging #11621 into master will decrease coverage by 0.09%.
The diff coverage is 50%.

@@            Coverage Diff            @@
##           master   #11621     +/-   ##
=========================================
- Coverage    66.2%    66.1%   -0.1%     
=========================================
  Files         403      401      -2     
  Lines       36674    36628     -46     
=========================================
- Hits        24280    24213     -67     
+ Misses      10924    10914     -10     
- Partials     1470     1501     +31

Impacted Files	Coverage Δ
etcdserver/api/etcdhttp/peer.go	87.67% <100%> (+0.34%)	⬆️
etcdserver/corrupt.go	41.11% <47.12%> (+0.35%)	⬆️
pkg/transport/timeout_conn.go	60% <0%> (-20%)	⬇️
auth/range_perm_cache.go	51.42% <0%> (-17.15%)	⬇️
proxy/httpproxy/director.go	52.43% <0%> (-15.86%)	⬇️
client/client.go	70.58% <0%> (-13.4%)	⬇️
clientv3/leasing/util.go	91.66% <0%> (-6.67%)	⬇️
raft/tracker/inflights.go	91.83% <0%> (-4.09%)	⬇️
proxy/grpcproxy/watcher.go	89.79% <0%> (-4.09%)	⬇️
etcdserver/api/snap/snapshotter.go	66.93% <0%> (-3.24%)	⬇️
... and 25 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ae98352...c031b27. Read the comment docs.

[YoyinZyc]

Clean implementation. Thanks Jingyi.

[YoyinZyc]

Can we rename the pps p eps? It is not easy to understand and remember. I think the naming way of getVersion() and getVersions() in cluster_util.go is much clear.

[jingyih]

Thanks! I tried to make it clearer. I did keep eps as short for endpoints.