Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

One node doesnt get balanced properly #644

Open
salarali opened this issue Aug 27, 2024 · 20 comments
Open

One node doesnt get balanced properly #644

salarali opened this issue Aug 27, 2024 · 20 comments

Comments

@salarali
Copy link

Bug Report

I have this issue sometimes that 1 node fails to be balanced properly. It happens randomly and I see no error in logs.

image

The connection count remains very low for a long time.
How should I go about debugging this? I checked the server and server is up and running.

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your version? (Required)

1.2.0
@salarali
Copy link
Author

It could be related that after some time, all session migrations stop all of a sudden.
image

@djshow832
Copy link
Collaborator

djshow832 commented Aug 28, 2024 •
edited
Loading

  1. Add from={the backend instance} in the PromQL in the Grafana panel to filter the migrations from this backend. E.g. sum(increase(tiproxy_balance_migrate_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", from="127.0.0.1:3080"}[1m])) by (reason)
  2. Check the CPU, memory, 4000 and 10080 ports, and network of this backend. I guess the reason comes from memory. Check if the memory usage is too high or if it increases too sharply.
  3. Search the log in TiProxy for update backend to see the status change of backends. E.g. [update backend] [namespace=default] [backend_addr=[10.90.2.153:4000](http://10.90.2.153:4000/)] [prev=healthy] [cur="down, err: http status 500: connect status port failed"] indicates that the backend 10.90.2.153:4000 is currently down because the 10080 status cannot be connected.

I hope that will help.

@salarali
Copy link
Author

So I see that the connections got moved away from it because of memory factor, but after that, no logs related to that backend at all. No update backend. The node is healthy as well. Memory use is also very low, no reason why it should not get more connections moved over to it.

@djshow832
Copy link
Collaborator

Can you query the metrics that TiProxy has read?

  1. Find the TiProxy owner by Grafana panel Server -> Owner, get the instance with label metric_reader.
  2. Query the metrics from the owner by HTTP API: curl 127.0.0.1:3080/api/backend/metrics. The result contains the current metrics of all backends. It may be hard to read. You can paste it here.

@salarali
Copy link
Author

Let me check those out.

I also added logs here:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L311

	if balanceCount > 0 {
		router.logger.Info("backends to balance", zap.String("from", busiestBackend.Addr()), zap.String("to", idlestBackend.Addr()), zap.String("reason", reason))
	}

And I am getting these logs there:

Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.159 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.169 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.179 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.189 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.199 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.209 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.219 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.229 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.239 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]

172.30.72.136 is dead and gone now. I guess that is the reason balancing just stops after a while.

@djshow832
Copy link
Collaborator

Please try to connect to the 4000 and 10080 ports. curl 127.0.0.1:10080/status.
BTW, you don't have to add logs, you can just turn the log level to debug.

@salarali
Copy link
Author

salarali commented Aug 28, 2024 •
edited
Loading

Curl commands from the tiproxy node:

curl 172.30.72.136:1008/status
curl: (7) Failed to connect to 172.30.72.136 port 1008 after 3131 ms: Couldn't connect to server

This is the dead node that is keeps trying to move connections away from. I use port 1008 instead of 10080.

Metrics from tiproxy:

{ "cpu": { "172.30.65.197:1008": { "Step1History": [ [ 1724815804.77, "3163.686875" ], [ 1724815809.775, "3167.0746875" ], [ 1724815814.799, "3170.431875" ], [ 1724815819.795, "3173.8028125" ], [ 1724815824.769, "3177.0978125" ], [ 1724815829.769, "3179.7228125" ], [ 1724815834.77, "3182.265" ], [ 1724815839.803, "3185.47" ], [ 1724815844.77, "3188.686875" ], [ 1724815849.77, "3191.1971875" ], [ 1724815854.773, "3192.9121875" ], [ 1724815859.768, "3193.3196875" ] ], "Step2History": [ [ 1724815804.77, "0.6347676070631327" ], [ 1724815809.775, "0.6768856143856362" ], [ 1724815814.799, "0.6682299960191553" ], [ 1724815819.795, "0.6747272818254312" ], [ 1724815824.769, "0.6624447125050408" ], [ 1724815829.769, "0.525" ], [ 1724815834.77, "0.5083358328333787" ], [ 1724815839.803, "0.6367971388833552" ], [ 1724815844.77, "0.6476494866116514" ], [ 1724815849.77, "0.502062500000011" ], [ 1724815854.773, "0.3427943234059855" ], [ 1724815859.768, "0.08158158158154152" ] ] }, "172.30.65.210:1008": { "Step1History": [ [ 1724815804.77, "3697.093125" ], [ 1724815809.77, "3697.4859375" ], [ 1724815814.771, "3697.8125" ], [ 1724815819.78, "3697.905" ], [ 1724815824.775, "3698.13875" ], [ 1724815829.77, "3699.020625" ], [ 1724815834.772, "3700.964375" ], [ 1724815839.772, "3703.625" ], [ 1724815844.77, "3706.604375" ], [ 1724815849.793, "3709.37625" ], [ 1724815854.787, "3713.2875" ], [ 1724815859.789, "3717.2003125" ] ], "Step2History": [ [ 1724815804.77, "0.1392499999999927" ], [ 1724815809.77, "0.07856250000004365" ], [ 1724815814.771, "0.06529944011195941" ], [ 1724815819.78, "0.018466759832341802" ], [ 1724815824.775, "0.0467967967967713" ], [ 1724815829.77, "0.17655155155155883" ], [ 1724815834.772, "0.3885945621751118" ], [ 1724815839.772, "0.5321249999999964" ], [ 1724815844.77, "0.5961134453781294" ], [ 1724815849.793, "0.5518365518614193" ], [ 1724815854.787, "0.7831898277933739" ], [ 1724815859.789, "0.782249600159976" ] ] }, "172.30.66.182:1008": { "Step1History": [ [ 1724815804.765, "872.634375" ], [ 1724815809.765, "872.6390625" ], [ 1724815814.766, "872.644375" ], [ 1724815819.765, "872.649375" ], [ 1724815824.765, "872.6546875" ], [ 1724815829.765, "872.6596875" ], [ 1724815834.766, "872.665" ], [ 1724815839.765, "872.67" ], [ 1724815844.765, "872.675625" ], [ 1724815849.765, "872.6803125" ], [ 1724815854.765, "872.685625" ], [ 1724815859.765, "872.6909375" ] ], "Step2History": [ [ 1724815804.765, "0.0011874999999918145" ], [ 1724815809.765, "0.0009375000000090949" ], [ 1724815814.766, "0.0010622875424805898" ], [ 1724815819.765, "0.001000200040007092" ], [ 1724815824.765, "0.0010625000000118235" ], [ 1724815829.765, "0.0009999999999990906" ], [ 1724815834.766, "0.0010622875424805898" ], [ 1724815839.765, "0.001000200040007092" ], [ 1724815844.765, "0.001125000000001819" ], [ 1724815849.765, "0.0009375000000090949" ], [ 1724815854.765, "0.001062499999989086" ], [ 1724815859.765, "0.0010625000000118235" ] ] }, "172.30.69.27:1008": { "Step1History": [ [ 1724815804.783, "2135.5996875" ], [ 1724815809.768, "2137.6321875" ], [ 1724815814.77, "2139.039375" ], [ 1724815819.768, "2140.269375" ], [ 1724815824.767, "2141.935625" ], [ 1724815829.767, "2143.73125" ], [ 1724815834.768, "2146.1565625" ], [ 1724815839.769, "2149.355625" ], [ 1724815844.767, "2152.294375" ], [ 1724815849.772, "2155.663125" ], [ 1724815854.782, "2159.8134375" ], [ 1724815859.781, "2164.080625" ] ], "Step2History": [ [ 1724815804.783, "0.7957718388512637" ], [ 1724815809.768, "0.4077231695084854" ], [ 1724815814.77, "0.2813249700119879" ], [ 1724815819.768, "0.24609843937575393" ], [ 1724815824.767, "0.3333166633327102" ], [ 1724815829.767, "0.35912499999994907" ], [ 1724815834.768, "0.4849655068986239" ], [ 1724815839.769, "0.6396845630874444" ], [ 1724815844.767, "0.587985194077591" ], [ 1724815849.772, "0.6730769230769412" ], [ 1724815854.782, "0.82840568862274" ], [ 1724815859.781, "0.8536082216443471" ] ] }, "172.30.77.112:1008": { "Step1History": [ [ 1724815804.783, "5181.1353125" ], [ 1724815809.787, "5185.5109375" ], [ 1724815814.786, "5189.7796875" ], [ 1724815819.773, "5193.738125" ], [ 1724815824.773, "5196.954375" ], [ 1724815829.779, "5198.663125" ], [ 1724815834.77, "5198.7990625" ], [ 1724815839.77, "5198.9175" ], [ 1724815844.77, "5199.1275" ], [ 1724815849.769, "5199.29375" ], [ 1724815854.774, "5199.314375" ], [ 1724815859.77, "5199.3678125" ] ], "Step2History": [ [ 1724815804.783, "0.8873850229952772" ], [ 1724815809.787, "0.8744254596324106" ], [ 1724815814.786, "0.8539207841568678" ], [ 1724815819.773, "0.7937512532583699" ], [ 1724815824.773, "0.64325000000008" ], [ 1724815829.779, "0.3413403915301202" ], [ 1724815834.77, "0.027236525746379865" ], [ 1724815839.77, "0.023687499999869032" ], [ 1724815844.77, "0.042000000000007275" ], [ 1724815849.769, "0.03325665133030972" ], [ 1724815854.774, "0.004120879120900927" ], [ 1724815859.77, "0.010696056845425407" ] ] }, "172.30.77.5:1008": { "Step1History": [ [ 1724815804.79, "4308.91625" ], [ 1724815809.773, "4313.6228125" ], [ 1724815814.772, "4318.255" ], [ 1724815819.772, "4322.8815625" ], [ 1724815824.772, "4327.45125" ], [ 1724815829.775, "4332.0190625" ], [ 1724815834.78, "4336.53625" ], [ 1724815839.776, "4341.2296875" ], [ 1724815844.771, "4345.9265625" ], [ 1724815849.773, "4350.518125" ], [ 1724815854.771, "4354.0234375" ], [ 1724815859.782, "4358.444375" ] ], "Step2History": [ [ 1724815804.79, "0.9370639824596517" ], [ 1724815809.773, "0.9445238811959791" ], [ 1724815814.772, "0.9266228245649786" ], [ 1724815819.772, "0.9253124999999273" ], [ 1724815824.772, "0.9139375000000655" ], [ 1724815829.775, "0.913014691185187" ], [ 1724815834.78, "0.9025349650350741" ], [ 1724815839.776, "0.9394390512410072" ], [ 1724815844.771, "0.9403153153152425" ], [ 1724815849.773, "0.9179453218712079" ], [ 1724815854.771, "0.7013430372149659" ], [ 1724815859.782, "0.8822465575733459" ] ] }, "172.30.81.38:1008": { "Step1History": [ [ 1724815799.933, "3745.8446875" ], [ 1724815804.806, "3750.320625" ], [ 1724815809.779, "3754.791875" ], [ 1724815814.813, "3759.3171875" ], [ 1724815819.783, "3763.7778125" ], [ 1724815824.812, "3768.3053125" ], [ 1724815829.78, "3772.7946875" ], [ 1724815834.795, "3777.3184375" ], [ 1724815839.889, "3781.8484375" ], [ 1724815844.778, "3786.3090625" ], [ 1724815850.034, "3790.824375" ], [ 1724815854.893, "3795.2865625" ], [ 1724815859.853, "3799.848125" ] ], "Step2History": [ [ 1724815799.933, "0.9056960764788144" ], [ 1724815804.806, "0.9185178534783239" ], [ 1724815809.779, "0.8991051679067071" ], [ 1724815814.813, "0.8989496424314516" ], [ 1724815819.783, "0.897510060362206" ], [ 1724815824.812, "0.9002783853649127" ], [ 1724815829.78, "0.9036584138485617" ], [ 1724815834.795, "0.9020438683948736" ], [ 1724815839.889, "0.8892815076560159" ], [ 1724815844.778, "0.9123798322765726" ], [ 1724815850.034, "0.8590777207001833" ], [ 1724815854.893, "0.9183345338546615" ], [ 1724815859.853, "0.9196698588709751" ] ] }, "172.30.83.151:1008": { "Step1History": [ [ 1724815804.765, "657.5271875" ], [ 1724815809.769, "658.0003125" ], [ 1724815814.765, "658.3203125" ], [ 1724815819.765, "658.87625" ], [ 1724815824.766, "659.3309375" ], [ 1724815829.766, "659.918125" ], [ 1724815834.765, "661.245625" ], [ 1724815839.765, "662.998125" ], [ 1724815844.77, "665.089375" ], [ 1724815849.877, "668.28125" ], [ 1724815854.766, "671.7965625" ], [ 1724815859.766, "673.4021875" ] ], "Step2History": [ [ 1724815804.765, "0.18472444488897327" ], [ 1724815809.769, "0.0945493605115871" ], [ 1724815814.765, "0.06405124099280424" ], [ 1724815819.765, "0.11118750000000546" ], [ 1724815824.766, "0.0909193161367681" ], [ 1724815829.766, "0.11743750000000545" ], [ 1724815834.765, "0.2655531106221217" ], [ 1724815839.765, "0.35049999999998815" ], [ 1724815844.77, "0.41783216783217964" ], [ 1724815849.877, "0.6249999999999964" ], [ 1724815854.766, "0.719024851707926" ], [ 1724815859.766, "0.3211249999999836" ] ] }, "172.30.84.204:1008": { "Step1History": [ [ 1724815804.771, "3422.655625" ], [ 1724815809.771, "3425.7834375" ], [ 1724815814.806, "3428.78375" ], [ 1724815819.793, "3430.7909375" ], [ 1724815824.77, "3432.81125" ], [ 1724815829.77, "3435.3896875" ], [ 1724815834.77, "3437.940625" ], [ 1724815839.807, "3441.090625" ], [ 1724815844.771, "3444.4559375" ], [ 1724815849.777, "3447.7721875" ], [ 1724815854.77, "3450.79" ], [ 1724815859.771, "3453.3978125" ] ], "Step2History": [ [ 1724815804.771, "0.46243749999994177" ], [ 1724815809.771, "0.6255625000000691" ], [ 1724815814.806, "0.5958912611717649" ], [ 1724815819.793, "0.4024839582915325" ], [ 1724815824.77, "0.40592977697413557" ], [ 1724815829.77, "0.5156874999999672" ], [ 1724815834.77, "0.5101875000000291" ], [ 1724815839.807, "0.625372245384085" ], [ 1724815844.771, "0.6779436946011427" ], [ 1724815849.777, "0.6624550539353394" ], [ 1724815854.77, "0.60440867214095" ], [ 1724815859.771, "0.5214582083583101" ] ] }, "172.30.87.250:1008": { "Step1History": [ [ 1724815804.776, "4343.0834375" ], [ 1724815809.774, "4345.7246875" ], [ 1724815814.789, "4347.8746875" ], [ 1724815819.776, "4350.93625" ], [ 1724815824.776, "4354.4703125" ], [ 1724815829.775, "4357.1771875" ], [ 1724815834.776, "4359.8353125" ], [ 1724815839.841, "4363.000625" ], [ 1724815844.775, "4366.26" ], [ 1724815849.782, "4368.994375" ], [ 1724815854.789, "4371.2934375" ], [ 1724815859.863, "4374.413125" ] ], "Step2History": [ [ 1724815804.776, "0.8648648648649887" ], [ 1724815809.774, "0.528461384553756" ], [ 1724815814.789, "0.4287138584246533" ], [ 1724815819.776, "0.6139086625225659" ], [ 1724815824.776, "0.7068124999999782" ], [ 1724815829.775, "0.5414832966594847" ], [ 1724815834.776, "0.5315186962607332" ], [ 1724815839.841, "0.624938302072921" ], [ 1724815844.775, "0.6605948520471312" ], [ 1724815849.782, "0.546110445376473" ], [ 1724815854.789, "0.45916966247258206" ], [ 1724815859.863, "0.6148378990933385" ] ] }, "172.30.88.130:1008": { "Step1History": [ [ 1724815804.809, "122952.1578125" ], [ 1724815809.808, "122952.865" ], [ 1724815814.797, "122953.4575" ], [ 1724815819.796, "122954.059375" ], [ 1724815824.808, "122954.6484375" ], [ 1724815829.792, "122955.26875" ], [ 1724815834.793, "122955.8365625" ], [ 1724815839.792, "122956.3340625" ], [ 1724815844.791, "122956.715625" ], [ 1724815849.792, "122957.10375" ], [ 1724815854.804, "122957.496875" ], [ 1724815859.79, "122957.891875" ] ], "Step2History": [ [ 1724815804.809, "0.2142197484001151" ], [ 1724815809.808, "0.14146579316084407" ], [ 1724815814.797, "0.11876127480433671" ], [ 1724815819.796, "0.12039907981456593" ], [ 1724815824.808, "0.11753042697584007" ], [ 1724815829.792, "0.1244607744789146" ], [ 1724815834.793, "0.1135397920411261" ], [ 1724815839.792, "0.09951990398033041" ], [ 1724815844.791, "0.07632776555287775" ], [ 1724815849.792, "0.07760947810391355" ], [ 1724815854.804, "0.07843675179615489" ], [ 1724815859.79, "0.07922182109989462" ] ] }, "172.30.88.59:1008": { "Step1History": [ [ 1724815804.811, "121455.769375" ], [ 1724815809.793, "121457.0215625" ], [ 1724815814.798, "121457.695625" ], [ 1724815819.791, "121458.3453125" ], [ 1724815824.789, "121458.909375" ], [ 1724815829.797, "121459.6084375" ], [ 1724815834.813, "121460.4428125" ], [ 1724815839.798, "121461.2521875" ], [ 1724815844.792, "121461.8559375" ], [ 1724815849.795, "121462.73125" ], [ 1724815854.799, "121463.2984375" ], [ 1724815859.789, "121463.693125" ] ], "Step2History": [ [ 1724815804.811, "0.15282755874188494" ], [ 1724815809.793, "0.25134233239557635" ], [ 1724815814.798, "0.13467782217677549" ], [ 1724815819.791, "0.13011966753478152" ], [ 1724815824.789, "0.11285764305896981" ], [ 1724815829.797, "0.13958915734894017" ], [ 1724815834.813, "0.16634270334754164" ], [ 1724815839.798, "0.16236208625819248" ], [ 1724815844.792, "0.12089507409077158" ], [ 1724815849.795, "0.17495752548331303" ], [ 1724815854.799, "0.11334682254138483" ], [ 1724815859.789, "0.07909569138498185" ] ] }, "172.30.89.188:1008": { "Step1History": [ [ 1724815804.766, "693.51625" ], [ 1724815809.766, "693.55" ], [ 1724815814.766, "693.561875" ], [ 1724815819.765, "693.57375" ], [ 1724815824.765, "693.636875" ], [ 1724815829.775, "694.445625" ], [ 1724815834.767, "695.4015625" ], [ 1724815839.769, "696.7715625" ], [ 1724815844.773, "698.7621875" ], [ 1724815849.767, "702.030625" ], [ 1724815854.769, "704.406875" ], [ 1724815859.803, "708.0746875" ] ], "Step2History": [ [ 1724815804.766, "0.04006249999999909" ], [ 1724815809.766, "0.006749999999988177" ], [ 1724815814.766, "0.0023750000000063664" ], [ 1724815819.765, "0.002375475095025372" ], [ 1724815824.765, "0.012625000000002728" ], [ 1724815829.775, "0.1614271457085665" ], [ 1724815834.767, "0.1914938902243599" ], [ 1724815839.769, "0.2738904438224719" ], [ 1724815844.773, "0.3978067545963275" ], [ 1724815849.767, "0.65447286744093" ], [ 1724815854.769, "0.47505997600960165" ], [ 1724815859.803, "0.7286079658323338" ] ] } }, "health_pd": { "172.30.65.197:1008": { "Step1History": [ [ 1724815744.785, "2" ], [ 1724815749.776, "2" ], [ 1724815754.768, "2" ], [ 1724815759.771, "2" ], [ 1724815764.769, "2" ], [ 1724815769.801, "2" ], [ 1724815774.809, "2" ], [ 1724815779.775, "2" ], [ 1724815784.775, "2" ], [ 1724815789.821, "2" ], [ 1724815794.82, "2" ], [ 1724815799.843, "2" ], [ 1724815804.77, "2" ], [ 1724815809.775, "2" ], [ 1724815814.799, "2" ], [ 1724815819.795, "2" ], [ 1724815824.769, "2" ], [ 1724815829.769, "2" ], [ 1724815834.77, "2" ], [ 1724815839.803, "2" ], [ 1724815844.77, "2" ], [ 1724815849.77, "2" ], [ 1724815854.773, "2" ], [ 1724815859.768, "2" ] ], "Step2History": [ [ 1724815744.785, "1" ], [ 1724815749.776, "1" ], [ 1724815754.768, "1" ], [ 1724815759.771, "1" ], [ 1724815764.769, "1" ], [ 1724815769.801, "1" ], [ 1724815774.809, "1" ], [ 1724815779.775, "1" ], [ 1724815784.775, "1" ], [ 1724815789.821, "1" ], [ 1724815794.82, "1" ], [ 1724815799.843, "0" ], [ 1724815804.77, "0" ], [ 1724815809.775, "0" ], [ 1724815814.799, "0" ], [ 1724815819.795, "0" ], [ 1724815824.769, "0" ], [ 1724815829.769, "0" ], [ 1724815834.77, "0" ], [ 1724815839.803, "0" ], [ 1724815844.77, "0" ], [ 1724815849.77, "0" ], [ 1724815854.773, "0" ], [ 1724815859.768, "0" ] ] }, "172.30.65.210:1008": { "Step1History": [ [ 1724815744.771, "1" ], [ 1724815749.771, "1" ], [ 1724815754.769, "1" ], [ 1724815759.78, "1" ], [ 1724815764.77, "1" ], [ 1724815769.77, "1" ], [ 1724815774.771, "1" ], [ 1724815779.771, "1" ], [ 1724815784.77, "1" ], [ 1724815789.77, "1" ], [ 1724815794.77, "1" ], [ 1724815799.77, "1" ], [ 1724815804.77, "1" ], [ 1724815809.77, "1" ], [ 1724815814.771, "1" ], [ 1724815819.78, "1" ], [ 1724815824.775, "1" ], [ 1724815829.77, "1" ], [ 1724815834.772, "1" ], [ 1724815839.772, "1" ], [ 1724815844.77, "1" ], [ 1724815849.793, "1" ], [ 1724815854.787, "1" ], [ 1724815859.789, "1" ] ], "Step2History": [ [ 1724815744.771, "0" ], [ 1724815749.771, "0" ], [ 1724815754.769, "0" ], [ 1724815759.78, "0" ], [ 1724815764.77, "0" ], [ 1724815769.77, "0" ], [ 1724815774.771, "0" ], [ 1724815779.771, "0" ], [ 1724815784.77, "0" ], [ 1724815789.77, "0" ], [ 1724815794.77, "0" ], [ 1724815799.77, "0" ], [ 1724815804.77, "0" ], [ 1724815809.77, "0" ], [ 1724815814.771, "0" ], [ 1724815819.78, "0" ], [ 1724815824.775, "0" ], [ 1724815829.77, "0" ], [ 1724815834.772, "0" ], [ 1724815839.772, "0" ], [ 1724815844.77, "0" ], [ 1724815849.793, "0" ], [ 1724815854.787, "0" ], [ 1724815859.789, "0" ] ] }, "172.30.66.182:1008": { "Step1History": [ [ 1724815744.765, "0" ], [ 1724815749.765, "0" ], [ 1724815754.765, "0" ], [ 1724815759.765, "0" ], [ 1724815764.765, "0" ], [ 1724815769.765, "0" ], [ 1724815774.765, "0" ], [ 1724815779.765, "0" ], [ 1724815784.766, "0" ], [ 1724815789.765, "0" ], [ 1724815794.765, "0" ], [ 1724815799.765, "0" ], [ 1724815804.765, "0" ], [ 1724815809.765, "0" ], [ 1724815814.766, "0" ], [ 1724815819.765, "0" ], [ 1724815824.765, "0" ], [ 1724815829.765, "0" ], [ 1724815834.766, "0" ], [ 1724815839.765, "0" ], [ 1724815844.765, "0" ], [ 1724815849.765, "0" ], [ 1724815854.765, "0" ], [ 1724815859.765, "0" ] ], "Step2History": [ [ 1724815744.765, "0" ], [ 1724815749.765, "0" ], [ 1724815754.765, "0" ], [ 1724815759.765, "0" ], [ 1724815764.765, "0" ], [ 1724815769.765, "0" ], [ 1724815774.765, "0" ], [ 1724815779.765, "0" ], [ 1724815784.766, "0" ], [ 1724815789.765, "0" ], [ 1724815794.765, "0" ], [ 1724815799.765, "0" ], [ 1724815804.765, "0" ], [ 1724815809.765, "0" ], [ 1724815814.766, "0" ], [ 1724815819.765, "0" ], [ 1724815824.765, "0" ], [ 1724815829.765, "0" ], [ 1724815834.766, "0" ], [ 1724815839.765, "0" ], [ 1724815844.765, "0" ], [ 1724815849.765, "0" ], [ 1724815854.765, "0" ], [ 1724815859.765, "0" ] ] }, "172.30.69.27:1008": { "Step1History": [ [ 1724815744.908, "0" ], [ 1724815749.834, "0" ], [ 1724815754.769, "0" ], [ 1724815759.77, "0" ], [ 1724815764.769, "0" ], [ 1724815769.795, "0" ], [ 1724815774.769, "0" ], [ 1724815779.779, "0" ], [ 1724815784.769, "0" ], [ 1724815789.778, "0" ], [ 1724815794.782, "0" ], [ 1724815799.769, "0" ], [ 1724815804.783, "0" ], [ 1724815809.768, "0" ], [ 1724815814.77, "0" ], [ 1724815819.768, "0" ], [ 1724815824.767, "0" ], [ 1724815829.767, "0" ], [ 1724815834.768, "0" ], [ 1724815839.769, "0" ], [ 1724815844.767, "0" ], [ 1724815849.772, "0" ], [ 1724815854.782, "0" ], [ 1724815859.781, "0" ] ], "Step2History": [ [ 1724815744.908, "0" ], [ 1724815749.834, "0" ], [ 1724815754.769, "0" ], [ 1724815759.77, "0" ], [ 1724815764.769, "0" ], [ 1724815769.795, "0" ], [ 1724815774.769, "0" ], [ 1724815779.779, "0" ], [ 1724815784.769, "0" ], [ 1724815789.778, "0" ], [ 1724815794.782, "0" ], [ 1724815799.769, "0" ], [ 1724815804.783, "0" ], [ 1724815809.768, "0" ], [ 1724815814.77, "0" ], [ 1724815819.768, "0" ], [ 1724815824.767, "0" ], [ 1724815829.767, "0" ], [ 1724815834.768, "0" ], [ 1724815839.769, "0" ], [ 1724815844.767, "0" ], [ 1724815849.772, "0" ], [ 1724815854.782, "0" ], [ 1724815859.781, "0" ] ] }, "172.30.77.112:1008": { "Step1History": [ [ 1724815744.795, "8" ], [ 1724815749.78, "8" ], [ 1724815754.772, "8" ], [ 1724815759.791, "8" ], [ 1724815764.788, "8" ], [ 1724815769.796, "8" ], [ 1724815774.779, "8" ], [ 1724815779.798, "8" ], [ 1724815784.776, "8" ], [ 1724815789.775, "8" ], [ 1724815794.779, "8" ], [ 1724815799.782, "8" ], [ 1724815804.783, "8" ], [ 1724815809.787, "8" ], [ 1724815814.786, "8" ], [ 1724815819.773, "8" ], [ 1724815824.773, "8" ], [ 1724815829.779, "8" ], [ 1724815834.77, "8" ], [ 1724815839.77, "8" ], [ 1724815844.77, "8" ], [ 1724815849.769, "8" ], [ 1724815854.774, "8" ], [ 1724815859.77, "8" ] ], "Step2History": [ [ 1724815744.795, "0" ], [ 1724815749.78, "0" ], [ 1724815754.772, "0" ], [ 1724815759.791, "0" ], [ 1724815764.788, "0" ], [ 1724815769.796, "0" ], [ 1724815774.779, "0" ], [ 1724815779.798, "0" ], [ 1724815784.776, "0" ], [ 1724815789.775, "0" ], [ 1724815794.779, "0" ], [ 1724815799.782, "0" ], [ 1724815804.783, "0" ], [ 1724815809.787, "0" ], [ 1724815814.786, "0" ], [ 1724815819.773, "0" ], [ 1724815824.773, "0" ], [ 1724815829.779, "0" ], [ 1724815834.77, "0" ], [ 1724815839.77, "0" ], [ 1724815844.77, "0" ], [ 1724815849.769, "0" ], [ 1724815854.774, "0" ], [ 1724815859.77, "0" ] ] }, "172.30.77.5:1008": { "Step1History": [ [ 1724815744.777, "3" ], [ 1724815749.772, "3" ], [ 1724815754.771, "3" ], [ 1724815759.78, "3" ], [ 1724815764.778, "3" ], [ 1724815769.774, "3" ], [ 1724815774.775, "3" ], [ 1724815779.783, "3" ], [ 1724815784.79, "3" ], [ 1724815789.777, "3" ], [ 1724815794.771, "3" ], [ 1724815799.773, "3" ], [ 1724815804.79, "3" ], [ 1724815809.773, "3" ], [ 1724815814.772, "3" ], [ 1724815819.772, "3" ], [ 1724815824.772, "3" ], [ 1724815829.775, "3" ], [ 1724815834.78, "3" ], [ 1724815839.776, "3" ], [ 1724815844.771, "3" ], [ 1724815849.773, "3" ], [ 1724815854.771, "3" ], [ 1724815859.782, "3" ] ], "Step2History": [ [ 1724815744.777, "0" ], [ 1724815749.772, "0" ], [ 1724815754.771, "0" ], [ 1724815759.78, "0" ], [ 1724815764.778, "0" ], [ 1724815769.774, "0" ], [ 1724815774.775, "0" ], [ 1724815779.783, "0" ], [ 1724815784.79, "0" ], [ 1724815789.777, "0" ], [ 1724815794.771, "0" ], [ 1724815799.773, "0" ], [ 1724815804.79, "0" ], [ 1724815809.773, "0" ], [ 1724815814.772, "0" ], [ 1724815819.772, "0" ], [ 1724815824.772, "0" ], [ 1724815829.775, "0" ], [ 1724815834.78, "0" ], [ 1724815839.776, "0" ], [ 1724815844.771, "0" ], [ 1724815849.773, "0" ], [ 1724815854.771, "0" ], [ 1724815859.782, "0" ] ] }, "172.30.81.38:1008": { "Step1History": [ [ 1724815739.92, "8" ], [ 1724815744.841, "8" ], [ 1724815749.819, "8" ], [ 1724815754.851, "8" ], [ 1724815759.849, "8" ], [ 1724815764.858, "8" ], [ 1724815769.814, "8" ], [ 1724815774.814, "8" ], [ 1724815779.781, "8" ], [ 1724815784.794, "8" ], [ 1724815789.789, "8" ], [ 1724815794.912, "8" ], [ 1724815799.933, "8" ], [ 1724815804.806, "8" ], [ 1724815809.779, "8" ], [ 1724815814.813, "8" ], [ 1724815819.783, "8" ], [ 1724815824.812, "8" ], [ 1724815829.78, "8" ], [ 1724815834.795, "8" ], [ 1724815839.889, "8" ], [ 1724815844.778, "8" ], [ 1724815850.034, "8" ], [ 1724815854.893, "8" ], [ 1724815859.853, "8" ] ], "Step2History": [ [ 1724815739.92, "0" ], [ 1724815744.841, "0" ], [ 1724815749.819, "0" ], [ 1724815754.851, "0" ], [ 1724815759.849, "0" ], [ 1724815764.858, "0" ], [ 1724815769.814, "0" ], [ 1724815774.814, "0" ], [ 1724815779.781, "0" ], [ 1724815784.794, "0" ], [ 1724815789.789, "0" ], [ 1724815794.912, "0" ], [ 1724815799.933, "0" ], [ 1724815804.806, "0" ], [ 1724815809.779, "0" ], [ 1724815814.813, "0" ], [ 1724815819.783, "0" ], [ 1724815824.812, "0" ], [ 1724815829.78, "0" ], [ 1724815834.795, "0" ], [ 1724815839.889, "0" ], [ 1724815844.778, "0" ], [ 1724815850.034, "0" ], [ 1724815854.893, "0" ], [ 1724815859.853, "0" ] ] }, "172.30.83.151:1008": { "Step1History": [ [ 1724815744.766, "0" ], [ 1724815749.765, "0" ], [ 1724815754.767, "0" ], [ 1724815759.77, "0" ], [ 1724815764.765, "0" ], [ 1724815769.77, "0" ], [ 1724815774.765, "0" ], [ 1724815779.766, "0" ], [ 1724815784.771, "0" ], [ 1724815789.768, "0" ], [ 1724815794.765, "0" ], [ 1724815799.766, "0" ], [ 1724815804.765, "0" ], [ 1724815809.769, "0" ], [ 1724815814.765, "0" ], [ 1724815819.765, "0" ], [ 1724815824.766, "0" ], [ 1724815829.766, "0" ], [ 1724815834.765, "0" ], [ 1724815839.765, "0" ], [ 1724815844.77, "0" ], [ 1724815849.877, "0" ], [ 1724815854.766, "0" ], [ 1724815859.766, "0" ] ], "Step2History": [ [ 1724815744.766, "0" ], [ 1724815749.765, "0" ], [ 1724815754.767, "0" ], [ 1724815759.77, "0" ], [ 1724815764.765, "0" ], [ 1724815769.77, "0" ], [ 1724815774.765, "0" ], [ 1724815779.766, "0" ], [ 1724815784.771, "0" ], [ 1724815789.768, "0" ], [ 1724815794.765, "0" ], [ 1724815799.766, "0" ], [ 1724815804.765, "0" ], [ 1724815809.769, "0" ], [ 1724815814.765, "0" ], [ 1724815819.765, "0" ], [ 1724815824.766, "0" ], [ 1724815829.766, "0" ], [ 1724815834.765, "0" ], [ 1724815839.765, "0" ], [ 1724815844.77, "0" ], [ 1724815849.877, "0" ], [ 1724815854.766, "0" ], [ 1724815859.766, "0" ] ] }, "172.30.84.204:1008": { "Step1History": [ [ 1724815744.771, "7" ], [ 1724815749.77, "7" ], [ 1724815754.77, "7" ], [ 1724815759.771, "7" ], [ 1724815764.837, "7" ], [ 1724815769.817, "7" ], [ 1724815774.772, "7" ], [ 1724815779.773, "7" ], [ 1724815784.771, "7" ], [ 1724815789.776, "7" ], [ 1724815794.771, "7" ], [ 1724815799.771, "7" ], [ 1724815804.771, "7" ], [ 1724815809.771, "7" ], [ 1724815814.806, "7" ], [ 1724815819.793, "7" ], [ 1724815824.77, "7" ], [ 1724815829.77, "7" ], [ 1724815834.77, "7" ], [ 1724815839.807, "7" ], [ 1724815844.771, "7" ], [ 1724815849.777, "7" ], [ 1724815854.77, "7" ], [ 1724815859.771, "7" ] ], "Step2History": [ [ 1724815744.771, "1" ], [ 1724815749.77, "1" ], [ 1724815754.77, "1" ], [ 1724815759.771, "1" ], [ 1724815764.837, "1" ], [ 1724815769.817, "1" ], [ 1724815774.772, "1" ], [ 1724815779.773, "1" ], [ 1724815784.771, "1" ], [ 1724815789.776, "1" ], [ 1724815794.771, "1" ], [ 1724815799.771, "0" ], [ 1724815804.771, "0" ], [ 1724815809.771, "0" ], [ 1724815814.806, "0" ], [ 1724815819.793, "0" ], [ 1724815824.77, "0" ], [ 1724815829.77, "0" ], [ 1724815834.77, "0" ], [ 1724815839.807, "0" ], [ 1724815844.771, "0" ], [ 1724815849.777, "0" ], [ 1724815854.77, "0" ], [ 1724815859.771, "0" ] ] }, "172.30.87.250:1008": { "Step1History": [ [ 1724815744.811, "2" ], [ 1724815749.776, "2" ], [ 1724815754.775, "2" ], [ 1724815759.783, "2" ], [ 1724815764.797, "2" ], [ 1724815769.804, "2" ], [ 1724815774.782, "2" ], [ 1724815779.777, "2" ], [ 1724815784.782, "2" ], [ 1724815789.778, "2" ], [ 1724815794.785, "2" ], [ 1724815799.781, "2" ], [ 1724815804.776, "2" ], [ 1724815809.774, "2" ], [ 1724815814.789, "2" ], [ 1724815819.776, "2" ], [ 1724815824.776, "2" ], [ 1724815829.775, "2" ], [ 1724815834.776, "2" ], [ 1724815839.841, "2" ], [ 1724815844.775, "2" ], [ 1724815849.782, "2" ], [ 1724815854.789, "2" ], [ 1724815859.863, "2" ] ], "Step2History": [ [ 1724815744.811, "1" ], [ 1724815749.776, "1" ], [ 1724815754.775, "1" ], [ 1724815759.783, "1" ], [ 1724815764.797, "1" ], [ 1724815769.804, "1" ], [ 1724815774.782, "1" ], [ 1724815779.777, "1" ], [ 1724815784.782, "1" ], [ 1724815789.778, "1" ], [ 1724815794.785, "1" ], [ 1724815799.781, "0" ], [ 1724815804.776, "0" ], [ 1724815809.774, "0" ], [ 1724815814.789, "0" ], [ 1724815819.776, "0" ], [ 1724815824.776, "0" ], [ 1724815829.775, "0" ], [ 1724815834.776, "0" ], [ 1724815839.841, "0" ], [ 1724815844.775, "0" ], [ 1724815849.782, "0" ], [ 1724815854.789, "0" ], [ 1724815859.863, "0" ] ] }, "172.30.88.130:1008": { "Step1History": [ [ 1724815744.792, "12049" ], [ 1724815749.792, "12049" ], [ 1724815754.816, "12049" ], [ 1724815759.808, "12049" ], [ 1724815764.794, "12049" ], [ 1724815769.806, "12049" ], [ 1724815774.795, "12049" ], [ 1724815779.796, "12049" ], [ 1724815784.791, "12049" ], [ 1724815789.797, "12049" ], [ 1724815794.794, "12049" ], [ 1724815799.801, "12049" ], [ 1724815804.809, "12049" ], [ 1724815809.808, "12049" ], [ 1724815814.797, "12049" ], [ 1724815819.796, "12049" ], [ 1724815824.808, "12049" ], [ 1724815829.792, "12049" ], [ 1724815834.793, "12049" ], [ 1724815839.792, "12049" ], [ 1724815844.791, "12049" ], [ 1724815849.792, "12049" ], [ 1724815854.804, "12049" ], [ 1724815859.79, "12049" ] ], "Step2History": [ [ 1724815744.792, "0" ], [ 1724815749.792, "0" ], [ 1724815754.816, "0" ], [ 1724815759.808, "0" ], [ 1724815764.794, "0" ], [ 1724815769.806, "0" ], [ 1724815774.795, "0" ], [ 1724815779.796, "0" ], [ 1724815784.791, "0" ], [ 1724815789.797, "0" ], [ 1724815794.794, "0" ], [ 1724815799.801, "0" ], [ 1724815804.809, "0" ], [ 1724815809.808, "0" ], [ 1724815814.797, "0" ], [ 1724815819.796, "0" ], [ 1724815824.808, "0" ], [ 1724815829.792, "0" ], [ 1724815834.793, "0" ], [ 1724815839.792, "0" ], [ 1724815844.791, "0" ], [ 1724815849.792, "0" ], [ 1724815854.804, "0" ], [ 1724815859.79, "0" ] ] }, "172.30.88.59:1008": { "Step1History": [ [ 1724815744.785, "10813" ], [ 1724815749.785, "10813" ], [ 1724815754.785, "10813" ], [ 1724815759.795, "10813" ], [ 1724815764.791, "10813" ], [ 1724815769.804, "10813" ], [ 1724815774.789, "10813" ], [ 1724815779.793, "10813" ], [ 1724815784.791, "10813" ], [ 1724815789.79, "10813" ], [ 1724815794.789, "10813" ], [ 1724815799.789, "10813" ], [ 1724815804.811, "10813" ], [ 1724815809.793, "10813" ], [ 1724815814.798, "10813" ], [ 1724815819.791, "10813" ], [ 1724815824.789, "10813" ], [ 1724815829.797, "10813" ], [ 1724815834.813, "10813" ], [ 1724815839.798, "10813" ], [ 1724815844.792, "10813" ], [ 1724815849.795, "10813" ], [ 1724815854.799, "10813" ], [ 1724815859.789, "10813" ] ], "Step2History": [ [ 1724815744.785, "0" ], [ 1724815749.785, "0" ], [ 1724815754.785, "0" ], [ 1724815759.795, "0" ], [ 1724815764.791, "0" ], [ 1724815769.804, "0" ], [ 1724815774.789, "0" ], [ 1724815779.793, "0" ], [ 1724815784.791, "0" ], [ 1724815789.79, "0" ], [ 1724815794.789, "0" ], [ 1724815799.789, "0" ], [ 1724815804.811, "0" ], [ 1724815809.793, "0" ], [ 1724815814.798, "0" ], [ 1724815819.791, "0" ], [ 1724815824.789, "0" ], [ 1724815829.797, "0" ], [ 1724815834.813, "0" ], [ 1724815839.798, "0" ], [ 1724815844.792, "0" ], [ 1724815849.795, "0" ], [ 1724815854.799, "0" ], [ 1724815859.789, "0" ] ] }, "172.30.89.188:1008": { "Step1History": [ [ 1724815744.77, "0" ], [ 1724815749.767, "0" ], [ 1724815754.766, "0" ], [ 1724815759.766, "0" ], [ 1724815764.767, "0" ], [ 1724815769.768, "0" ], [ 1724815774.765, "0" ], [ 1724815779.767, "0" ], [ 1724815784.77, "0" ], [ 1724815789.767, "0" ], [ 1724815794.766, "0" ], [ 1724815799.766, "0" ], [ 1724815804.766, "0" ], [ 1724815809.766, "0" ], [ 1724815814.766, "0" ], [ 1724815819.765, "0" ], [ 1724815824.765, "0" ], [ 1724815829.775, "0" ], [ 1724815834.767, "0" ], [ 1724815839.769, "0" ], [ 1724815844.773, "0" ], [ 1724815849.767, "0" ], [ 1724815854.769, "0" ], [ 1724815859.803, "0" ] ], "Step2History": [ [ 1724815744.77, "0" ], [ 1724815749.767, "0" ], [ 1724815754.766, "0" ], [ 1724815759.766, "0" ], [ 1724815764.767, "0" ], [ 1724815769.768, "0" ], [ 1724815774.765, "0" ], [ 1724815779.767, "0" ], [ 1724815784.77, "0" ], [ 1724815789.767, "0" ], [ 1724815794.766, "0" ], [ 1724815799.766, "0" ], [ 1724815804.766, "0" ], [ 1724815809.766, "0" ], [ 1724815814.766, "0" ], [ 1724815819.765, "0" ], [ 1724815824.765, "0" ], [ 1724815829.775, "0" ], [ 1724815834.767, "0" ], [ 1724815839.769, "0" ], [ 1724815844.773, "0" ], [ 1724815849.767, "0" ], [ 1724815854.769, "0" ], [ 1724815859.803, "0" ] ] } }, "health_tikv": { "172.30.65.197:1008": { "Step1History": [ [ 1724815744.785, "6054" ], [ 1724815749.776, "6054" ], [ 1724815754.768, "6055" ], [ 1724815759.771, "6055" ], [ 1724815764.769, "6055" ], [ 1724815769.801, "6055" ], [ 1724815774.809, "6055" ], [ 1724815779.775, "6055" ], [ 1724815784.775, "6055" ], [ 1724815789.821, "6055" ], [ 1724815794.82, "6055" ], [ 1724815799.843, "6055" ], [ 1724815804.77, "6055" ], [ 1724815809.775, "6056" ], [ 1724815814.799, "6056" ], [ 1724815819.795, "6056" ], [ 1724815824.769, "6056" ], [ 1724815829.769, "6056" ], [ 1724815834.77, "6056" ], [ 1724815839.803, "6056" ], [ 1724815844.77, "6056" ], [ 1724815849.77, "6057" ], [ 1724815854.773, "6057" ], [ 1724815859.768, "6057" ] ], "Step2History": [ [ 1724815744.785, "1" ], [ 1724815749.776, "1" ], [ 1724815754.768, "2" ], [ 1724815759.771, "2" ], [ 1724815764.769, "2" ], [ 1724815769.801, "2" ], [ 1724815774.809, "2" ], [ 1724815779.775, "2" ], [ 1724815784.775, "2" ], [ 1724815789.821, "2" ], [ 1724815794.82, "2" ], [ 1724815799.843, "2" ], [ 1724815804.77, "2" ], [ 1724815809.775, "3" ], [ 1724815814.799, "3" ], [ 1724815819.795, "3" ], [ 1724815824.769, "3" ], [ 1724815829.769, "3" ], [ 1724815834.77, "3" ], [ 1724815839.803, "3" ], [ 1724815844.77, "3" ], [ 1724815849.77, "4" ], [ 1724815854.773, "4" ], [ 1724815859.768, "3" ] ] }, "172.30.65.210:1008": { "Step1History": [ [ 1724815744.771, "5965" ], [ 1724815749.771, "5965" ], [ 1724815754.769, "5965" ], [ 1724815759.78, "5966" ], [ 1724815764.77, "5966" ], [ 1724815769.77, "5966" ], [ 1724815774.771, "5967" ], [ 1724815779.771, "5967" ], [ 1724815784.77, "5967" ], [ 1724815789.77, "5967" ], [ 1724815794.77, "5967" ], [ 1724815799.77, "5967" ], [ 1724815804.77, "5967" ], [ 1724815809.77, "5968" ], [ 1724815814.771, "5968" ], [ 1724815819.78, "5968" ], [ 1724815824.775, "5968" ], [ 1724815829.77, "5968" ], [ 1724815834.772, "5968" ], [ 1724815839.772, "5968" ], [ 1724815844.77, "5968" ], [ 1724815849.793, "5970" ], [ 1724815854.787, "5970" ], [ 1724815859.789, "5970" ] ], "Step2History": [ [ 1724815744.771, "1" ], [ 1724815749.771, "1" ], [ 1724815754.769, "1" ], [ 1724815759.78, "2" ], [ 1724815764.77, "2" ], [ 1724815769.77, "2" ], [ 1724815774.771, "3" ], [ 1724815779.771, "3" ], [ 1724815784.77, "3" ], [ 1724815789.77, "3" ], [ 1724815794.77, "3" ], [ 1724815799.77, "3" ], [ 1724815804.77, "3" ], [ 1724815809.77, "4" ], [ 1724815814.771, "4" ], [ 1724815819.78, "4" ], [ 1724815824.775, "4" ], [ 1724815829.77, "4" ], [ 1724815834.772, "4" ], [ 1724815839.772, "4" ], [ 1724815844.77, "4" ], [ 1724815849.793, "6" ], [ 1724815854.787, "6" ], [ 1724815859.789, "5" ] ] }, "172.30.66.182:1008": { "Step1History": [ [ 1724815744.765, "378" ], [ 1724815749.765, "378" ], [ 1724815754.765, "378" ], [ 1724815759.765, "378" ], [ 1724815764.765, "378" ], [ 1724815769.765, "378" ], [ 1724815774.765, "378" ], [ 1724815779.765, "378" ], [ 1724815784.766, "378" ], [ 1724815789.765, "378" ], [ 1724815794.765, "378" ], [ 1724815799.765, "378" ], [ 1724815804.765, "378" ], [ 1724815809.765, "378" ], [ 1724815814.766, "378" ], [ 1724815819.765, "378" ], [ 1724815824.765, "378" ], [ 1724815829.765, "378" ], [ 1724815834.766, "378" ], [ 1724815839.765, "378" ], [ 1724815844.765, "378" ], [ 1724815849.765, "378" ], [ 1724815854.765, "378" ], [ 1724815859.765, "378" ] ], "Step2History": [ [ 1724815744.765, "0" ], [ 1724815749.765, "0" ], [ 1724815754.765, "0" ], [ 1724815759.765, "0" ], [ 1724815764.765, "0" ], [ 1724815769.765, "0" ], [ 1724815774.765, "0" ], [ 1724815779.765, "0" ], [ 1724815784.766, "0" ], [ 1724815789.765, "0" ], [ 1724815794.765, "0" ], [ 1724815799.765, "0" ], [ 1724815804.765, "0" ], [ 1724815809.765, "0" ], [ 1724815814.766, "0" ], [ 1724815819.765, "0" ], [ 1724815824.765, "0" ], [ 1724815829.765, "0" ], [ 1724815834.766, "0" ], [ 1724815839.765, "0" ], [ 1724815844.765, "0" ], [ 1724815849.765, "0" ], [ 1724815854.765, "0" ], [ 1724815859.765, "0" ] ] }, "172.30.69.27:1008": { "Step1History": [ [ 1724815744.908, "1490" ], [ 1724815749.834, "1490" ], [ 1724815754.769, "1491" ], [ 1724815759.77, "1491" ], [ 1724815764.769, "1491" ], [ 1724815769.795, "1499" ], [ 1724815774.769, "1505" ], [ 1724815779.779, "1505" ], [ 1724815784.769, "1505" ], [ 1724815789.778, "1505" ], [ 1724815794.782, "1505" ], [ 1724815799.769, "1507" ], [ 1724815804.783, "1507" ], [ 1724815809.768, "1508" ], [ 1724815814.77, "1508" ], [ 1724815819.768, "1508" ], [ 1724815824.767, "1508" ], [ 1724815829.767, "1508" ], [ 1724815834.768, "1508" ], [ 1724815839.769, "1508" ], [ 1724815844.767, "1508" ], [ 1724815849.772, "1508" ], [ 1724815854.782, "1508" ], [ 1724815859.781, "1508" ] ], "Step2History": [ [ 1724815744.908, "2" ], [ 1724815749.834, "2" ], [ 1724815754.769, "3" ], [ 1724815759.77, "3" ], [ 1724815764.769, "3" ], [ 1724815769.795, "11" ], [ 1724815774.769, "17" ], [ 1724815779.779, "17" ], [ 1724815784.769, "17" ], [ 1724815789.778, "17" ], [ 1724815794.782, "17" ], [ 1724815799.769, "19" ], [ 1724815804.783, "19" ], [ 1724815809.768, "20" ], [ 1724815814.77, "19" ], [ 1724815819.768, "19" ], [ 1724815824.767, "19" ], [ 1724815829.767, "19" ], [ 1724815834.768, "19" ], [ 1724815839.769, "19" ], [ 1724815844.767, "19" ], [ 1724815849.772, "19" ], [ 1724815854.782, "19" ], [ 1724815859.781, "18" ] ] }, "172.30.77.112:1008": { "Step1History": [ [ 1724815744.795, "7104" ], [ 1724815749.78, "7104" ], [ 1724815754.772, "7105" ], [ 1724815759.791, "7105" ], [ 1724815764.788, "7105" ], [ 1724815769.796, "7107" ], [ 1724815774.779, "7107" ], [ 1724815779.798, "7107" ], [ 1724815784.776, "7107" ], [ 1724815789.775, "7107" ], [ 1724815794.779, "7107" ], [ 1724815799.782, "7107" ], [ 1724815804.783, "7107" ], [ 1724815809.787, "7108" ], [ 1724815814.786, "7108" ], [ 1724815819.773, "7108" ], [ 1724815824.773, "7108" ], [ 1724815829.779, "7108" ], [ 1724815834.77, "7108" ], [ 1724815839.77, "7108" ], [ 1724815844.77, "7108" ], [ 1724815849.769, "7108" ], [ 1724815854.774, "7108" ], [ 1724815859.77, "7108" ] ], "Step2History": [ [ 1724815744.795, "3" ], [ 1724815749.78, "3" ], [ 1724815754.772, "4" ], [ 1724815759.791, "4" ], [ 1724815764.788, "4" ], [ 1724815769.796, "6" ], [ 1724815774.779, "6" ], [ 1724815779.798, "6" ], [ 1724815784.776, "5" ], [ 1724815789.775, "5" ], [ 1724815794.779, "5" ], [ 1724815799.782, "5" ], [ 1724815804.783, "5" ], [ 1724815809.787, "6" ], [ 1724815814.786, "6" ], [ 1724815819.773, "5" ], [ 1724815824.773, "5" ], [ 1724815829.779, "5" ], [ 1724815834.77, "5" ], [ 1724815839.77, "5" ], [ 1724815844.77, "5" ], [ 1724815849.769, "5" ], [ 1724815854.774, "5" ], [ 1724815859.77, "4" ] ] }, "172.30.77.5:1008": { "Step1History": [ [ 1724815744.777, "5861" ], [ 1724815749.772, "5861" ], [ 1724815754.771, "5862" ], [ 1724815759.78, "5862" ], [ 1724815764.778, "5862" ], [ 1724815769.774, "5865" ], [ 1724815774.775, "5867" ], [ 1724815779.783, "5867" ], [ 1724815784.79, "5867" ], [ 1724815789.777, "5867" ], [ 1724815794.771, "5867" ], [ 1724815799.773, "5868" ], [ 1724815804.79, "5868" ], [ 1724815809.773, "5869" ], [ 1724815814.772, "5869" ], [ 1724815819.772, "5869" ], [ 1724815824.772, "5869" ], [ 1724815829.775, "5869" ], [ 1724815834.78, "5870" ], [ 1724815839.776, "5870" ], [ 1724815844.771, "5870" ], [ 1724815849.773, "5871" ], [ 1724815854.771, "5871" ], [ 1724815859.782, "5871" ] ], "Step2History": [ [ 1724815744.777, "2" ], [ 1724815749.772, "2" ], [ 1724815754.771, "3" ], [ 1724815759.78, "3" ], [ 1724815764.778, "3" ], [ 1724815769.774, "6" ], [ 1724815774.775, "8" ], [ 1724815779.783, "7" ], [ 1724815784.79, "7" ], [ 1724815789.777, "7" ], [ 1724815794.771, "7" ], [ 1724815799.773, "8" ], [ 1724815804.79, "8" ], [ 1724815809.773, "9" ], [ 1724815814.772, "9" ], [ 1724815819.772, "9" ], [ 1724815824.772, "9" ], [ 1724815829.775, "9" ], [ 1724815834.78, "10" ], [ 1724815839.776, "10" ], [ 1724815844.771, "10" ], [ 1724815849.773, "11" ], [ 1724815854.771, "11" ], [ 1724815859.782, "10" ] ] }, "172.30.81.38:1008": { "Step1History": [ [ 1724815739.92, "6087" ], [ 1724815744.841, "6087" ], [ 1724815749.819, "6087" ], [ 1724815754.851, "6089" ], [ 1724815759.849, "6089" ], [ 1724815764.858, "6089" ], [ 1724815769.814, "6092" ], [ 1724815774.814, "6102" ], [ 1724815779.781, "6102" ], [ 1724815784.794, "6104" ], [ 1724815789.789, "6109" ], [ 1724815794.912, "6110" ], [ 1724815799.933, "6111" ], [ 1724815804.806, "6111" ], [ 1724815809.779, "6114" ], [ 1724815814.813, "6114" ], [ 1724815819.783, "6115" ], [ 1724815824.812, "6115" ], [ 1724815829.78, "6115" ], [ 1724815834.795, "6115" ], [ 1724815839.889, "6116" ], [ 1724815844.778, "6116" ], [ 1724815850.034, "6120" ], [ 1724815854.893, "6120" ], [ 1724815859.853, "6120" ] ], "Step2History": [ [ 1724815739.92, "63" ], [ 1724815744.841, "62" ], [ 1724815749.819, "62" ], [ 1724815754.851, "64" ], [ 1724815759.849, "64" ], [ 1724815764.858, "64" ], [ 1724815769.814, "67" ], [ 1724815774.814, "77" ], [ 1724815779.781, "76" ], [ 1724815784.794, "78" ], [ 1724815789.789, "83" ], [ 1724815794.912, "84" ], [ 1724815799.933, "73" ], [ 1724815804.806, "73" ], [ 1724815809.779, "76" ], [ 1724815814.813, "76" ], [ 1724815819.783, "74" ], [ 1724815824.812, "43" ], [ 1724815829.78, "43" ], [ 1724815834.795, "42" ], [ 1724815839.889, "43" ], [ 1724815844.778, "42" ], [ 1724815850.034, "46" ], [ 1724815854.893, "45" ], [ 1724815859.853, "33" ] ] }, "172.30.83.151:1008": { "Step1History": [ [ 1724815744.766, "508" ], [ 1724815749.765, "508" ], [ 1724815754.767, "509" ], [ 1724815759.77, "509" ], [ 1724815764.765, "509" ], [ 1724815769.77, "509" ], [ 1724815774.765, "509" ], [ 1724815779.766, "509" ], [ 1724815784.771, "509" ], [ 1724815789.768, "509" ], [ 1724815794.765, "509" ], [ 1724815799.766, "509" ], [ 1724815804.765, "509" ], [ 1724815809.769, "510" ], [ 1724815814.765, "510" ], [ 1724815819.765, "510" ], [ 1724815824.766, "510" ], [ 1724815829.766, "510" ], [ 1724815834.765, "510" ], [ 1724815839.765, "510" ], [ 1724815844.77, "510" ], [ 1724815849.877, "510" ], [ 1724815854.766, "510" ], [ 1724815859.766, "510" ] ], "Step2History": [ [ 1724815744.766, "1" ], [ 1724815749.765, "1" ], [ 1724815754.767, "2" ], [ 1724815759.77, "2" ], [ 1724815764.765, "2" ], [ 1724815769.77, "2" ], [ 1724815774.765, "2" ], [ 1724815779.766, "2" ], [ 1724815784.771, "2" ], [ 1724815789.768, "2" ], [ 1724815794.765, "2" ], [ 1724815799.766, "2" ], [ 1724815804.765, "2" ], [ 1724815809.769, "3" ], [ 1724815814.765, "3" ], [ 1724815819.765, "3" ], [ 1724815824.766, "3" ], [ 1724815829.766, "3" ], [ 1724815834.765, "3" ], [ 1724815839.765, "3" ], [ 1724815844.77, "3" ], [ 1724815849.877, "3" ], [ 1724815854.766, "3" ], [ 1724815859.766, "2" ] ] }, "172.30.84.204:1008": { "Step1History": [ [ 1724815744.771, "5977" ], [ 1724815749.77, "5977" ], [ 1724815754.77, "5978" ], [ 1724815759.771, "5978" ], [ 1724815764.837, "5978" ], [ 1724815769.817, "5978" ], [ 1724815774.772, "5978" ], [ 1724815779.773, "5978" ], [ 1724815784.771, "5978" ], [ 1724815789.776, "5978" ], [ 1724815794.771, "5978" ], [ 1724815799.771, "5978" ], [ 1724815804.771, "5978" ], [ 1724815809.771, "5979" ], [ 1724815814.806, "5979" ], [ 1724815819.793, "5979" ], [ 1724815824.77, "5979" ], [ 1724815829.77, "5979" ], [ 1724815834.77, "5979" ], [ 1724815839.807, "5979" ], [ 1724815844.771, "5979" ], [ 1724815849.777, "5979" ], [ 1724815854.77, "5979" ], [ 1724815859.771, "5979" ] ], "Step2History": [ [ 1724815744.771, "1" ], [ 1724815749.77, "1" ], [ 1724815754.77, "2" ], [ 1724815759.771, "2" ], [ 1724815764.837, "2" ], [ 1724815769.817, "2" ], [ 1724815774.772, "2" ], [ 1724815779.773, "2" ], [ 1724815784.771, "2" ], [ 1724815789.776, "2" ], [ 1724815794.771, "2" ], [ 1724815799.771, "2" ], [ 1724815804.771, "2" ], [ 1724815809.771, "3" ], [ 1724815814.806, "3" ], [ 1724815819.793, "3" ], [ 1724815824.77, "3" ], [ 1724815829.77, "3" ], [ 1724815834.77, "3" ], [ 1724815839.807, "3" ], [ 1724815844.771, "3" ], [ 1724815849.777, "3" ], [ 1724815854.77, "3" ], [ 1724815859.771, "2" ] ] }, "172.30.87.250:1008": { "Step1History": [ [ 1724815744.811, "6229" ], [ 1724815749.776, "6229" ], [ 1724815754.775, "6230" ], [ 1724815759.783, "6230" ], [ 1724815764.797, "6230" ], [ 1724815769.804, "6230" ], [ 1724815774.782, "6234" ], [ 1724815779.777, "6234" ], [ 1724815784.782, "6236" ], [ 1724815789.778, "6236" ], [ 1724815794.785, "6236" ], [ 1724815799.781, "6236" ], [ 1724815804.776, "6236" ], [ 1724815809.774, "6237" ], [ 1724815814.789, "6237" ], [ 1724815819.776, "6237" ], [ 1724815824.776, "6238" ], [ 1724815829.775, "6238" ], [ 1724815834.776, "6238" ], [ 1724815839.841, "6238" ], [ 1724815844.775, "6238" ], [ 1724815849.782, "6238" ], [ 1724815854.789, "6238" ], [ 1724815859.863, "6238" ] ], "Step2History": [ [ 1724815744.811, "5" ], [ 1724815749.776, "5" ], [ 1724815754.775, "6" ], [ 1724815759.783, "6" ], [ 1724815764.797, "6" ], [ 1724815769.804, "6" ], [ 1724815774.782, "9" ], [ 1724815779.777, "9" ], [ 1724815784.782, "11" ], [ 1724815789.778, "11" ], [ 1724815794.785, "11" ], [ 1724815799.781, "11" ], [ 1724815804.776, "10" ], [ 1724815809.774, "11" ], [ 1724815814.789, "11" ], [ 1724815819.776, "9" ], [ 1724815824.776, "10" ], [ 1724815829.775, "10" ], [ 1724815834.776, "10" ], [ 1724815839.841, "10" ], [ 1724815844.775, "10" ], [ 1724815849.782, "10" ], [ 1724815854.789, "10" ], [ 1724815859.863, "9" ] ] }, "172.30.88.130:1008": { "Step1History": [ [ 1724815744.792, "1256663" ], [ 1724815749.792, "1256663" ], [ 1724815754.816, "1256663" ], [ 1724815759.808, "1256663" ], [ 1724815764.794, "1256663" ], [ 1724815769.806, "1256663" ], [ 1724815774.795, "1256663" ], [ 1724815779.796, "1256663" ], [ 1724815784.791, "1256663" ], [ 1724815789.797, "1256663" ], [ 1724815794.794, "1256663" ], [ 1724815799.801, "1256663" ], [ 1724815804.809, "1256663" ], [ 1724815809.808, "1256663" ], [ 1724815814.797, "1256663" ], [ 1724815819.796, "1256663" ], [ 1724815824.808, "1256663" ], [ 1724815829.792, "1256663" ], [ 1724815834.793, "1256663" ], [ 1724815839.792, "1256663" ], [ 1724815844.791, "1256663" ], [ 1724815849.792, "1256663" ], [ 1724815854.804, "1256663" ], [ 1724815859.79, "1256663" ] ], "Step2History": [ [ 1724815744.792, "0" ], [ 1724815749.792, "0" ], [ 1724815754.816, "0" ], [ 1724815759.808, "0" ], [ 1724815764.794, "0" ], [ 1724815769.806, "0" ], [ 1724815774.795, "0" ], [ 1724815779.796, "0" ], [ 1724815784.791, "0" ], [ 1724815789.797, "0" ], [ 1724815794.794, "0" ], [ 1724815799.801, "0" ], [ 1724815804.809, "0" ], [ 1724815809.808, "0" ], [ 1724815814.797, "0" ], [ 1724815819.796, "0" ], [ 1724815824.808, "0" ], [ 1724815829.792, "0" ], [ 1724815834.793, "0" ], [ 1724815839.792, "0" ], [ 1724815844.791, "0" ], [ 1724815849.792, "0" ], [ 1724815854.804, "0" ], [ 1724815859.79, "0" ] ] }, "172.30.88.59:1008": { "Step1History": [ [ 1724815744.785, "239099" ], [ 1724815749.785, "239099" ], [ 1724815754.785, "239099" ], [ 1724815759.795, "239099" ], [ 1724815764.791, "239099" ], [ 1724815769.804, "239099" ], [ 1724815774.789, "239099" ], [ 1724815779.793, "239099" ], [ 1724815784.791, "239099" ], [ 1724815789.79, "239099" ], [ 1724815794.789, "239099" ], [ 1724815799.789, "239099" ], [ 1724815804.811, "239099" ], [ 1724815809.793, "239099" ], [ 1724815814.798, "239099" ], [ 1724815819.791, "239099" ], [ 1724815824.789, "239099" ], [ 1724815829.797, "239099" ], [ 1724815834.813, "239099" ], [ 1724815839.798, "239099" ], [ 1724815844.792, "239099" ], [ 1724815849.795, "239099" ], [ 1724815854.799, "239099" ], [ 1724815859.789, "239099" ] ], "Step2History": [ [ 1724815744.785, "0" ], [ 1724815749.785, "0" ], [ 1724815754.785, "0" ], [ 1724815759.795, "0" ], [ 1724815764.791, "0" ], [ 1724815769.804, "0" ], [ 1724815774.789, "0" ], [ 1724815779.793, "0" ], [ 1724815784.791, "0" ], [ 1724815789.79, "0" ], [ 1724815794.789, "0" ], [ 1724815799.789, "0" ], [ 1724815804.811, "0" ], [ 1724815809.793, "0" ], [ 1724815814.798, "0" ], [ 1724815819.791, "0" ], [ 1724815824.789, "0" ], [ 1724815829.797, "0" ], [ 1724815834.813, "0" ], [ 1724815839.798, "0" ], [ 1724815844.792, "0" ], [ 1724815849.795, "0" ], [ 1724815854.799, "0" ], [ 1724815859.789, "0" ] ] }, "172.30.89.188:1008": { "Step1History": [ [ 1724815744.77, "887" ], [ 1724815749.767, "887" ], [ 1724815754.766, "887" ], [ 1724815759.766, "888" ], [ 1724815764.767, "888" ], [ 1724815769.768, "888" ], [ 1724815774.765, "888" ], [ 1724815779.767, "888" ], [ 1724815784.77, "888" ], [ 1724815789.767, "888" ], [ 1724815794.766, "888" ], [ 1724815799.766, "888" ], [ 1724815804.766, "888" ], [ 1724815809.766, "888" ], [ 1724815814.766, "888" ], [ 1724815819.765, "888" ], [ 1724815824.765, "889" ], [ 1724815829.775, "889" ], [ 1724815834.767, "889" ], [ 1724815839.769, "889" ], [ 1724815844.773, "889" ], [ 1724815849.767, "889" ], [ 1724815854.769, "889" ], [ 1724815859.803, "889" ] ], "Step2History": [ [ 1724815744.77, "1" ], [ 1724815749.767, "1" ], [ 1724815754.766, "1" ], [ 1724815759.766, "2" ], [ 1724815764.767, "2" ], [ 1724815769.768, "2" ], [ 1724815774.765, "2" ], [ 1724815779.767, "2" ], [ 1724815784.77, "2" ], [ 1724815789.767, "2" ], [ 1724815794.766, "2" ], [ 1724815799.766, "2" ], [ 1724815804.766, "2" ], [ 1724815809.766, "2" ], [ 1724815814.766, "2" ], [ 1724815819.765, "2" ], [ 1724815824.765, "3" ], [ 1724815829.775, "3" ], [ 1724815834.767, "3" ], [ 1724815839.769, "3" ], [ 1724815844.773, "3" ], [ 1724815849.767, "3" ], [ 1724815854.769, "3" ], [ 1724815859.803, "3" ] ] } }, "memory": { "172.30.65.197:1008": { "Step1History": [ [ 1724815804.77, "0.24263713228970615" ], [ 1724815809.775, "0.2380175909461568" ], [ 1724815814.799, "0.20954973315366215" ], [ 1724815819.795, "0.2129663933239538" ], [ 1724815824.769, "0.22278062174639884" ], [ 1724815829.769, "0.23035520941219562" ], [ 1724815834.77, "0.2402673784218882" ], [ 1724815839.803, "0.24422311397859042" ], [ 1724815844.77, "0.23774517628687003" ], [ 1724815849.77, "0.22118337967414184" ], [ 1724815854.773, "0.1759448513443784" ], [ 1724815859.768, "0.19206860666985917" ] ], "Step2History": [ [ 1724815804.77, "0.24263713228970615" ], [ 1724815809.775, "0.2380175909461568" ], [ 1724815814.799, "0.20954973315366215" ], [ 1724815819.795, "0.2129663933239538" ], [ 1724815824.769, "0.22278062174639884" ], [ 1724815829.769, "0.23035520941219562" ], [ 1724815834.77, "0.2402673784218882" ], [ 1724815839.803, "0.24422311397859042" ], [ 1724815844.77, "0.23774517628687003" ], [ 1724815849.77, "0.22118337967414184" ], [ 1724815854.773, "0.1759448513443784" ], [ 1724815859.768, "0.19206860666985917" ] ] }, "172.30.65.210:1008": { "Step1History": [ [ 1724815804.77, "0.18209772280400868" ], [ 1724815809.77, "0.15836163896736097" ], [ 1724815814.771, "0.1416561177910345" ], [ 1724815819.78, "0.1416561177910345" ], [ 1724815824.775, "0.1416561177910345" ], [ 1724815829.77, "0.14430156544076977" ], [ 1724815834.772, "0.153477849835982" ], [ 1724815839.772, "0.206075224310549" ], [ 1724815844.77, "0.20064233681415294" ], [ 1724815849.793, "0.2301867367532726" ], [ 1724815854.787, "0.22245635982740677" ], [ 1724815859.789, "0.20878353212240902" ] ], "Step2History": [ [ 1724815804.77, "0.18209772280400868" ], [ 1724815809.77, "0.15836163896736097" ], [ 1724815814.771, "0.1416561177910345" ], [ 1724815819.78, "0.1416561177910345" ], [ 1724815824.775, "0.1416561177910345" ], [ 1724815829.77, "0.14430156544076977" ], [ 1724815834.772, "0.153477849835982" ], [ 1724815839.772, "0.206075224310549" ], [ 1724815844.77, "0.20064233681415294" ], [ 1724815849.793, "0.2301867367532726" ], [ 1724815854.787, "0.22245635982740677" ], [ 1724815859.789, "0.20878353212240902" ] ] }, "172.30.66.182:1008": { "Step1History": [ [ 1724815804.765, "0.027509673411760048" ], [ 1724815809.765, "0.027509673411760048" ], [ 1724815814.766, "0.027509673411760048" ], [ 1724815819.765, "0.027509673411760048" ], [ 1724815824.765, "0.027509673411760048" ], [ 1724815829.765, "0.027509673411760048" ], [ 1724815834.766, "0.027509673411760048" ], [ 1724815839.765, "0.027509673411760048" ], [ 1724815844.765, "0.027509673411760048" ], [ 1724815849.765, "0.027509673411760048" ], [ 1724815854.765, "0.027509673411760048" ], [ 1724815859.765, "0.027509673411760048" ] ], "Step2History": [ [ 1724815804.765, "0.027509673411760048" ], [ 1724815809.765, "0.027509673411760048" ], [ 1724815814.766, "0.027509673411760048" ], [ 1724815819.765, "0.027509673411760048" ], [ 1724815824.765, "0.027509673411760048" ], [ 1724815829.765, "0.027509673411760048" ], [ 1724815834.766, "0.027509673411760048" ], [ 1724815839.765, "0.027509673411760048" ], [ 1724815844.765, "0.027509673411760048" ], [ 1724815849.765, "0.027509673411760048" ], [ 1724815854.765, "0.027509673411760048" ], [ 1724815859.765, "0.027509673411760048" ] ] }, "172.30.69.27:1008": { "Step1History": [ [ 1724815804.783, "0.18863338542807398" ], [ 1724815809.768, "0.13688103466836934" ], [ 1724815814.77, "0.13213998017677564" ], [ 1724815819.768, "0.13874147273440468" ], [ 1724815824.767, "0.1547977895001256" ], [ 1724815829.767, "0.15743847314151085" ], [ 1724815834.768, "0.22739156312682285" ], [ 1724815839.769, "0.2176118582973558" ], [ 1724815844.767, "0.2380711087022965" ], [ 1724815849.772, "0.22479901453084417" ], [ 1724815854.782, "0.20415551455621098" ], [ 1724815859.781, "0.187785453812011" ] ], "Step2History": [ [ 1724815804.783, "0.18863338542807398" ], [ 1724815809.768, "0.13688103466836934" ], [ 1724815814.77, "0.13213998017677564" ], [ 1724815819.768, "0.13874147273440468" ], [ 1724815824.767, "0.1547977895001256" ], [ 1724815829.767, "0.15743847314151085" ], [ 1724815834.768, "0.22739156312682285" ], [ 1724815839.769, "0.2176118582973558" ], [ 1724815844.767, "0.2380711087022965" ], [ 1724815849.772, "0.22479901453084417" ], [ 1724815854.782, "0.20415551455621098" ], [ 1724815859.781, "0.187785453812011" ] ] }, "172.30.77.112:1008": { "Step1History": [ [ 1724815804.783, "0.17858411197028248" ], [ 1724815809.787, "0.17929679524539593" ], [ 1724815814.786, "0.176972887224167" ], [ 1724815819.773, "0.17505187819482443" ], [ 1724815824.773, "0.21841549082278755" ], [ 1724815829.779, "0.1276186269014271" ], [ 1724815834.77, "0.1276186269014271" ], [ 1724815839.77, "0.09252824067027741" ], [ 1724815844.77, "0.043529981698783506" ], [ 1724815849.769, "0.044682376757578794" ], [ 1724815854.774, "0.044682376757578794" ], [ 1724815859.77, "0.044499983295035656" ] ], "Step2History": [ [ 1724815804.783, "0.17858411197028248" ], [ 1724815809.787, "0.17929679524539593" ], [ 1724815814.786, "0.176972887224167" ], [ 1724815819.773, "0.17505187819482443" ], [ 1724815824.773, "0.21841549082278755" ], [ 1724815829.779, "0.1276186269014271" ], [ 1724815834.77, "0.1276186269014271" ], [ 1724815839.77, "0.09252824067027741" ], [ 1724815844.77, "0.043529981698783506" ], [ 1724815849.769, "0.044682376757578794" ], [ 1724815854.774, "0.044682376757578794" ], [ 1724815859.77, "0.044499983295035656" ] ] }, "172.30.77.5:1008": { "Step1History": [ [ 1724815804.79, "0.16319500385451585" ], [ 1724815809.773, "0.16071375981726008" ], [ 1724815814.772, "0.16013657236404943" ], [ 1724815819.772, "0.16013657236404943" ], [ 1724815824.772, "0.15959161930499924" ], [ 1724815829.775, "0.15918995771787914" ], [ 1724815834.78, "0.165552693025863" ], [ 1724815839.776, "0.16010625594727665" ], [ 1724815844.771, "0.16010625594727665" ], [ 1724815849.773, "0.15969079729701302" ], [ 1724815854.771, "0.1759422527943693" ], [ 1724815859.782, "0.17331598491851072" ] ], "Step2History": [ [ 1724815804.79, "0.16319500385451585" ], [ 1724815809.773, "0.16071375981726008" ], [ 1724815814.772, "0.16013657236404943" ], [ 1724815819.772, "0.16013657236404943" ], [ 1724815824.772, "0.15959161930499924" ], [ 1724815829.775, "0.15918995771787914" ], [ 1724815834.78, "0.165552693025863" ], [ 1724815839.776, "0.16010625594727665" ], [ 1724815844.771, "0.16010625594727665" ], [ 1724815849.773, "0.15969079729701302" ], [ 1724815854.771, "0.1759422527943693" ], [ 1724815859.782, "0.17331598491851072" ] ] }, "172.30.81.38:1008": { "Step1History": [ [ 1724815799.933, "0.3140681166575717" ], [ 1724815804.806, "0.3130256650122565" ], [ 1724815809.779, "0.31266465217170725" ], [ 1724815814.813, "0.32144812242387794" ], [ 1724815819.783, "0.31690641384012486" ], [ 1724815824.812, "0.31225352443812543" ], [ 1724815829.78, "0.31632687531785836" ], [ 1724815834.795, "0.3130504749778195" ], [ 1724815839.889, "0.3176026396318473" ], [ 1724815844.778, "0.31197758317525487" ], [ 1724815850.034, "0.3199667014377406" ], [ 1724815854.893, "0.3129332308762187" ], [ 1724815859.853, "0.31149313920927363" ] ], "Step2History": [ [ 1724815799.933, "0.3140681166575717" ], [ 1724815804.806, "0.3130256650122565" ], [ 1724815809.779, "0.31266465217170725" ], [ 1724815814.813, "0.32144812242387794" ], [ 1724815819.783, "0.31690641384012486" ], [ 1724815824.812, "0.31225352443812543" ], [ 1724815829.78, "0.31632687531785836" ], [ 1724815834.795, "0.3130504749778195" ], [ 1724815839.889, "0.3176026396318473" ], [ 1724815844.778, "0.31197758317525487" ], [ 1724815850.034, "0.3199667014377406" ], [ 1724815854.893, "0.3129332308762187" ], [ 1724815859.853, "0.31149313920927363" ] ] }, "172.30.83.151:1008": { "Step1History": [ [ 1724815804.765, "0.16552095359360905" ], [ 1724815809.769, "0.16944129937399693" ], [ 1724815814.765, "0.16944129937399693" ], [ 1724815819.765, "0.16073535253043086" ], [ 1724815824.766, "0.15882207728087727" ], [ 1724815829.766, "0.15821141802873998" ], [ 1724815834.765, "0.16320620236765027" ], [ 1724815839.765, "0.1592533747121487" ], [ 1724815844.77, "0.19570960832426934" ], [ 1724815849.877, "0.24901100424058614" ], [ 1724815854.766, "0.2290746192196183" ], [ 1724815859.766, "0.23411413574082804" ] ], "Step2History": [ [ 1724815804.765, "0.16552095359360905" ], [ 1724815809.769, "0.16944129937399693" ], [ 1724815814.765, "0.16944129937399693" ], [ 1724815819.765, "0.16073535253043086" ], [ 1724815824.766, "0.15882207728087727" ], [ 1724815829.766, "0.15821141802873998" ], [ 1724815834.765, "0.16320620236765027" ], [ 1724815839.765, "0.1592533747121487" ], [ 1724815844.77, "0.19570960832426934" ], [ 1724815849.877, "0.24901100424058614" ], [ 1724815854.766, "0.2290746192196183" ], [ 1724815859.766, "0.23411413574082804" ] ] }, "172.30.84.204:1008": { "Step1History": [ [ 1724815804.771, "0.23773125548324986" ], [ 1724815809.771, "0.24411657342821752" ], [ 1724815814.806, "0.22533153167199368" ], [ 1724815819.793, "0.21710650219082514" ], [ 1724815824.77, "0.2369101755506142" ], [ 1724815829.77, "0.24715446399956442" ], [ 1724815834.77, "0.22005486652733489" ], [ 1724815839.807, "0.25203850061189664" ], [ 1724815844.771, "0.23628992641153856" ], [ 1724815849.777, "0.22982621887462987" ], [ 1724815854.77, "0.22690049904534224" ], [ 1724815859.771, "0.22760148884541473" ] ], "Step2History": [ [ 1724815804.771, "0.23773125548324986" ], [ 1724815809.771, "0.24411657342821752" ], [ 1724815814.806, "0.22533153167199368" ], [ 1724815819.793, "0.21710650219082514" ], [ 1724815824.77, "0.2369101755506142" ], [ 1724815829.77, "0.24715446399956442" ], [ 1724815834.77, "0.22005486652733489" ], [ 1724815839.807, "0.25203850061189664" ], [ 1724815844.771, "0.23628992641153856" ], [ 1724815849.777, "0.22982621887462987" ], [ 1724815854.77, "0.22690049904534224" ], [ 1724815859.771, "0.22760148884541473" ] ] }, "172.30.87.250:1008": { "Step1History": [ [ 1724815804.776, "0.1797639774148882" ], [ 1724815809.774, "0.17198794025314826" ], [ 1724815814.789, "0.1781031327376467" ], [ 1724815819.776, "0.24580624963651235" ], [ 1724815824.776, "0.23315848803986422" ], [ 1724815829.775, "0.2277457702411578" ], [ 1724815834.776, "0.25882294346421364" ], [ 1724815839.841, "0.2529507772758039" ], [ 1724815844.775, "0.24565540999550822" ], [ 1724815849.782, "0.2440245105136096" ], [ 1724815854.789, "0.2345670258852703" ], [ 1724815859.863, "0.25535777702708556" ] ], "Step2History": [ [ 1724815804.776, "0.1797639774148882" ], [ 1724815809.774, "0.17198794025314826" ], [ 1724815814.789, "0.1781031327376467" ], [ 1724815819.776, "0.24580624963651235" ], [ 1724815824.776, "0.23315848803986422" ], [ 1724815829.775, "0.2277457702411578" ], [ 1724815834.776, "0.25882294346421364" ], [ 1724815839.841, "0.2529507772758039" ], [ 1724815844.775, "0.24565540999550822" ], [ 1724815849.782, "0.2440245105136096" ], [ 1724815854.789, "0.2345670258852703" ], [ 1724815859.863, "0.25535777702708556" ] ] }, "172.30.88.130:1008": { "Step1History": [ [ 1724815804.809, "0.315289990246793" ], [ 1724815809.808, "0.315828460900512" ], [ 1724815814.797, "0.3165582074426355" ], [ 1724815819.796, "0.31829108599076206" ], [ 1724815824.808, "0.32033803520501947" ], [ 1724815829.792, "0.3233179875793098" ], [ 1724815834.793, "0.3453882468581576" ], [ 1724815839.792, "0.3642132683807374" ], [ 1724815844.791, "0.38052076596899964" ], [ 1724815849.792, "0.39913946639401526" ], [ 1724815854.804, "0.41716359616841486" ], [ 1724815859.79, "0.4321467765639376" ] ], "Step2History": [ [ 1724815804.809, "0.315289990246793" ], [ 1724815809.808, "0.315828460900512" ], [ 1724815814.797, "0.3165582074426355" ], [ 1724815819.796, "0.31829108599076206" ], [ 1724815824.808, "0.32033803520501947" ], [ 1724815829.792, "0.3233179875793098" ], [ 1724815834.793, "0.3453882468581576" ], [ 1724815839.792, "0.3642132683807374" ], [ 1724815844.791, "0.38052076596899964" ], [ 1724815849.792, "0.39913946639401526" ], [ 1724815854.804, "0.41716359616841486" ], [ 1724815859.79, "0.4321467765639376" ] ] }, "172.30.88.59:1008": { "Step1History": [ [ 1724815804.811, "0.22900191206591125" ], [ 1724815809.793, "0.26508167877739275" ], [ 1724815814.798, "0.2658520797229772" ], [ 1724815819.791, "0.2669548304168547" ], [ 1724815824.789, "0.2685220576702732" ], [ 1724815829.797, "0.27810328420592595" ], [ 1724815834.813, "0.3006624126863914" ], [ 1724815839.798, "0.33820980336281214" ], [ 1724815844.792, "0.3648435194301485" ], [ 1724815849.795, "0.36148931555308894" ], [ 1724815854.799, "0.3196161115556831" ], [ 1724815859.789, "0.32142059729172373" ] ], "Step2History": [ [ 1724815804.811, "0.22900191206591125" ], [ 1724815809.793, "0.26508167877739275" ], [ 1724815814.798, "0.2658520797229772" ], [ 1724815819.791, "0.2669548304168547" ], [ 1724815824.789, "0.2685220576702732" ], [ 1724815829.797, "0.27810328420592595" ], [ 1724815834.813, "0.3006624126863914" ], [ 1724815839.798, "0.33820980336281214" ], [ 1724815844.792, "0.3648435194301485" ], [ 1724815849.795, "0.36148931555308894" ], [ 1724815854.799, "0.3196161115556831" ], [ 1724815859.789, "0.32142059729172373" ] ] }, "172.30.89.188:1008": { "Step1History": [ [ 1724815804.766, "0.15876088761518692" ], [ 1724815809.766, "0.15876905448664408" ], [ 1724815814.766, "0.1587933076200623" ], [ 1724815819.765, "0.1587933076200623" ], [ 1724815824.765, "0.15880141262128114" ], [ 1724815829.775, "0.16320979084147236" ], [ 1724815834.767, "0.16857610596144496" ], [ 1724815839.769, "0.16133351399442922" ], [ 1724815844.773, "0.24437147881006208" ], [ 1724815849.767, "0.2376609708925277" ], [ 1724815854.769, "0.23099500954657778" ], [ 1724815859.803, "0.2664031613216968" ] ], "Step2History": [ [ 1724815804.766, "0.15876088761518692" ], [ 1724815809.766, "0.15876905448664408" ], [ 1724815814.766, "0.1587933076200623" ], [ 1724815819.765, "0.1587933076200623" ], [ 1724815824.765, "0.15880141262128114" ], [ 1724815829.775, "0.16320979084147236" ], [ 1724815834.767, "0.16857610596144496" ], [ 1724815839.769, "0.16133351399442922" ], [ 1724815844.773, "0.24437147881006208" ], [ 1724815849.767, "0.2376609708925277" ], [ 1724815854.769, "0.23099500954657778" ], [ 1724815859.803, "0.2664031613216968" ] ] } } }

@djshow832
Copy link
Collaborator

I didn't see 172.30.72.136 in the metrics. Maybe the instance is unavailable.

@salarali
Copy link
Author

Yeah. The server is actually not there anymore. The autoscaling removed it. But for some reason, tiproxy was unable to detect that.
I see this in the logs for tiproxy:

Aug 28 01:16:58 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:16:58.584 +00:00] [INFO] [main.nsmgr.observer] [observer/backend_observer.go:200] [update backend] [namespace=default] [backend_addr=172.30.72.136:5678] [prev="down, err: Get \"http://172.30.72.136:1008/status\": dial tcp 172.30.72.136:1008: connect: connection refused: connect status port failed"] [cur=healthy]

But I still see this after this:

Aug 28 01:17:05 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:17:05.010 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.76.23:5678] [to=172.30.72.136:5678] [reason=cpu]
Aug 28 01:17:05 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:17:05.019 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.76.23:5678] [to=172.30.72.136:5678] [reason=cpu]

So for some reason, its list of backends didnt update properly.

@djshow832
Copy link
Collaborator

djshow832 commented Aug 28, 2024 •
edited
Loading

Yeah. The server is actually not there anymore. The autoscaling removed it. But for some reason, tiproxy was unable to detect that. I see this in the logs for tiproxy:

Aug 28 01:16:58 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:16:58.584 +00:00] [INFO] [main.nsmgr.observer] [observer/backend_observer.go:200] [update backend] [namespace=default] [backend_addr=172.30.72.136:5678] [prev="down, err: Get \"http://172.30.72.136:1008/status\": dial tcp 172.30.72.136:1008: connect: connection refused: connect status port failed"] [cur=healthy]

But I still see this after this:

Aug 28 01:17:05 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:17:05.010 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.76.23:5678] [to=172.30.72.136:5678] [reason=cpu]
Aug 28 01:17:05 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:17:05.019 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.76.23:5678] [to=172.30.72.136:5678] [reason=cpu]

So for some reason, its list of backends didnt update properly.

The first log indicates that the instance was unhealthy but now is healthy. prev indicates the previous status and cur indicates the current status. I guess there should be another update backend afterward saying that it's down again.

@salarali
Copy link
Author

Right. You are correct. Sorry.

Aug 28 03:08:07 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:08:07.288 +00:00] [INFO] [main.nsmgr.observer] [observer/backend_observer.go:215] [update backend] [namespace=default] [backend_addr=172.30.72.136:5678] [prev=healthy] [cur="down, err: http status 500: connect status port failed"]
Aug 28 03:08:07 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:08:07.289 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.77.5:5678] [reason=status]
Aug 28 03:08:07 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:08:07.299 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.81.38:5678] [reason=status]
Aug 28 03:08:07 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:08:07.309 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.81.38:5678] [reason=status]
Aug 28 03:08:07 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:08:07.319 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.81.38:5678] [reason=status]

After this failure, the reason did change to status.

@djshow832
Copy link
Collaborator

djshow832 commented Aug 28, 2024 •
edited
Loading

If there's no more problem, I'll close this issue.

@salarali
Copy link
Author

So, the issue is still there. Even though tiproxy detected that the backend is down, its still trying to route from it. My naive guess is that this did not get run:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L298

@salarali
Copy link
Author

I'm still testing this out to see if this is an issue, but one possible issue I can see is here:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L277-L283

As far as I can see, the backend wont be removed if there are still connections associated to it. Is that a possibility?

I will let you know if I confirm this hypothesis.

@salarali
Copy link
Author

It seems like it is possible. I put some extra logs in there and I see this:

Aug 28 04:55:09 ip-172-30-94-236.ec2.internal tiproxy[4064141]: [2024/08/28 04:55:09.214 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:291] [current backends] [namespace=default] [addr=172.30.77.5:5678] [healthy=false] [connScore=-3] [connListLen=5]

I put this log:

	for addr, backend := range router.backends {
		router.logger.Info("current backends", zap.String("addr", addr), zap.Bool("healthy", backend.Healthy()), zap.Int("connScore", backend.connScore), zap.Int("connListLen", backend.connList.Len()))
	}

in this function:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L250

@djshow832
Copy link
Collaborator

djshow832 commented Aug 28, 2024 •
edited
Loading

Even though tiproxy detected that the backend is down, its still trying to route from it.

It's expected. TiProxy should migrate connections from the unhealthy backend. Why shouldn't it? The problem should be that the connection migrations never succeed. If so, the errors should be logged with redirect connection failed.

[connScore=-3] [connListLen=5]

This is unexpected. The connScore should always be >=0. connScore = connections on it + connections migrating to it - connections migrating from it. connections migrating from it should never be greater than connections on it.
Now that you can reproduce the problem, can you help to locate the reason?

@salarali
Copy link
Author

salarali commented Aug 28, 2024 •
edited
Loading

Right, I am getting those errors.

Aug 28 10:19:30 ip-172-30-94-236.ec2.internal tiproxy[4092693]: [2024/08/28 10:19:30.290 +00:00] [WARN] [main.proxy.conn.be] [backend/backend_conn_mgr.go:598] [redirect connection failed] [connID=1237819] [client_addr=172.30.72.32:34772] [addr=0.0.0.0:5678] [ns=default] [from=172.30.78.14:5678] [to=172.30.87.124:5678] [redirect_err="this is an error from the backend connection: failed to read the connection: EOF"] [redirect_errVerbose="this is an error from the backend connection: failed to read the connection: EOF\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*basicReadWriter).Read\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:105\ngithub.com/pingcap/tiproxy/pkg/proxy/net.ReadFull\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:183\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*PacketIO).readOnePacket\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:247\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*PacketIO).ReadPacket\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:270\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*CmdProcessor).query\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/cmd_processor_query.go:30\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).querySessionStates\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:431\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).tryRedirect\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:510\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).processSignals.func1\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:458\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).processSignals\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:460\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).Connect.func1\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:214\ngithub.com/pingcap/tiproxy/lib/util/waitgroup.(*WaitGroup).RunWithRecover.func1\n\t/Users/salar/repos/tiproxy/lib/util/waitgroup/waitgroup.go:38\nruntime.goexit\n\t/opt/homebrew/Cellar/go/1.22.4/libexec/src/runtime/asm_arm64.s:1222"]
Aug 28 10:19:30 ip-172-30-94-236.ec2.internal tiproxy[4092693]: [2024/08/28 10:19:30.343 +00:00] [WARN] [main.proxy.conn.be] [backend/backend_conn_mgr.go:598] [redirect connection failed] [connID=1239985] [client_addr=172.30.72.32:36640] [addr=0.0.0.0:5678] [ns=default] [from=172.30.78.14:5678] [to=172.30.87.124:5678] [redirect_err="this is an error from the backend connection: failed to read the connection: EOF"] [redirect_errVerbose="this is an error from the backend connection: failed to read the connection: EOF\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*basicReadWriter).Read\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:105\ngithub.com/pingcap/tiproxy/pkg/proxy/net.ReadFull\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:183\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*PacketIO).readOnePacket\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:247\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*PacketIO).ReadPacket\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:270\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*CmdProcessor).query\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/cmd_processor_query.go:30\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).querySessionStates\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:431\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).tryRedirect\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:510\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).processSignals.func1\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:458\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).processSignals\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:460\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).Connect.func1\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:214\ngithub.com/pingcap/tiproxy/lib/util/waitgroup.(*WaitGroup).RunWithRecover.func1\n\t/Users/salar/repos/tiproxy/lib/util/waitgroup/waitgroup.go:38\nruntime.goexit\n\t/opt/homebrew/Cellar/go/1.22.4/libexec/src/runtime/asm_arm64.s:1222"]

I will debug more and try to figure out the reason

@djshow832
Copy link
Collaborator

These errors should not be related to the problem. When the query fails, the connections are disconnected and then the router clears them.
https://github.com/pingcap/tiproxy/blob/v1.2.0/pkg/proxy/backend/backend_conn_mgr.go#L513

@salarali
Copy link
Author

salarali commented Sep 6, 2024

One thing I wanted to run by you. The reason why connScore might be negative, might be because of this:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L385

It decrements and increments the score, but the redirect can fail: https://github.com/pingcap/tiproxy/blob/main/pkg/proxy/backend/backend_conn_mgr.go#L595

That has fixed that the connScore is never negative for me, but I am still getting connLen to be greater than 0 for bad nodes, so still investigating that.

@djshow832
Copy link
Collaborator

One thing I wanted to run by you. The reason why connScore might be negative, might be because of this:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L385

It decrements and increments the score, but the redirect can fail: https://github.com/pingcap/tiproxy/blob/main/pkg/proxy/backend/backend_conn_mgr.go#L595

Yes, that's possible, it should be handled. Appreciate that.

@djshow832 djshow832 mentioned this issue Sep 11, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@salarali @djshow832