APP-911 - rc: make reconnection logic more reliable

[edaniels]

This provides more reliable reconnection logic in addition to disconnection detection logic (partially provided by viamrobotics/goutils@fd2acb7). Previously we were reconnecting in a few different ways and treating some errors as disconnects when that wasn't the case. This should clean up the assumptions and logic a bit.

[github-actions]

Package	Line Rate	Delta	Health
go.viam.com/rdk/components/arm	59%	0.00%	➖
go.viam.com/rdk/components/arm/universalrobots	12%	0.00%	❌
go.viam.com/rdk/components/arm/xarm	2%	0.00%	❌
go.viam.com/rdk/components/arm/yahboom	7%	0.00%	❌
go.viam.com/rdk/components/audioinput	55%	-0.34%	➖
go.viam.com/rdk/components/base	68%	0.00%	➖
go.viam.com/rdk/components/base/agilex	62%	0.00%	➖
go.viam.com/rdk/components/base/boat	41%	0.00%	❌
go.viam.com/rdk/components/base/wheeled	76%	+0.35%	✅
go.viam.com/rdk/components/board	69%	0.00%	➖
go.viam.com/rdk/components/board/arduino	10%	-0.20%	❌
go.viam.com/rdk/components/board/commonsysfs	47%	0.00%	❌
go.viam.com/rdk/components/board/fake	39%	0.00%	❌
go.viam.com/rdk/components/board/numato	19%	0.00%	❌
go.viam.com/rdk/components/board/pi	50%	0.00%	➖
go.viam.com/rdk/components/camera	66%	-0.27%	➖
go.viam.com/rdk/components/camera/fake	67%	0.00%	➖
go.viam.com/rdk/components/camera/ffmpeg	72%	+1.27%	✅
go.viam.com/rdk/components/camera/transformpipeline	81%	+0.80%	✅
go.viam.com/rdk/components/camera/videosource	55%	-0.20%	➖
go.viam.com/rdk/components/encoder/fake	77%	0.00%	✅
go.viam.com/rdk/components/gantry	68%	0.00%	➖
go.viam.com/rdk/components/gantry/multiaxis	84%	0.00%	✅
go.viam.com/rdk/components/gantry/oneaxis	86%	0.00%	✅
go.viam.com/rdk/components/generic	84%	0.00%	✅
go.viam.com/rdk/components/gripper	82%	0.00%	✅
go.viam.com/rdk/components/input	86%	0.00%	✅
go.viam.com/rdk/components/input/gpio	83%	-3.74%	✅
go.viam.com/rdk/components/motor	82%	0.00%	✅
go.viam.com/rdk/components/motor/dmc4000	69%	0.00%	➖
go.viam.com/rdk/components/motor/fake	58%	-1.56%	➖
go.viam.com/rdk/components/motor/gpio	64%	-1.01%	➖
go.viam.com/rdk/components/motor/gpiostepper	56%	-2.32%	➖
go.viam.com/rdk/components/motor/tmcstepper	62%	-3.28%	➖
go.viam.com/rdk/components/movementsensor	67%	0.00%	➖
go.viam.com/rdk/components/movementsensor/cameramono	40%	+1.19%	❌
go.viam.com/rdk/components/movementsensor/gpsnmea	37%	-0.23%	❌
go.viam.com/rdk/components/movementsensor/gpsrtk	28%	+0.10%	❌
go.viam.com/rdk/components/posetracker	88%	0.00%	✅
go.viam.com/rdk/components/sensor	88%	0.00%	✅
go.viam.com/rdk/components/sensor/ultrasonic	34%	+2.59%	❌
go.viam.com/rdk/components/servo	77%	0.00%	✅
go.viam.com/rdk/config	77%	+0.06%	✅
go.viam.com/rdk/control	57%	0.00%	➖
go.viam.com/rdk/data	77%	-0.36%	✅
go.viam.com/rdk/grpc	25%	0.00%	❌
go.viam.com/rdk/ml	67%	0.00%	➖
go.viam.com/rdk/ml/inference	70%	0.00%	✅
go.viam.com/rdk/motionplan	69%	-0.02%	➖
go.viam.com/rdk/operation	84%	0.00%	✅
go.viam.com/rdk/pointcloud	71%	0.00%	✅
go.viam.com/rdk/protoutils	62%	0.00%	➖
go.viam.com/rdk/referenceframe	78%	0.00%	✅
go.viam.com/rdk/registry	88%	0.00%	✅
go.viam.com/rdk/resource	85%	0.00%	✅
go.viam.com/rdk/rimage	78%	0.00%	✅
go.viam.com/rdk/rimage/depthadapter	94%	0.00%	✅
go.viam.com/rdk/rimage/transform	73%	0.00%	✅
go.viam.com/rdk/rimage/transform/cmd/extrinsic_calibration	67%	0.00%	➖
go.viam.com/rdk/robot	93%	0.00%	✅
go.viam.com/rdk/robot/client	80%	+0.88%	✅
go.viam.com/rdk/robot/framesystem	68%	0.00%	➖
go.viam.com/rdk/robot/impl	80%	+0.35%	✅
go.viam.com/rdk/robot/server	58%	-0.97%	➖
go.viam.com/rdk/robot/web	61%	+0.22%	➖
go.viam.com/rdk/robot/web/stream	87%	0.00%	✅
go.viam.com/rdk/services/armremotecontrol	75%	0.00%	✅
go.viam.com/rdk/services/armremotecontrol/builtin	25%	0.00%	❌
go.viam.com/rdk/services/baseremotecontrol	75%	0.00%	✅
go.viam.com/rdk/services/baseremotecontrol/builtin	71%	0.00%	✅
go.viam.com/rdk/services/datamanager	62%	0.00%	➖
go.viam.com/rdk/services/datamanager/builtin	78%	0.00%	✅
go.viam.com/rdk/services/datamanager/datacapture	21%	-13.26%	❌
go.viam.com/rdk/services/datamanager/datasync	72%	+2.05%	✅
go.viam.com/rdk/services/motion	68%	0.00%	➖
go.viam.com/rdk/services/motion/builtin	89%	0.00%	✅
go.viam.com/rdk/services/navigation	54%	0.00%	➖
go.viam.com/rdk/services/sensors	78%	0.00%	✅
go.viam.com/rdk/services/sensors/builtin	97%	0.00%	✅
go.viam.com/rdk/services/shell	15%	0.00%	❌
go.viam.com/rdk/services/slam	86%	0.00%	✅
go.viam.com/rdk/services/slam/builtin	73%	0.00%	✅
go.viam.com/rdk/services/vision	82%	0.00%	✅
go.viam.com/rdk/services/vision/builtin	74%	0.00%	✅
go.viam.com/rdk/spatialmath	85%	0.00%	✅
go.viam.com/rdk/subtype	96%	0.00%	✅
go.viam.com/rdk/utils	71%	-0.13%	✅
go.viam.com/rdk/vision	26%	0.00%	❌
go.viam.com/rdk/vision/chess	80%	0.00%	✅
go.viam.com/rdk/vision/delaunay	87%	0.00%	✅
go.viam.com/rdk/vision/keypoints	92%	0.00%	✅
go.viam.com/rdk/vision/objectdetection	82%	0.00%	✅
go.viam.com/rdk/vision/odometry	60%	0.00%	➖
go.viam.com/rdk/vision/odometry/cmd	0%	0.00%	❌
go.viam.com/rdk/vision/segmentation	49%	0.00%	❌
go.viam.com/rdk/web/server	26%	0.00%	❌
Summary	66% (19081 / 28889)	+0.01%	➖

[stevebriskin]

opened a retroactive ticket. aptly numbered APP-911

[DTCurrie]

✅