Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DATA-749] Fix SLAM integration test timeout #1600

Merged
merged 4 commits into from
Nov 17, 2022

Conversation

kkufieta
Copy link
Contributor

@kkufieta kkufieta commented Nov 16, 2022

https://viam.atlassian.net/browse/DATA-749

Orbslam is known to hang frequently indefinitely, especially when running it with a mono camera. When orbslam hangs, it will cause a timeout and is the most likely suspect for the observed timeout issues mentioned in the ticket.

This PR updates the integration test to make sure adding an image to orbslam does not exceed 3 minutes. If it does, we assume that orbslam hangs and handle it accordingly.

@viambot viambot added the safe to test This pull request is marked safe to test from a trusted zone label Nov 16, 2022
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 16, 2022
@kkufieta kkufieta changed the title [DATA-749] Fix timeout issue for SLAM integration tests [DATA-749] Fix SLAM integration test timeout Nov 16, 2022
@kkufieta kkufieta marked this pull request as ready for review November 16, 2022 17:54
@kkufieta kkufieta requested review from JohnN193 and a user November 16, 2022 17:54
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 16, 2022
Copy link
Member

@JohnN193 JohnN193 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a few nits

if !orbslam_hangs {
test.That(t, err, test.ShouldBeNil)
} else if err != nil {
t.Skip("Skipping test because orbslam failed to shut down")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] log should mention the hang issue

@@ -293,13 +335,27 @@ func integrationTestHelper(t *testing.T, mode slam.Mode) {
break
}
test.That(t, strings.Contains(line, "Fail to track local map!"), test.ShouldBeFalse)
if time.Since(start_time_sent_image) > time.Duration(dataInsertionMaxTimeoutMin)*time.Minute {
orbslam_hangs = true
t.Log("orbslam hangs")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] log should be more descriptive, maybe mention exiting data loop?

@@ -153,13 +159,27 @@ func integrationTestHelper(t *testing.T, mode slam.Mode) {
break
}
test.That(t, strings.Contains(line, "Fail to track local map!"), test.ShouldBeFalse)
if time.Since(start_time_sent_image) > time.Duration(dataInsertionMaxTimeoutMin)*time.Minute {
orbslam_hangs = true
t.Log("orbslam hangs")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] log should be more descriptive, maybe mention exiting data loop?

Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great check to limit/control time allowed for images to be passed. Suggested some reordering of the for loop to prevent hanging log from appearing and then a successful pass.

services/slam/builtin/orbslam_int_test.go Show resolved Hide resolved
services/slam/builtin/orbslam_int_test.go Show resolved Hide resolved
services/slam/builtin/orbslam_int_test.go Show resolved Hide resolved
services/slam/builtin/orbslam_int_test.go Show resolved Hide resolved
services/slam/builtin/orbslam_int_test.go Show resolved Hide resolved
services/slam/builtin/orbslam_int_test.go Show resolved Hide resolved
services/slam/builtin/orbslam_int_test.go Show resolved Hide resolved
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 16, 2022
@github-actions
Copy link
Contributor

Code Coverage

Package Line Rate Delta Health
go.viam.com/rdk/components/arm 59% 0.00%
go.viam.com/rdk/components/arm/universalrobots 13% 0.00%
go.viam.com/rdk/components/arm/xarm 2% 0.00%
go.viam.com/rdk/components/arm/yahboom 7% 0.00%
go.viam.com/rdk/components/audioinput 55% 0.00%
go.viam.com/rdk/components/base 68% 0.00%
go.viam.com/rdk/components/base/agilex 62% 0.00%
go.viam.com/rdk/components/base/boat 41% 0.00%
go.viam.com/rdk/components/base/wheeled 76% 0.00%
go.viam.com/rdk/components/board 69% 0.00%
go.viam.com/rdk/components/board/arduino 10% 0.00%
go.viam.com/rdk/components/board/commonsysfs 47% 0.00%
go.viam.com/rdk/components/board/fake 38% 0.00%
go.viam.com/rdk/components/board/numato 19% 0.00%
go.viam.com/rdk/components/board/pi 50% 0.00%
go.viam.com/rdk/components/camera 66% 0.00%
go.viam.com/rdk/components/camera/fake 67% 0.00%
go.viam.com/rdk/components/camera/ffmpeg 77% 0.00%
go.viam.com/rdk/components/camera/transformpipeline 80% -1.24%
go.viam.com/rdk/components/camera/videosource 57% 0.00%
go.viam.com/rdk/components/encoder/fake 77% 0.00%
go.viam.com/rdk/components/gantry 68% 0.00%
go.viam.com/rdk/components/gantry/multiaxis 84% 0.00%
go.viam.com/rdk/components/gantry/oneaxis 86% 0.00%
go.viam.com/rdk/components/generic 84% 0.00%
go.viam.com/rdk/components/gripper 81% 0.00%
go.viam.com/rdk/components/input 87% 0.00%
go.viam.com/rdk/components/input/gpio 83% 0.00%
go.viam.com/rdk/components/motor 82% 0.00%
go.viam.com/rdk/components/motor/dimensionengineering 63% 0.00%
go.viam.com/rdk/components/motor/dmc4000 69% 0.00%
go.viam.com/rdk/components/motor/fake 58% 0.00%
go.viam.com/rdk/components/motor/gpio 64% 0.00%
go.viam.com/rdk/components/motor/gpiostepper 57% 0.00%
go.viam.com/rdk/components/motor/tmcstepper 62% 0.00%
go.viam.com/rdk/components/movementsensor 76% 0.00%
go.viam.com/rdk/components/movementsensor/cameramono 40% 0.00%
go.viam.com/rdk/components/movementsensor/gpsnmea 37% 0.00%
go.viam.com/rdk/components/movementsensor/gpsrtk 28% 0.00%
go.viam.com/rdk/components/posetracker 87% 0.00%
go.viam.com/rdk/components/sensor 87% 0.00%
go.viam.com/rdk/components/sensor/ultrasonic 34% 0.00%
go.viam.com/rdk/components/servo 79% 0.00%
go.viam.com/rdk/components/servo/gpio 71% 0.00%
go.viam.com/rdk/config 76% 0.00%
go.viam.com/rdk/control 57% 0.00%
go.viam.com/rdk/data 77% 0.00%
go.viam.com/rdk/grpc 25% 0.00%
go.viam.com/rdk/ml 67% 0.00%
go.viam.com/rdk/ml/inference 70% 0.00%
go.viam.com/rdk/motionplan 71% 0.00%
go.viam.com/rdk/operation 81% 0.00%
go.viam.com/rdk/pointcloud 71% -0.09%
go.viam.com/rdk/protoutils 62% 0.00%
go.viam.com/rdk/referenceframe 78% 0.00%
go.viam.com/rdk/registry 88% 0.00%
go.viam.com/rdk/resource 85% 0.00%
go.viam.com/rdk/rimage 78% 0.00%
go.viam.com/rdk/rimage/depthadapter 94% 0.00%
go.viam.com/rdk/rimage/transform 73% 0.00%
go.viam.com/rdk/rimage/transform/cmd/extrinsic_calibration 67% 0.00%
go.viam.com/rdk/robot 93% 0.00%
go.viam.com/rdk/robot/client 81% 0.00%
go.viam.com/rdk/robot/framesystem 68% 0.00%
go.viam.com/rdk/robot/impl 80% +0.24%
go.viam.com/rdk/robot/server 59% 0.00%
go.viam.com/rdk/robot/web 61% 0.00%
go.viam.com/rdk/robot/web/stream 87% 0.00%
go.viam.com/rdk/services/armremotecontrol 75% 0.00%
go.viam.com/rdk/services/armremotecontrol/builtin 25% 0.00%
go.viam.com/rdk/services/baseremotecontrol 75% 0.00%
go.viam.com/rdk/services/baseremotecontrol/builtin 66% 0.00%
go.viam.com/rdk/services/datamanager 66% 0.00%
go.viam.com/rdk/services/datamanager/builtin 80% 0.00%
go.viam.com/rdk/services/datamanager/datacapture 21% 0.00%
go.viam.com/rdk/services/datamanager/datasync 72% 0.00%
go.viam.com/rdk/services/motion 68% 0.00%
go.viam.com/rdk/services/motion/builtin 89% 0.00%
go.viam.com/rdk/services/navigation 54% 0.00%
go.viam.com/rdk/services/sensors 78% 0.00%
go.viam.com/rdk/services/sensors/builtin 97% 0.00%
go.viam.com/rdk/services/shell 14% 0.00%
go.viam.com/rdk/services/slam 85% 0.00%
go.viam.com/rdk/services/slam/builtin 72% 0.00%
go.viam.com/rdk/services/vision 80% 0.00%
go.viam.com/rdk/services/vision/builtin 74% 0.00%
go.viam.com/rdk/spatialmath 85% 0.00%
go.viam.com/rdk/subtype 96% 0.00%
go.viam.com/rdk/utils 71% 0.00%
go.viam.com/rdk/vision 26% 0.00%
go.viam.com/rdk/vision/chess 80% 0.00%
go.viam.com/rdk/vision/delaunay 87% 0.00%
go.viam.com/rdk/vision/keypoints 92% 0.00%
go.viam.com/rdk/vision/objectdetection 82% 0.00%
go.viam.com/rdk/vision/odometry 60% 0.00%
go.viam.com/rdk/vision/odometry/cmd 0% 0.00%
go.viam.com/rdk/vision/segmentation 49% 0.00%
go.viam.com/rdk/web/server 26% 0.00%
Summary 66% (19543 / 29500) 0.00%

@kkufieta kkufieta requested a review from a user November 17, 2022 10:03
Copy link

@ghost ghost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

services/slam/builtin/orbslam_int_test.go Show resolved Hide resolved
services/slam/builtin/orbslam_int_test.go Show resolved Hide resolved
@kkufieta kkufieta merged commit 6276377 into viamrobotics:main Nov 17, 2022
@kkufieta kkufieta deleted the kk/DATA-749-fix-test-timeout branch November 17, 2022 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
safe to test This pull request is marked safe to test from a trusted zone
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants