Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DATA-764 RDK blocks user-specified video path on reconnect #1595

Merged
merged 8 commits into from
Nov 21, 2022

Conversation

bazile-clyde
Copy link
Contributor

@bazile-clyde bazile-clyde commented Nov 15, 2022

This commit must be merged after the necessary changes in gostream are pushed to master (see edaniels/gostream#15).

This commit sets up a go routine to monitor the status a video driver at the path specified by the user. If that webcam is unplugged this go routine will close the associated driver within at most a half second. To verify this works

  1. Ensure that your local RDK points to a local copy of the gostream library with commit unblock video path after webcam unplug edaniels/gostream#15:
pi@raspberrypi:~/rdk $ go mod edit -replace github.com/edaniels/gostream=/home/pi/gostream

assuming your local copy of gostream is located at /home/pi/gostream.

  1. Run the RDK with this commit on a raspberry pi which has a webcam connected to it:
pi@raspberrypi:~/rdk $ go run web/cmd/server/main.go -config /etc/viam.json -debug

The video_path should be set video0 or another preferred path.

  1. Observe the video stream is visible in the app.
  2. Also observer that the video path of the camera is video0 or your specified path from step 2:
pi@raspberrypi:~ $ ls -l /dev/v4l/by-path
total 0
lrwxrwxrwx 1 root root 13 Nov 15 00:07 platform-bcm2835-codec-video-index0 -> ../../video10
lrwxrwxrwx 1 root root 13 Nov 15 00:07 platform-bcm2835-isp-video-index0 -> ../../video20
lrwxrwxrwx 1 root root 13 Nov 15 00:07 platform-bcm2835-isp-video-index1 -> ../../video21
lrwxrwxrwx 1 root root 13 Nov 15 00:07 platform-bcm2835-isp-video-index2 -> ../../video22
lrwxrwxrwx 1 root root 13 Nov 15 00:07 platform-bcm2835-isp-video-index3 -> ../../video23
lrwxrwxrwx 1 root root 12 Nov 15 20:51 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.3:1.0-video-index0 -> ../../video0
lrwxrwxrwx 1 root root 12 Nov 15 20:51 platform-fd500000.pcie-pci-0000:01:00.0-usb-0:1.3:1.0-video-index1 -> ../../video1
lrwxrwxrwx 1 root root 13 Nov 15 00:07 platform-feb10000.codec-video-index0 -> ../../video19
  1. Unplug then re-plug the camera back in and ensure the video path maps back to its original path, that is, video0 (and video1) in this example.

Note, you will not be able to stream video from the app after the camera is plugged back in. This will be resolved in DATA-597

@bazile-clyde bazile-clyde changed the title RDK blocks user-specified video path on reconnect DATA-764 RDK blocks user-specified video path on reconnect Nov 15, 2022
@kharijarrett kharijarrett added the safe to test This pull request is marked safe to test from a trusted zone label Nov 15, 2022
return cam, err
}

utils2.PanicCapturingGo(func() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] goutils

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

}

utils2.PanicCapturingGo(func() {
ticker := time.NewTicker(time.Second / 2)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] 500*time.Millisecond

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

utils2.PanicCapturingGo(func() {
ticker := time.NewTicker(time.Second / 2)
defer ticker.Stop()
defer utils2.TryClose(ctx, cam)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this double close okay?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still an issue now that I'm using cam.Close directly? I'm not sure I understand what's double-closed here. If this is in reference to the close done by getStatusVideoSource, getStatusVideoSource now removed.

defer utils2.TryClose(ctx, cam)

for {
select {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the for loop you should use the goutils.SelectContextOrWaitChan or SelectContextOrWait. The ticker isn't very necessary making the latter usage easier. The risk of the select as it is is that there's a 50/50 chance of each case happening if both case are eligible

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh fancy. Done!

constraints mediadevices.MediaStreamConstraints,
logger golog.Logger,
) (driver.State, error) {
name := resolveSourceName(path, fromLabel)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we call this once to avoid the extra syscalls?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed. No longer needed.

for {
select {
case <-ticker.C:
if _, err := getStatusVideoSource(attrs.Path, false, constraints, logger); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's going to reopen the camera if this closes it behind other caller's backs? This goroutine also needs to respect the Close of the camera resource itself such that it waits to shut this goroutine down.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first question shouldn't be an issue anymore since I'm no longer using getStatusVideoSource to get the source. I'm not sure if the second part is still relevant though.

As far as re-opening the camera in general that'll be done in DATA-597.

@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 18, 2022
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 18, 2022
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 18, 2022
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 18, 2022
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 18, 2022
@bazile-clyde
Copy link
Contributor Author

@bhaney and @kharijarrett the gostream changes have been merged. This can be thoroughly reviewed now.

// mediadevices connects to the OS to get the properties for a driver. If the OS no longer knows
// about a specific driver then properties will be empty, and we can safely assume the driver no
// longer exists and should be closed.
if len(gostream.PropertiesFromMediaSource[image.Image, prop.Video](src)) == 0 {
Copy link
Contributor Author

@bazile-clyde bazile-clyde Nov 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For gostream.PropertiesFromMediaSource[T, U any](gostream.MediaSource[T]) there's no way to derive U from MediaSource[T] and the compiler complains of this. Because of that I pass the type information explicitly.

src := camera.SourceFromCamera(cam)
defer func() {
if err := cam.Close(ctx); err != nil {
logger.Debugf("failed to close camera: %v", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debugw(msg, "error", err) instead

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 21, 2022
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 21, 2022
}

goutils.PanicCapturingGo(func() {
src := camera.SourceFromCamera(cam)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very solid. We love a good fix! My only thing is maybe SourceFromCamera should return an error instead of just a nil src (when things go wrong). Then also, maybe check that error in here and handle it however if you can't get a source from ya camera.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. Done!

@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Nov 21, 2022
Copy link
Member

@kharijarrett kharijarrett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay! LGTM!

@github-actions
Copy link
Contributor

Code Coverage

Note: merge base coverage results not available, comparing against closest 6fe86df~1=(f70bc29) instead

Package Line Rate Delta Health
go.viam.com/rdk/components/arm 58% 0.00%
go.viam.com/rdk/components/arm/universalrobots 13% 0.00%
go.viam.com/rdk/components/arm/xarm 2% 0.00%
go.viam.com/rdk/components/arm/yahboom 7% 0.00%
go.viam.com/rdk/components/audioinput 55% -0.34%
go.viam.com/rdk/components/base 68% 0.00%
go.viam.com/rdk/components/base/agilex 62% 0.00%
go.viam.com/rdk/components/base/boat 41% 0.00%
go.viam.com/rdk/components/base/wheeled 76% 0.00%
go.viam.com/rdk/components/board 69% 0.00%
go.viam.com/rdk/components/board/arduino 10% 0.00%
go.viam.com/rdk/components/board/commonsysfs 47% 0.00%
go.viam.com/rdk/components/board/fake 38% 0.00%
go.viam.com/rdk/components/board/numato 19% 0.00%
go.viam.com/rdk/components/board/pi 50% 0.00%
go.viam.com/rdk/components/camera 65% -0.53%
go.viam.com/rdk/components/camera/fake 67% 0.00%
go.viam.com/rdk/components/camera/ffmpeg 77% 0.00%
go.viam.com/rdk/components/camera/transformpipeline 80% -0.41%
go.viam.com/rdk/components/camera/videosource 55% -2.30%
go.viam.com/rdk/components/encoder/fake 77% 0.00%
go.viam.com/rdk/components/gantry 68% 0.00%
go.viam.com/rdk/components/gantry/multiaxis 84% 0.00%
go.viam.com/rdk/components/gantry/oneaxis 86% 0.00%
go.viam.com/rdk/components/generic 84% 0.00%
go.viam.com/rdk/components/gripper 81% 0.00%
go.viam.com/rdk/components/input 87% 0.00%
go.viam.com/rdk/components/input/gpio 83% 0.00%
go.viam.com/rdk/components/motor 82% 0.00%
go.viam.com/rdk/components/motor/dimensionengineering 63% 0.00%
go.viam.com/rdk/components/motor/dmc4000 69% 0.00%
go.viam.com/rdk/components/motor/fake 57% 0.00%
go.viam.com/rdk/components/motor/gpio 64% 0.00%
go.viam.com/rdk/components/motor/gpiostepper 57% +0.59%
go.viam.com/rdk/components/motor/tmcstepper 62% 0.00%
go.viam.com/rdk/components/movementsensor 76% 0.00%
go.viam.com/rdk/components/movementsensor/cameramono 40% 0.00%
go.viam.com/rdk/components/movementsensor/gpsnmea 37% 0.00%
go.viam.com/rdk/components/movementsensor/gpsrtk 28% 0.00%
go.viam.com/rdk/components/posetracker 87% 0.00%
go.viam.com/rdk/components/sensor 87% 0.00%
go.viam.com/rdk/components/sensor/ultrasonic 34% 0.00%
go.viam.com/rdk/components/servo 79% 0.00%
go.viam.com/rdk/components/servo/gpio 71% 0.00%
go.viam.com/rdk/config 76% -0.08%
go.viam.com/rdk/control 57% 0.00%
go.viam.com/rdk/data 77% 0.00%
go.viam.com/rdk/grpc 25% 0.00%
go.viam.com/rdk/ml 67% 0.00%
go.viam.com/rdk/ml/inference 70% 0.00%
go.viam.com/rdk/motionplan 71% 0.00%
go.viam.com/rdk/operation 81% 0.00%
go.viam.com/rdk/pointcloud 71% -0.09%
go.viam.com/rdk/protoutils 62% 0.00%
go.viam.com/rdk/referenceframe 78% 0.00%
go.viam.com/rdk/registry 88% 0.00%
go.viam.com/rdk/resource 85% 0.00%
go.viam.com/rdk/rimage 78% 0.00%
go.viam.com/rdk/rimage/depthadapter 94% 0.00%
go.viam.com/rdk/rimage/transform 73% 0.00%
go.viam.com/rdk/rimage/transform/cmd/extrinsic_calibration 67% 0.00%
go.viam.com/rdk/robot 93% 0.00%
go.viam.com/rdk/robot/client 81% 0.00%
go.viam.com/rdk/robot/framesystem 68% 0.00%
go.viam.com/rdk/robot/impl 80% 0.00%
go.viam.com/rdk/robot/server 59% +0.97%
go.viam.com/rdk/robot/web 60% 0.00%
go.viam.com/rdk/robot/web/stream 87% 0.00%
go.viam.com/rdk/services/armremotecontrol 75% 0.00%
go.viam.com/rdk/services/armremotecontrol/builtin 25% 0.00%
go.viam.com/rdk/services/baseremotecontrol 75% 0.00%
go.viam.com/rdk/services/baseremotecontrol/builtin 66% 0.00%
go.viam.com/rdk/services/datamanager 66% 0.00%
go.viam.com/rdk/services/datamanager/builtin 80% 0.00%
go.viam.com/rdk/services/datamanager/datacapture 21% 0.00%
go.viam.com/rdk/services/datamanager/datasync 72% 0.00%
go.viam.com/rdk/services/motion 68% 0.00%
go.viam.com/rdk/services/motion/builtin 89% 0.00%
go.viam.com/rdk/services/navigation 54% 0.00%
go.viam.com/rdk/services/sensors 78% 0.00%
go.viam.com/rdk/services/sensors/builtin 97% 0.00%
go.viam.com/rdk/services/shell 14% 0.00%
go.viam.com/rdk/services/slam 85% 0.00%
go.viam.com/rdk/services/slam/builtin 72% 0.00%
go.viam.com/rdk/services/vision 80% 0.00%
go.viam.com/rdk/services/vision/builtin 74% 0.00%
go.viam.com/rdk/spatialmath 86% 0.00%
go.viam.com/rdk/subtype 96% 0.00%
go.viam.com/rdk/utils 71% 0.00%
go.viam.com/rdk/vision 26% 0.00%
go.viam.com/rdk/vision/chess 80% 0.00%
go.viam.com/rdk/vision/delaunay 87% 0.00%
go.viam.com/rdk/vision/keypoints 92% 0.00%
go.viam.com/rdk/vision/objectdetection 82% 0.00%
go.viam.com/rdk/vision/odometry 60% 0.00%
go.viam.com/rdk/vision/odometry/cmd 0% 0.00%
go.viam.com/rdk/vision/segmentation 49% 0.00%
go.viam.com/rdk/web/server 26% 0.00%
Summary 66% (19539 / 29517) -0.07%

Copy link
Member

@bhaney bhaney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just one question on the helper function SourceFromCamera

Comment on lines +197 to +198
if asSrc, ok := cam.(*videoSource); ok {
return asSrc.videoSource, nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

camera's already fulfill the VideoSource interface too, you could also just use cam.(gostream.VideoSource) directly. Unless you want to get rid of all the extra camera methods on purpose

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it compiles but it does not work and initially I was confused as to why it wouldn't. I printed out the type info to help my understanding.

fmt.Printf("cam: %T\n", cam)
fmt.Printf("asSrc: %T\n", asSrc)
fmt.Printf("asSrc.videoSource: %T\n", asSrc.videoSource)
...
cam: *camera.videoSource
asSrc: *camera.videoSource
asSrc.videoSource: *gostream.mediaSource[image.Image,github.com/pion/mediadevices/pkg/prop.Video]

What we ultimately need is the last type, a gostream.mediaSource[image.Image, prop.Video] which contains the driver we'll eventually query to retrieve its properties. We can't access any of the internals of camera.Camera (it's just an interface) so the chain of casting calls is used to cast that interface into types we know the internals of.

Trying to do all of this in one function wouldn't work either since that would require gostream to know about the private struct camera.videoSource.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
safe to test This pull request is marked safe to test from a trusted zone
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants